Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
].each do |token, user, invited, has_profile|
test "visit home page for user #{token}" do
+ Rails.configuration.Users.AnonymousUserToken = ""
if !token
visit ('/')
else
test "no SSH public key notification when shell_in_a_box_url is configured" do
Rails.configuration.Services.WebShell.ExternalURL = URI('http://example.com')
+ Rails.configuration.Users.AnonymousUserToken = ""
visit page_with_token('job_reader')
click_link 'notifications-menu'
assert_no_selector 'a', text:'Click here to set up an SSH public key for use with Arvados.'
['job_reader2', false],
].each do |user, readable|
test "view job with components as #{user} user" do
+ Rails.configuration.Users.AnonymousUserToken = ""
job = api_fixture('jobs')['running_job_with_components']
component1 = api_fixture('jobs')['completed_job_in_publicly_accessible_project']
component2 = api_fixture('pipeline_instances')['running_pipeline_with_complete_job']
|"v1.1.4":#v1_1_4|"v1.1.3":#v1_1_3|"v1.1.2":#v1_1_2|"v1.1.1":#v1_1_1|"v1.1.0":#v1_1_0|
|\5. "older":#older|
-h3(#master). development master (as of 2019-08-09)
+h3(#master). development master (as of 2019-08-12)
+
+h4. Keep-web dropped support on command line flags configuration
+
+As we're migrating to a central cluster configuration file, the already deprecated way of getting configurations via environment variables and command line flags isn't valid anymore. Current keep-web supports both the now legacy @keep-web.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check "keep-web's install guide":{{site.baseurl}}/install/install-keep-web.html for more details.
h4. Jobs API is read-only
<notextile>
<pre><code>~$ <span class="userinput">keep-web -h</span>
Usage of keep-web:
- -allow-anonymous
- Serve public data to anonymous clients. Try the token supplied in the ARVADOS_API_TOKEN environment variable when none of the tokens provided in an HTTP request succeed in reading the desired collection. (default false)
- -attachment-only-host string
- Accept credentials, and add "Content-Disposition: attachment" response headers, for requests at this hostname:port. Prohibiting inline display makes it possible to serve untrusted and non-public content from a single origin, i.e., without wildcard DNS or TLS.
- -listen string
- Address to listen on: "host:port", or ":port" to listen on all interfaces. (default ":80")
- -trust-all-content
- Serve non-public content from a single origin. Dangerous: read docs before using!
+ -config file
+ Site configuration file (default may be overridden by setting an ARVADOS_CONFIG environment variable) (default "/etc/arvados/config.yml")
+ -dump-config
+ write current configuration to stdout and exit
+ -legacy-crunch-dispatch-slurm-config file
+ Legacy crunch-dispatch-slurm configuration file (default "/etc/arvados/crunch-dispatch-slurm/crunch-dispatch-slurm.yml")
+ -legacy-keepstore-config file
+ Legacy keepstore configuration file (default "/etc/arvados/keepstore/keepstore.yml")
+ -legacy-keepweb-config file
+ Legacy keep-web configuration file (default "/etc/arvados/keep-web/keep-web.yml")
+ -legacy-ws-config file
+ Legacy arvados-ws configuration file (default "/etc/arvados/ws/ws.yml")
+ -skip-legacy
+ Don't load legacy config files
+ -version
+ print version information and exit.
</code></pre>
</notextile>
Install runit to supervise the Keep-web daemon. {% include 'install_runit' %}
+Set the cluster config file like the following:
+
+<notextile>
+<pre><code>Clusters:
+ <span class="userinput">uuid_prefix</span>:
+ SystemRootToken: "{{railsout}}"
+ Services:
+ Controller:
+ ExternalURL: "https://<span class="userinput">uuid_prefix</span>.your.domain"
+ Insecure: false
+ WebDAV:
+ InternalURLs:
+ "http://keep_web_hostname_goes_here:9002/": {}
+ WebDAVDownload:
+ InternalURLs:
+ "http://keep_web_hostname_goes_here:9002/": {}
+ ExternalURL: "https://download.<span class="userinput">uuid_prefix</span>.your.domain/"
+ Users:
+ AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxx"
+ Collections:
+ TrustAllContent: false
+</code></pre>
+</notextile>
+
The basic command to start Keep-web in the service run script is:
<notextile>
-<pre><code>export ARVADOS_API_HOST=<span class="userinput">uuid_prefix</span>.your.domain
-export ARVADOS_API_TOKEN="<span class="userinput">{{railsout}}</span>"
-exec sudo -u nobody keep-web \
- -listen=<span class="userinput">:9002</span> \
- -attachment-only-host=<span class="userinput">download.uuid_prefix.your.domain</span> \
- -allow-anonymous \
- 2>&1
+<pre><code>exec sudo -u nobody keep-web
</code></pre>
</notextile>
-Omit the @-allow-anonymous@ argument if you do not want to serve public data.
+{% include 'notebox_begin' %}
+Please take into consideration that the config file should be world-readable.
+{% include 'notebox_end' %}
+
+Set @Users.AnonymousUserToken: ""@ (empty string) if you do not want to serve public data.
-Set @ARVADOS_API_HOST_INSECURE=1@ if your API server's TLS certificate is not signed by a recognized CA.
+Set @TLS.Insecure: true@ if your API server's TLS certificate is not signed by a recognized CA.
h3. Set up a reverse proxy with TLS support
If neither of the above wildcard options is feasible, you have two choices:
# Serve web content at @collections.uuid_prefix.your.domain@, but only for unauthenticated requests (public data and collection sharing links). Authenticated requests will always result in file downloads, using the @download@ name. For example, the Workbench "preview" button and the "view entire log file" link will invoke file downloads instead of displaying content in the browser window.
-# In the special case where you know you are immune to XSS exploits, you can enable the "trust all content" mode in Keep-web (with the @-trust-all-content@ command line flag) and Workbench (with the @trust_all_content@ item in @application.yml@). With both of these enabled, inline web content can be served from a single @collections@ host name; no wildcard DNS or certificate is needed. Do not do this without understanding the security implications described in the "Keep-web documentation":http://godoc.org/github.com/curoverse/arvados/services/keep-web.
+# In the special case where you know you are immune to XSS exploits, you can enable the "trust all content" mode in Keep-web and Workbench (setting @Collections.TrustAllContent: true@ on the config file). With this enabled, inline web content can be served from a single @collections@ host name; no wildcard DNS or certificate is needed. Do not do this without understanding the security implications described in the "Keep-web documentation":http://godoc.org/github.com/curoverse/arvados/services/keep-web.
h3. Tell Workbench about the Keep-web service
Workbench has features like "download file from collection" and "show image" which work better if the content is served by Keep-web rather than Workbench itself. We recommend using the two different hostnames ("download" and "collections" above) for file downloads and inline content respectively.
-Add the following entry to your Workbench configuration file (@/etc/arvados/workbench/application.yml@). This URL will be used for file downloads.
+Add the following entry to your cluster configuration file (@/etc/arvados/config.yml@). This URL will be used for file downloads.
<notextile>
-<pre><code>keep_web_download_url: https://download.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}
+<pre><code>Clusters:
+ zzzzz:
+ Services:
+ WebDAVDownload:
+ ExternalURL: "https://download.<span class="userinput">uuid_prefix</span>.your.domain/"
</code></pre>
</notextile>
-Additionally, add *one* of the following entries to your Workbench configuration file, depending on your DNS setup. This URL will be used to serve user content that can be displayed in the browser, like image previews and static HTML pages.
+Additionally, add *one* of the following entries to your Workbench cluster configuration file, depending on your DNS setup. This URL will be used to serve user content that can be displayed in the browser, like image previews and static HTML pages.
<notextile>
-<pre><code>keep_web_url: https://%{uuid_or_pdh}--collections.<span class="userinput">uuid_prefix</span>.your.domain
-keep_web_url: https://%{uuid_or_pdh}.collections.<span class="userinput">uuid_prefix</span>.your.domain
-keep_web_url: https://collections.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}
+<pre><code>Clusters:
+ zzzzz:
+ Services:
+ WebDAV:
+ ExternalURL: "https://*--collections.<span class="userinput">uuid_prefix</span>.your.domain"
+ ExternalURL: "https://*.collections.<span class="userinput">uuid_prefix</span>.your.domain"
+ ExternalURL: "https://collections.<span class="userinput">uuid_prefix</span>.your.domain"
</code></pre>
</notextile>
# to run an open instance where anyone can create an account and use
# the system without requiring manual approval.
#
- # The params auto_setup_new_users_with_* are meaningful only when auto_setup_new_users is turned on.
- # auto_setup_name_blacklist is a list of usernames to be blacklisted for auto setup.
+ # The params AutoSetupNewUsersWith* are meaningful only when AutoSetupNewUsers is turned on.
+ # AutoSetupUsernameBlacklist is a list of usernames to be blacklisted for auto setup.
AutoSetupNewUsers: false
AutoSetupNewUsersWithVmUUID: ""
AutoSetupNewUsersWithRepository: false
syslog: {}
SAMPLE: {}
- # When new_users_are_active is set to true, new users will be active
+ # When NewUsersAreActive is set to true, new users will be active
# immediately. This skips the "self-activate" step which enforces
# user agreements. Should only be enabled for development.
NewUsersAreActive: false
# should be an address associated with a Google account.
AutoAdminUserWithEmail: ""
- # If auto_admin_first_user is set to true, the first user to log in when no
+ # If AutoAdminFirstUser is set to true, the first user to log in when no
# other admin users exist will automatically become an admin user.
AutoAdminFirstUser: false
NewUserNotificationRecipients: {}
NewInactiveUserNotificationRecipients: {}
- # Set anonymous_user_token to enable anonymous user access. You can get
+ # Set AnonymousUserToken to enable anonymous user access. You can get
# the token by running "bundle exec ./script/get_anonymous_user_token.rb"
# in the directory where your API server is running.
AnonymousUserToken: ""
# Maximum number of log rows to delete in a single SQL transaction.
#
- # If max_audit_log_delete_batch is 0, log entries will never be
+ # If MaxDeleteBatch is 0, log entries will never be
# deleted by Arvados. Cleanup can be done by an external process
# without affecting any Arvados system processes, as long as very
# recent (<5 minutes old) logs are not deleted.
# identical to the permission key given to Keep. IMPORTANT: This is
# a site secret. It should be at least 50 characters.
#
- # Modifying blob_signing_key will invalidate all existing
+ # Modifying BlobSigningKey will invalidate all existing
# signatures, which can cause programs to fail (e.g., arv-put,
# arv-get, and Crunch jobs). To avoid errors, rotate keys only when
# no such processes are running.
# keepstore servers. Otherwise, reading data blocks and saving
# collections will fail with HTTP 403 permission errors.
#
- # Modifying blob_signature_ttl invalidates existing signatures; see
- # blob_signing_key note above.
+ # Modifying BlobSigningTTL invalidates existing signatures; see
+ # BlobSigningKey note above.
#
# The default is 2 weeks.
BlobSigningTTL: 336h
# Default lifetime for ephemeral collections: 2 weeks. This must not
- # be less than blob_signature_ttl.
+ # be less than BlobSigningTTL.
DefaultTrashLifetime: 336h
# Interval (seconds) between trash sweeps. During a trash sweep,
# If true, enable collection versioning.
# When a collection's preserve_version field is true or the current version
- # is older than the amount of seconds defined on preserve_version_if_idle,
+ # is older than the amount of seconds defined on PreserveVersionIfIdle,
# a snapshot of the collection's previous state is created and linked to
# the current collection.
CollectionVersioning: false
# The default setting (false) is appropriate for a multi-user site.
TrustAllContent: false
+ # Cache parameters for WebDAV content serving:
+ # * TTL: Maximum time to cache manifests and permission checks.
+ # * UUIDTTL: Maximum time to cache collection state.
+ # * MaxCollectionEntries: Maximum number of collection cache entries.
+ # * MaxCollectionBytes: Approximate memory limit for collection cache.
+ # * MaxPermissionEntries: Maximum number of permission cache entries.
+ # * MaxUUIDEntries: Maximum number of UUID cache entries.
+ WebDAVCache:
+ TTL: 300s
+ UUIDTTL: 5s
+ MaxCollectionEntries: 1000
+ MaxCollectionBytes: 100000000
+ MaxPermissionEntries: 1000
+ MaxUUIDEntries: 1000
+
Login:
# These settings are provided by your OAuth2 provider (e.g.,
# sso-provider).
cfg.Clusters[cluster.ClusterID] = *cluster
return nil
}
+
+const defaultKeepWebConfigPath = "/etc/arvados/keep-web/keep-web.yml"
+
+type oldKeepWebConfig struct {
+ Client *arvados.Client
+
+ Listen string
+
+ AnonymousTokens []string
+ AttachmentOnlyHost string
+ TrustAllContent bool
+
+ Cache struct {
+ TTL arvados.Duration
+ UUIDTTL arvados.Duration
+ MaxCollectionEntries int
+ MaxCollectionBytes int64
+ MaxPermissionEntries int
+ MaxUUIDEntries int
+ }
+
+ // Hack to support old command line flag, which is a bool
+ // meaning "get actual token from environment".
+ deprecatedAllowAnonymous bool
+
+ // Authorization token to be included in all health check requests.
+ ManagementToken string
+}
+
+func (ldr *Loader) loadOldKeepWebConfig(cfg *arvados.Config) error {
+ if ldr.KeepWebPath == "" {
+ return nil
+ }
+ var oc oldKeepWebConfig
+ err := ldr.loadOldConfigHelper("keep-web", ldr.KeepWebPath, &oc)
+ if os.IsNotExist(err) && ldr.KeepWebPath == defaultKeepWebConfigPath {
+ return nil
+ } else if err != nil {
+ return err
+ }
+
+ cluster, err := cfg.GetCluster("")
+ if err != nil {
+ return err
+ }
+
+ loadOldClientConfig(cluster, oc.Client)
+
+ cluster.Services.WebDAV.InternalURLs[arvados.URL{Host: oc.Listen}] = arvados.ServiceInstance{}
+ cluster.Services.WebDAVDownload.InternalURLs[arvados.URL{Host: oc.Listen}] = arvados.ServiceInstance{}
+ cluster.Services.WebDAVDownload.ExternalURL = arvados.URL{Host: oc.AttachmentOnlyHost}
+ cluster.TLS.Insecure = oc.Client.Insecure
+ cluster.ManagementToken = oc.ManagementToken
+ cluster.Collections.TrustAllContent = oc.TrustAllContent
+ cluster.Collections.WebDAVCache.TTL = oc.Cache.TTL
+ cluster.Collections.WebDAVCache.UUIDTTL = oc.Cache.UUIDTTL
+ cluster.Collections.WebDAVCache.MaxCollectionEntries = oc.Cache.MaxCollectionEntries
+ cluster.Collections.WebDAVCache.MaxCollectionBytes = oc.Cache.MaxCollectionBytes
+ cluster.Collections.WebDAVCache.MaxPermissionEntries = oc.Cache.MaxPermissionEntries
+ cluster.Collections.WebDAVCache.MaxUUIDEntries = oc.Cache.MaxUUIDEntries
+ if len(oc.AnonymousTokens) > 0 {
+ cluster.Users.AnonymousUserToken = oc.AnonymousTokens[0]
+ if len(oc.AnonymousTokens) > 1 {
+ ldr.Logger.Warn("More than 1 anonymous tokens configured, using only the first and discarding the rest.")
+ }
+ }
+
+ cfg.Clusters[cluster.ClusterID] = *cluster
+ return nil
+}
package config
import (
+ "flag"
+ "io/ioutil"
"os"
+ "time"
+ "git.curoverse.com/arvados.git/sdk/go/arvados"
check "gopkg.in/check.v1"
)
listen: ":9006"
`)
}
+
+func (s *LoadSuite) TestLegacyKeepWebConfig(c *check.C) {
+ content := []byte(`
+{
+ "Client": {
+ "Scheme": "",
+ "APIHost": "example.com",
+ "AuthToken": "abcdefg",
+ },
+ "Listen": ":80",
+ "AnonymousTokens": [
+ "anonusertoken"
+ ],
+ "AttachmentOnlyHost": "download.example.com",
+ "TrustAllContent": true,
+ "Cache": {
+ "TTL": "1m",
+ "UUIDTTL": "1s",
+ "MaxCollectionEntries": 42,
+ "MaxCollectionBytes": 1234567890,
+ "MaxPermissionEntries": 100,
+ "MaxUUIDEntries": 100
+ },
+ "ManagementToken": "xyzzy"
+}
+`)
+ tmpfile, err := ioutil.TempFile("", "example")
+ if err != nil {
+ c.Error(err)
+ }
+ defer os.Remove(tmpfile.Name())
+
+ if _, err := tmpfile.Write(content); err != nil {
+ c.Error(err)
+ }
+ if err := tmpfile.Close(); err != nil {
+ c.Error(err)
+ }
+ flags := flag.NewFlagSet("keep-web", flag.ExitOnError)
+ ldr := testLoader(c, "Clusters: {zzzzz: {}}", nil)
+ ldr.SetupFlags(flags)
+ args := ldr.MungeLegacyConfigArgs(ldr.Logger, []string{"-config", tmpfile.Name()}, "-legacy-keepweb-config")
+ flags.Parse(args)
+ cfg, err := ldr.Load()
+ if err != nil {
+ c.Error(err)
+ }
+ c.Check(cfg, check.NotNil)
+ cluster, err := cfg.GetCluster("")
+ if err != nil {
+ c.Error(err)
+ }
+ c.Check(cluster, check.NotNil)
+
+ c.Check(cluster.Services.Controller.ExternalURL, check.Equals, arvados.URL{Scheme: "https", Host: "example.com"})
+ c.Check(cluster.SystemRootToken, check.Equals, "abcdefg")
+
+ c.Check(cluster.Collections.WebDAVCache.TTL, check.Equals, arvados.Duration(60*time.Second))
+ c.Check(cluster.Collections.WebDAVCache.UUIDTTL, check.Equals, arvados.Duration(time.Second))
+ c.Check(cluster.Collections.WebDAVCache.MaxCollectionEntries, check.Equals, 42)
+ c.Check(cluster.Collections.WebDAVCache.MaxCollectionBytes, check.Equals, int64(1234567890))
+ c.Check(cluster.Collections.WebDAVCache.MaxPermissionEntries, check.Equals, 100)
+ c.Check(cluster.Collections.WebDAVCache.MaxUUIDEntries, check.Equals, 100)
+
+ c.Check(cluster.Services.WebDAVDownload.ExternalURL, check.Equals, arvados.URL{Host: "download.example.com"})
+ c.Check(cluster.Services.WebDAVDownload.InternalURLs[arvados.URL{Host: ":80"}], check.NotNil)
+ c.Check(cluster.Services.WebDAV.InternalURLs[arvados.URL{Host: ":80"}], check.NotNil)
+
+ c.Check(cluster.Collections.TrustAllContent, check.Equals, true)
+ c.Check(cluster.Users.AnonymousUserToken, check.Equals, "anonusertoken")
+ c.Check(cluster.ManagementToken, check.Equals, "xyzzy")
+}
"Collections.PreserveVersionIfIdle": true,
"Collections.TrashSweepInterval": false,
"Collections.TrustAllContent": false,
+ "Collections.WebDAVCache": false,
"Containers": true,
"Containers.CloudVMs": false,
"Containers.CrunchRunCommand": false,
# to run an open instance where anyone can create an account and use
# the system without requiring manual approval.
#
- # The params auto_setup_new_users_with_* are meaningful only when auto_setup_new_users is turned on.
- # auto_setup_name_blacklist is a list of usernames to be blacklisted for auto setup.
+ # The params AutoSetupNewUsersWith* are meaningful only when AutoSetupNewUsers is turned on.
+ # AutoSetupUsernameBlacklist is a list of usernames to be blacklisted for auto setup.
AutoSetupNewUsers: false
AutoSetupNewUsersWithVmUUID: ""
AutoSetupNewUsersWithRepository: false
syslog: {}
SAMPLE: {}
- # When new_users_are_active is set to true, new users will be active
+ # When NewUsersAreActive is set to true, new users will be active
# immediately. This skips the "self-activate" step which enforces
# user agreements. Should only be enabled for development.
NewUsersAreActive: false
# should be an address associated with a Google account.
AutoAdminUserWithEmail: ""
- # If auto_admin_first_user is set to true, the first user to log in when no
+ # If AutoAdminFirstUser is set to true, the first user to log in when no
# other admin users exist will automatically become an admin user.
AutoAdminFirstUser: false
NewUserNotificationRecipients: {}
NewInactiveUserNotificationRecipients: {}
- # Set anonymous_user_token to enable anonymous user access. You can get
+ # Set AnonymousUserToken to enable anonymous user access. You can get
# the token by running "bundle exec ./script/get_anonymous_user_token.rb"
# in the directory where your API server is running.
AnonymousUserToken: ""
# Maximum number of log rows to delete in a single SQL transaction.
#
- # If max_audit_log_delete_batch is 0, log entries will never be
+ # If MaxDeleteBatch is 0, log entries will never be
# deleted by Arvados. Cleanup can be done by an external process
# without affecting any Arvados system processes, as long as very
# recent (<5 minutes old) logs are not deleted.
# identical to the permission key given to Keep. IMPORTANT: This is
# a site secret. It should be at least 50 characters.
#
- # Modifying blob_signing_key will invalidate all existing
+ # Modifying BlobSigningKey will invalidate all existing
# signatures, which can cause programs to fail (e.g., arv-put,
# arv-get, and Crunch jobs). To avoid errors, rotate keys only when
# no such processes are running.
# keepstore servers. Otherwise, reading data blocks and saving
# collections will fail with HTTP 403 permission errors.
#
- # Modifying blob_signature_ttl invalidates existing signatures; see
- # blob_signing_key note above.
+ # Modifying BlobSigningTTL invalidates existing signatures; see
+ # BlobSigningKey note above.
#
# The default is 2 weeks.
BlobSigningTTL: 336h
# Default lifetime for ephemeral collections: 2 weeks. This must not
- # be less than blob_signature_ttl.
+ # be less than BlobSigningTTL.
DefaultTrashLifetime: 336h
# Interval (seconds) between trash sweeps. During a trash sweep,
# If true, enable collection versioning.
# When a collection's preserve_version field is true or the current version
- # is older than the amount of seconds defined on preserve_version_if_idle,
+ # is older than the amount of seconds defined on PreserveVersionIfIdle,
# a snapshot of the collection's previous state is created and linked to
# the current collection.
CollectionVersioning: false
# The default setting (false) is appropriate for a multi-user site.
TrustAllContent: false
+ # Cache parameters for WebDAV content serving:
+ # * TTL: Maximum time to cache manifests and permission checks.
+ # * UUIDTTL: Maximum time to cache collection state.
+ # * MaxCollectionEntries: Maximum number of collection cache entries.
+ # * MaxCollectionBytes: Approximate memory limit for collection cache.
+ # * MaxPermissionEntries: Maximum number of permission cache entries.
+ # * MaxUUIDEntries: Maximum number of UUID cache entries.
+ WebDAVCache:
+ TTL: 300s
+ UUIDTTL: 5s
+ MaxCollectionEntries: 1000
+ MaxCollectionBytes: 100000000
+ MaxPermissionEntries: 1000
+ MaxUUIDEntries: 1000
+
Login:
# These settings are provided by your OAuth2 provider (e.g.,
# sso-provider).
Path string
KeepstorePath string
+ KeepWebPath string
CrunchDispatchSlurmPath string
WebsocketPath string
func (ldr *Loader) SetupFlags(flagset *flag.FlagSet) {
flagset.StringVar(&ldr.Path, "config", arvados.DefaultConfigFile, "Site configuration `file` (default may be overridden by setting an ARVADOS_CONFIG environment variable)")
flagset.StringVar(&ldr.KeepstorePath, "legacy-keepstore-config", defaultKeepstoreConfigPath, "Legacy keepstore configuration `file`")
+ flagset.StringVar(&ldr.KeepWebPath, "legacy-keepweb-config", defaultKeepWebConfigPath, "Legacy keep-web configuration `file`")
flagset.StringVar(&ldr.CrunchDispatchSlurmPath, "legacy-crunch-dispatch-slurm-config", defaultCrunchDispatchSlurmConfigPath, "Legacy crunch-dispatch-slurm configuration `file`")
flagset.StringVar(&ldr.WebsocketPath, "legacy-ws-config", defaultWebsocketConfigPath, "Legacy arvados-ws configuration `file`")
flagset.BoolVar(&ldr.SkipLegacy, "skip-legacy", false, "Don't load legacy config files")
if legacyConfigArg != "-legacy-ws-config" {
ldr.WebsocketPath = ""
}
+ if legacyConfigArg != "-legacy-keepweb-config" {
+ ldr.KeepWebPath = ""
+ }
return munged
}
// legacy config file for the current component
for _, err := range []error{
ldr.loadOldKeepstoreConfig(&cfg),
+ ldr.loadOldKeepWebConfig(&cfg),
ldr.loadOldCrunchDispatchSlurmConfig(&cfg),
ldr.loadOldWebsocketConfig(&cfg),
} {
}
}
+type WebDAVCacheConfig struct {
+ TTL Duration
+ UUIDTTL Duration
+ MaxCollectionEntries int
+ MaxCollectionBytes int64
+ MaxPermissionEntries int
+ MaxUUIDEntries int
+}
type Cluster struct {
ClusterID string `json:"-"`
ManagementToken string
PreserveVersionIfIdle Duration
TrashSweepInterval Duration
TrustAllContent bool
+
+ WebDAVCache WebDAVCacheConfig
}
Git struct {
Repositories string
ssl_certificate_key "{{SSLKEY}}";
location / {
proxy_pass http://keep-web;
+ proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
+ proxy_redirect off;
client_max_body_size 0;
proxy_http_version 1.1;
proxy_request_buffering off;
-
- # Unlike other proxy sections, here we need to override the
- # requested Host header and use proxy_redirect because of the
- # way the test suite orchestrates services. Keep-web's "download
- # only" behavior relies on the Host header matching a configured
- # value, but when run_test_servers.py writes keep-web's command
- # line, the keep-web-dl TLS port (which clients will connect to
- # and include in their Host header) has not yet been assigned.
- #
- # In production, "proxy_set_header Host $http_host;
- # proxy_redirect off;" works: keep-web's redirect URLs will
- # match the request URL received by Nginx.
- #
- # Here, keep-web will issue redirects to https://download/ and
- # Nginx will rewrite them.
- #
- proxy_set_header Host download;
- proxy_redirect https://download/ https://$host:{{KEEPWEBDLSSLPORT}}/;
}
}
upstream ws {
keepwebport = internal_port_from_config("WebDAV")
env = os.environ.copy()
- env['ARVADOS_API_TOKEN'] = auth_token('anonymous')
logf = open(_logfilename('keep-web'), 'a')
keepweb = subprocess.Popen(
- ['keep-web',
- '-allow-anonymous',
- '-attachment-only-host=download',
- '-management-token=e687950a23c3a9bceec28c6223a06c79',
- '-listen=:'+str(keepwebport)],
+ ['keep-web'],
env=env, stdin=open('/dev/null'), stdout=logf, stderr=logf)
with open(_pidfile('keep-web'), 'w') as f:
f.write(str(keepweb.pid))
"TLS": {
"Insecure": True
},
- "Services": services
+ "Services": services,
+ "Users": {
+ "AnonymousUserToken": auth_token('anonymous')
+ },
+ "Collections": {
+ "TrustAllContent": True
+ }
}
}
}
const metricsUpdateInterval = time.Second / 10
type cache struct {
- TTL arvados.Duration
- UUIDTTL arvados.Duration
- MaxCollectionEntries int
- MaxCollectionBytes int64
- MaxPermissionEntries int
- MaxUUIDEntries int
-
+ config *arvados.WebDAVCacheConfig
registry *prometheus.Registry
metrics cacheMetrics
pdhs *lru.TwoQueueCache
func (c *cache) setup() {
var err error
- c.pdhs, err = lru.New2Q(c.MaxUUIDEntries)
+ c.pdhs, err = lru.New2Q(c.config.MaxUUIDEntries)
if err != nil {
panic(err)
}
- c.collections, err = lru.New2Q(c.MaxCollectionEntries)
+ c.collections, err = lru.New2Q(c.config.MaxCollectionEntries)
if err != nil {
panic(err)
}
- c.permissions, err = lru.New2Q(c.MaxPermissionEntries)
+ c.permissions, err = lru.New2Q(c.config.MaxPermissionEntries)
if err != nil {
panic(err)
}
})
if err == nil {
c.collections.Add(client.AuthToken+"\000"+coll.PortableDataHash, &cachedCollection{
- expire: time.Now().Add(time.Duration(c.TTL)),
+ expire: time.Now().Add(time.Duration(c.config.TTL)),
collection: &updated,
})
}
}
if current.PortableDataHash == pdh {
c.permissions.Add(permKey, &cachedPermission{
- expire: time.Now().Add(time.Duration(c.TTL)),
+ expire: time.Now().Add(time.Duration(c.config.TTL)),
})
if pdh != targetID {
c.pdhs.Add(targetID, &cachedPDH{
- expire: time.Now().Add(time.Duration(c.UUIDTTL)),
+ expire: time.Now().Add(time.Duration(c.config.UUIDTTL)),
pdh: pdh,
})
}
if err != nil {
return nil, err
}
- exp := time.Now().Add(time.Duration(c.TTL))
+ exp := time.Now().Add(time.Duration(c.config.TTL))
c.permissions.Add(permKey, &cachedPermission{
expire: exp,
})
c.pdhs.Add(targetID, &cachedPDH{
- expire: time.Now().Add(time.Duration(c.UUIDTTL)),
+ expire: time.Now().Add(time.Duration(c.config.UUIDTTL)),
pdh: collection.PortableDataHash,
})
c.collections.Add(arv.ApiToken+"\000"+collection.PortableDataHash, &cachedCollection{
expire: exp,
collection: collection,
})
- if int64(len(collection.ManifestText)) > c.MaxCollectionBytes/int64(c.MaxCollectionEntries) {
+ if int64(len(collection.ManifestText)) > c.config.MaxCollectionBytes/int64(c.config.MaxCollectionEntries) {
go c.pruneCollections()
}
return collection, nil
}
}
for i, k := range keys {
- if size <= c.MaxCollectionBytes {
+ if size <= c.config.MaxCollectionBytes {
break
}
if expired[i] {
arv, err := arvadosclient.MakeArvadosClient()
c.Assert(err, check.Equals, nil)
- cache := DefaultConfig().Cache
+ cache := newConfig(s.Config).Cache
cache.registry = prometheus.NewRegistry()
// Hit the same collection 5 times using the same token. Only
arv, err := arvadosclient.MakeArvadosClient()
c.Assert(err, check.Equals, nil)
- cache := DefaultConfig().Cache
+ cache := newConfig(s.Config).Cache
cache.registry = prometheus.NewRegistry()
for _, forceReload := range []bool{false, true, false, true} {
arv, err := arvadosclient.MakeArvadosClient()
c.Assert(err, check.Equals, nil)
- cache := DefaultConfig().Cache
+ cache := newConfig(s.Config).Cache
cache.registry = prometheus.NewRegistry()
for _, forceReload := range []bool{false, true, false, true} {
}
func (s *IntegrationSuite) testCadaver(c *check.C, password string, pathFunc func(arvados.Collection) (string, string, string), skip func(string) bool) {
- s.testServer.Config.AnonymousTokens = []string{arvadostest.AnonymousToken}
+ s.testServer.Config.cluster.Users.AnonymousUserToken = arvadostest.AnonymousToken
testdata := []byte("the human tragedy consists in the necessity of living with the consequences of actions performed under the pressure of compulsions we do not understand")
// unauthenticated request, which it only does in
// AttachmentOnlyHost, TrustAllContent, and
// per-collection vhost cases.
- s.testServer.Config.AttachmentOnlyHost = s.testServer.Addr
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = s.testServer.Addr
cmd.Env = append(os.Environ(), "HOME="+tempdir)
f, err := os.OpenFile(filepath.Join(tempdir, ".netrc"), os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0600)
//
// Configuration
//
-// The default configuration file location is
-// /etc/arvados/keep-web/keep-web.yml.
+// The default cluster configuration file location is
+// /etc/arvados/config.yml.
//
// Example configuration file
//
-// Client:
-// APIHost: "zzzzz.arvadosapi.com:443"
-// AuthToken: ""
-// Insecure: false
-// Listen: :1234
-// AnonymousTokens:
-// - xxxxxxxxxxxxxxxxxxxx
-// AttachmentOnlyHost: ""
-// TrustAllContent: false
+// Clusters:
+// zzzzz:
+// SystemRootToken: ""
+// Services:
+// Controller:
+// ExternalURL: "https://example.com"
+// Insecure: false
+// WebDAV:
+// InternalURLs:
+// "http://:1234/": {}
+// WebDAVDownload:
+// InternalURLs:
+// "http://:1234/": {}
+// ExternalURL: "https://download.example.com/"
+// Users:
+// AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxx"
+// Collections:
+// TrustAllContent: false
//
// Starting the server
//
// Start a server using the default config file
-// /etc/arvados/keep-web/keep-web.yml:
+// /etc/arvados/config.yml:
//
// keep-web
//
-// Start a server using the config file /path/to/keep-web.yml:
+// Start a server using the config file /path/to/config.yml:
//
-// keep-web -config /path/to/keep-web.yml
+// keep-web -config /path/to/config.yml
//
// Proxy configuration
//
//
// Anonymous downloads
//
-// The "AnonymousTokens" configuration entry is an array of tokens to
-// use when processing anonymous requests, i.e., whenever a web client
+// The "Users.AnonymousUserToken" configuration entry used when
+// when processing anonymous requests, i.e., whenever a web client
// does not supply its own Arvados API token via path, query string,
// cookie, or request header.
//
-// "AnonymousTokens":["xxxxxxxxxxxxxxxxxxxxxxx"]
+// Clusters:
+// zzzzz:
+// Users:
+// AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxxxxx"
//
// See http://doc.arvados.org/install/install-keep-web.html for examples.
//
// only when the designated origin matches exactly the Host header
// provided by the client or downstream proxy.
//
-// "AttachmentOnlyHost":"domain.example:9999"
+// Clusters:
+// zzzzz:
+// Services:
+// WebDAVDownload:
+// ExternalURL: "https://domain.example:9999"
//
// Trust All Content mode
//
//
// In such cases you can enable trust-all-content mode.
//
-// "TrustAllContent":true
+// Clusters:
+// zzzzz:
+// Collections:
+// TrustAllContent: true
//
// When TrustAllContent is enabled, the only effect of the
-// AttachmentOnlyHost flag is to add a "Content-Disposition:
+// Attachment-Only host setting is to add a "Content-Disposition:
// attachment" header.
//
-// "AttachmentOnlyHost":"domain.example:9999",
-// "TrustAllContent":true
+// Clusters:
+// zzzzz:
+// Services:
+// WebDAVDownload:
+// ExternalURL: "https://domain.example:9999"
+// Collections:
+// TrustAllContent: true
//
// Depending on your site configuration, you might also want to enable
// the "trust all content" setting in Workbench. Normally, Workbench
keepclient.RefreshServiceDiscoveryOnSIGHUP()
h.healthHandler = &health.Handler{
- Token: h.Config.ManagementToken,
+ Token: h.Config.cluster.ManagementToken,
Prefix: "/_health/",
}
var pathToken bool
var attachment bool
var useSiteFS bool
- credentialsOK := h.Config.TrustAllContent
+ credentialsOK := h.Config.cluster.Collections.TrustAllContent
- if r.Host != "" && r.Host == h.Config.AttachmentOnlyHost {
+ if r.Host != "" && r.Host == h.Config.cluster.Services.WebDAVDownload.ExternalURL.Host {
credentialsOK = true
attachment = true
} else if r.FormValue("disposition") == "attachment" {
}
if tokens == nil {
- tokens = append(reqTokens, h.Config.AnonymousTokens...)
+ tokens = append(reqTokens, h.Config.cluster.Users.AnonymousUserToken)
}
if len(targetPath) > 0 && targetPath[0] == "_" {
"regexp"
"strings"
+ "git.curoverse.com/arvados.git/lib/config"
"git.curoverse.com/arvados.git/sdk/go/arvados"
"git.curoverse.com/arvados.git/sdk/go/arvadostest"
"git.curoverse.com/arvados.git/sdk/go/auth"
var _ = check.Suite(&UnitSuite{})
-type UnitSuite struct{}
+type UnitSuite struct {
+ Config *arvados.Config
+}
+
+func (s *UnitSuite) SetUpTest(c *check.C) {
+ ldr := config.NewLoader(bytes.NewBufferString("Clusters: {zzzzz: {}}"), nil)
+ ldr.Path = "-"
+ cfg, err := ldr.Load()
+ c.Assert(err, check.IsNil)
+ s.Config = cfg
+}
func (s *UnitSuite) TestCORSPreflight(c *check.C) {
- h := handler{Config: DefaultConfig()}
+ h := handler{Config: newConfig(s.Config)}
u := mustParseURL("http://keep-web.example/c=" + arvadostest.FooCollection + "/foo")
req := &http.Request{
Method: "OPTIONS",
RequestURI: u.RequestURI(),
}
resp := httptest.NewRecorder()
- cfg := DefaultConfig()
- cfg.AnonymousTokens = []string{arvadostest.AnonymousToken}
+ cfg := newConfig(s.Config)
+ cfg.cluster.Users.AnonymousUserToken = arvadostest.AnonymousToken
h := handler{Config: cfg}
h.ServeHTTP(resp, req)
c.Check(resp.Code, check.Equals, http.StatusNotFound)
}
func (s *IntegrationSuite) TestVhostRedirectQueryTokenSiteFS(c *check.C) {
- s.testServer.Config.AttachmentOnlyHost = "download.example.com"
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = "download.example.com"
resp := s.testVhostRedirectTokenToCookie(c, "GET",
"download.example.com/by_id/"+arvadostest.FooCollection+"/foo",
"?api_token="+arvadostest.ActiveToken,
}
func (s *IntegrationSuite) TestPastCollectionVersionFileAccess(c *check.C) {
- s.testServer.Config.AttachmentOnlyHost = "download.example.com"
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = "download.example.com"
resp := s.testVhostRedirectTokenToCookie(c, "GET",
"download.example.com/c="+arvadostest.WazVersion1Collection+"/waz",
"?api_token="+arvadostest.ActiveToken,
}
func (s *IntegrationSuite) TestVhostRedirectQueryTokenTrustAllContent(c *check.C) {
- s.testServer.Config.TrustAllContent = true
+ s.testServer.Config.cluster.Collections.TrustAllContent = true
s.testVhostRedirectTokenToCookie(c, "GET",
"example.com/c="+arvadostest.FooCollection+"/foo",
"?api_token="+arvadostest.ActiveToken,
}
func (s *IntegrationSuite) TestVhostRedirectQueryTokenAttachmentOnlyHost(c *check.C) {
- s.testServer.Config.AttachmentOnlyHost = "example.com:1234"
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = "example.com:1234"
s.testVhostRedirectTokenToCookie(c, "GET",
"example.com/c="+arvadostest.FooCollection+"/foo",
}
func (s *IntegrationSuite) TestAnonymousTokenOK(c *check.C) {
- s.testServer.Config.AnonymousTokens = []string{arvadostest.AnonymousToken}
+ s.testServer.Config.cluster.Users.AnonymousUserToken = arvadostest.AnonymousToken
s.testVhostRedirectTokenToCookie(c, "GET",
"example.com/c="+arvadostest.HelloWorldCollection+"/Hello%20world.txt",
"",
}
func (s *IntegrationSuite) TestAnonymousTokenError(c *check.C) {
- s.testServer.Config.AnonymousTokens = []string{"anonymousTokenConfiguredButInvalid"}
+ s.testServer.Config.cluster.Users.AnonymousUserToken = "anonymousTokenConfiguredButInvalid"
s.testVhostRedirectTokenToCookie(c, "GET",
"example.com/c="+arvadostest.HelloWorldCollection+"/Hello%20world.txt",
"",
}
func (s *IntegrationSuite) TestSpecialCharsInPath(c *check.C) {
- s.testServer.Config.AttachmentOnlyHost = "download.example.com"
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = "download.example.com"
client := s.testServer.Config.Client
client.AuthToken = arvadostest.ActiveToken
}
func (s *IntegrationSuite) TestDirectoryListingWithAnonymousToken(c *check.C) {
- s.testServer.Config.AnonymousTokens = []string{arvadostest.AnonymousToken}
+ s.testServer.Config.cluster.Users.AnonymousUserToken = arvadostest.AnonymousToken
s.testDirectoryListing(c)
}
func (s *IntegrationSuite) TestDirectoryListingWithNoAnonymousToken(c *check.C) {
- s.testServer.Config.AnonymousTokens = nil
+ s.testServer.Config.cluster.Users.AnonymousUserToken = ""
s.testDirectoryListing(c)
}
func (s *IntegrationSuite) testDirectoryListing(c *check.C) {
- s.testServer.Config.AttachmentOnlyHost = "download.example.com"
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = "download.example.com"
authHeader := http.Header{
"Authorization": {"OAuth2 " + arvadostest.ActiveToken},
}
cutDirs: 2,
},
} {
- c.Logf("HTML: %q => %q", trial.uri, trial.expect)
+ comment := check.Commentf("HTML: %q => %q", trial.uri, trial.expect)
resp := httptest.NewRecorder()
u := mustParseURL("//" + trial.uri)
req := &http.Request{
s.testServer.Handler.ServeHTTP(resp, req)
}
if trial.redirect != "" {
- c.Check(req.URL.Path, check.Equals, trial.redirect)
+ c.Check(req.URL.Path, check.Equals, trial.redirect, comment)
}
if trial.expect == nil {
- c.Check(resp.Code, check.Equals, http.StatusNotFound)
+ c.Check(resp.Code, check.Equals, http.StatusNotFound, comment)
} else {
- c.Check(resp.Code, check.Equals, http.StatusOK)
+ c.Check(resp.Code, check.Equals, http.StatusOK, comment)
for _, e := range trial.expect {
- c.Check(resp.Body.String(), check.Matches, `(?ms).*href="./`+e+`".*`)
+ c.Check(resp.Body.String(), check.Matches, `(?ms).*href="./`+e+`".*`, comment)
}
- c.Check(resp.Body.String(), check.Matches, `(?ms).*--cut-dirs=`+fmt.Sprintf("%d", trial.cutDirs)+` .*`)
+ c.Check(resp.Body.String(), check.Matches, `(?ms).*--cut-dirs=`+fmt.Sprintf("%d", trial.cutDirs)+` .*`, comment)
}
- c.Logf("WebDAV: %q => %q", trial.uri, trial.expect)
+ comment = check.Commentf("WebDAV: %q => %q", trial.uri, trial.expect)
req = &http.Request{
Method: "OPTIONS",
Host: u.Host,
resp = httptest.NewRecorder()
s.testServer.Handler.ServeHTTP(resp, req)
if trial.expect == nil {
- c.Check(resp.Code, check.Equals, http.StatusNotFound)
+ c.Check(resp.Code, check.Equals, http.StatusNotFound, comment)
} else {
- c.Check(resp.Code, check.Equals, http.StatusOK)
+ c.Check(resp.Code, check.Equals, http.StatusOK, comment)
}
req = &http.Request{
resp = httptest.NewRecorder()
s.testServer.Handler.ServeHTTP(resp, req)
if trial.expect == nil {
- c.Check(resp.Code, check.Equals, http.StatusNotFound)
+ c.Check(resp.Code, check.Equals, http.StatusNotFound, comment)
} else {
- c.Check(resp.Code, check.Equals, http.StatusMultiStatus)
+ c.Check(resp.Code, check.Equals, http.StatusMultiStatus, comment)
for _, e := range trial.expect {
- c.Check(resp.Body.String(), check.Matches, `(?ms).*<D:href>`+filepath.Join(u.Path, e)+`</D:href>.*`)
+ c.Check(resp.Body.String(), check.Matches, `(?ms).*<D:href>`+filepath.Join(u.Path, e)+`</D:href>.*`, comment)
}
}
}
var updated arvados.Collection
for _, fnm := range []string{"foo.txt", "bar.txt"} {
- s.testServer.Config.AttachmentOnlyHost = "example.com"
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = "example.com"
u, _ := url.Parse("http://example.com/c=" + newCollection.UUID + "/" + fnm)
req := &http.Request{
Method: "DELETE",
}
func (s *IntegrationSuite) TestHealthCheckPing(c *check.C) {
- s.testServer.Config.ManagementToken = arvadostest.ManagementToken
+ s.testServer.Config.cluster.ManagementToken = arvadostest.ManagementToken
authHeader := http.Header{
"Authorization": {"Bearer " + arvadostest.ManagementToken},
}
"flag"
"fmt"
"os"
- "time"
+ "git.curoverse.com/arvados.git/lib/config"
"git.curoverse.com/arvados.git/sdk/go/arvados"
- "git.curoverse.com/arvados.git/sdk/go/config"
"github.com/coreos/go-systemd/daemon"
+ "github.com/ghodss/yaml"
log "github.com/sirupsen/logrus"
)
var (
- defaultConfigPath = "/etc/arvados/keep-web/keep-web.yml"
- version = "dev"
+ version = "dev"
)
// Config specifies server configuration.
type Config struct {
- Client arvados.Client
-
- Listen string
-
- AnonymousTokens []string
- AttachmentOnlyHost string
- TrustAllContent bool
-
- Cache cache
-
- // Hack to support old command line flag, which is a bool
- // meaning "get actual token from environment".
- deprecatedAllowAnonymous bool
-
- //Authorization token to be included in all health check requests.
- ManagementToken string
+ Client arvados.Client
+ Cache cache
+ cluster *arvados.Cluster
}
-// DefaultConfig returns the default configuration.
-func DefaultConfig() *Config {
- return &Config{
- Listen: ":80",
- Cache: cache{
- TTL: arvados.Duration(5 * time.Minute),
- UUIDTTL: arvados.Duration(5 * time.Second),
- MaxCollectionEntries: 1000,
- MaxCollectionBytes: 100000000,
- MaxPermissionEntries: 1000,
- MaxUUIDEntries: 1000,
- },
+func newConfig(arvCfg *arvados.Config) *Config {
+ cfg := Config{}
+ var cls *arvados.Cluster
+ var err error
+ if cls, err = arvCfg.GetCluster(""); err != nil {
+ log.Fatal(err)
}
+ cfg.cluster = cls
+ cfg.Cache.config = &cfg.cluster.Collections.WebDAVCache
+ return &cfg
}
func init() {
})
}
-func main() {
- cfg := DefaultConfig()
-
- var configPath string
- deprecated := " (DEPRECATED -- use config file instead)"
- flag.StringVar(&configPath, "config", defaultConfigPath,
- "`path` to JSON or YAML configuration file")
- flag.StringVar(&cfg.Listen, "listen", "",
- "address:port or :port to listen on"+deprecated)
- flag.BoolVar(&cfg.deprecatedAllowAnonymous, "allow-anonymous", false,
- "Load an anonymous token from the ARVADOS_API_TOKEN environment variable"+deprecated)
- flag.StringVar(&cfg.AttachmentOnlyHost, "attachment-only-host", "",
- "Only serve attachments at the given `host:port`"+deprecated)
- flag.BoolVar(&cfg.TrustAllContent, "trust-all-content", false,
- "Serve non-public content from a single origin. Dangerous: read docs before using!"+deprecated)
- flag.StringVar(&cfg.ManagementToken, "management-token", "",
- "Authorization token to be included in all health check requests.")
-
- dumpConfig := flag.Bool("dump-config", false,
+func configure(logger log.FieldLogger, args []string) *Config {
+ flags := flag.NewFlagSet(args[0], flag.ExitOnError)
+
+ loader := config.NewLoader(os.Stdin, logger)
+ loader.SetupFlags(flags)
+
+ dumpConfig := flags.Bool("dump-config", false,
"write current configuration to stdout and exit")
- getVersion := flag.Bool("version", false,
+ getVersion := flags.Bool("version", false,
"print version information and exit.")
- flag.Usage = usage
- flag.Parse()
+
+ args = loader.MungeLegacyConfigArgs(logger, args[1:], "-legacy-keepweb-config")
+ flags.Parse(args)
// Print version information if requested
if *getVersion {
fmt.Printf("keep-web %s\n", version)
- return
+ return nil
}
- if err := config.LoadFile(cfg, configPath); err != nil {
- if h := os.Getenv("ARVADOS_API_HOST"); h != "" && configPath == defaultConfigPath {
- log.Printf("DEPRECATED: Using ARVADOS_API_HOST environment variable. Use config file instead.")
- cfg.Client.APIHost = h
- } else {
+ arvCfg, err := loader.Load()
+ if err != nil {
+ log.Fatal(err)
+ }
+ cfg := newConfig(arvCfg)
+
+ if *dumpConfig {
+ out, err := yaml.Marshal(cfg)
+ if err != nil {
log.Fatal(err)
}
+ _, err = os.Stdout.Write(out)
+ if err != nil {
+ log.Fatal(err)
+ }
+ return nil
}
- if cfg.deprecatedAllowAnonymous {
- log.Printf("DEPRECATED: Using -allow-anonymous command line flag with ARVADOS_API_TOKEN environment variable. Use config file instead.")
- cfg.AnonymousTokens = []string{os.Getenv("ARVADOS_API_TOKEN")}
- }
+ return cfg
+}
- if *dumpConfig {
- log.Fatal(config.DumpAndExit(cfg))
+func main() {
+ logger := log.New()
+
+ cfg := configure(logger, os.Args)
+ if cfg == nil {
+ return
}
log.Printf("keep-web %s started", version)
- os.Setenv("ARVADOS_API_HOST", cfg.Client.APIHost)
+ os.Setenv("ARVADOS_API_HOST", cfg.cluster.Services.Controller.ExternalURL.Host)
srv := &server{Config: cfg}
if err := srv.Start(); err != nil {
log.Fatal(err)
"context"
"net/http"
+ "git.curoverse.com/arvados.git/sdk/go/arvados"
"git.curoverse.com/arvados.git/sdk/go/ctxlog"
"git.curoverse.com/arvados.git/sdk/go/httpserver"
"github.com/prometheus/client_golang/prometheus"
h.Config.Cache.registry = reg
ctx := ctxlog.Context(context.Background(), logrus.StandardLogger())
mh := httpserver.Instrument(reg, nil, httpserver.HandlerWithContext(ctx, httpserver.AddRequestIDs(httpserver.LogRequests(h))))
- h.MetricsAPI = mh.ServeAPI(h.Config.ManagementToken, http.NotFoundHandler())
+ h.MetricsAPI = mh.ServeAPI(h.Config.cluster.ManagementToken, http.NotFoundHandler())
srv.Handler = mh
- srv.Addr = srv.Config.Listen
+ var listen arvados.URL
+ for listen = range srv.Config.cluster.Services.WebDAV.InternalURLs {
+ break
+ }
+ if len(srv.Config.cluster.Services.WebDAV.InternalURLs) > 1 {
+ logrus.Warn("Services.WebDAV.InternalURLs has more than one key; picked: ", listen)
+ }
+ srv.Addr = listen.Host
return srv.Server.Start()
}
package main
import (
+ "bytes"
"crypto/md5"
"encoding/json"
"fmt"
"strings"
"testing"
+ "git.curoverse.com/arvados.git/lib/config"
"git.curoverse.com/arvados.git/sdk/go/arvados"
"git.curoverse.com/arvados.git/sdk/go/arvadosclient"
"git.curoverse.com/arvados.git/sdk/go/arvadostest"
}
func (s *IntegrationSuite) Test200(c *check.C) {
- s.testServer.Config.AnonymousTokens = []string{arvadostest.AnonymousToken}
+ s.testServer.Config.cluster.Users.AnonymousUserToken = arvadostest.AnonymousToken
for _, spec := range []curlCase{
// My collection
{
}
func (s *IntegrationSuite) TestMetrics(c *check.C) {
- s.testServer.Config.AttachmentOnlyHost = s.testServer.Addr
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = s.testServer.Addr
origin := "http://" + s.testServer.Addr
req, _ := http.NewRequest("GET", origin+"/notfound", nil)
_, err := http.DefaultClient.Do(req)
func (s *IntegrationSuite) SetUpTest(c *check.C) {
arvadostest.ResetEnv()
- cfg := DefaultConfig()
+ ldr := config.NewLoader(bytes.NewBufferString("Clusters: {zzzzz: {}}"), nil)
+ ldr.Path = "-"
+ arvCfg, err := ldr.Load()
+ c.Check(err, check.IsNil)
+ cfg := newConfig(arvCfg)
+ c.Assert(err, check.IsNil)
cfg.Client = arvados.Client{
APIHost: testAPIHost,
Insecure: true,
}
- cfg.Listen = "127.0.0.1:0"
- cfg.ManagementToken = arvadostest.ManagementToken
+ listen := "127.0.0.1:0"
+ cfg.cluster.Services.WebDAV.InternalURLs[arvados.URL{Host: listen}] = arvados.ServiceInstance{}
+ cfg.cluster.Services.WebDAVDownload.InternalURLs[arvados.URL{Host: listen}] = arvados.ServiceInstance{}
+ cfg.cluster.ManagementToken = arvadostest.ManagementToken
+ cfg.cluster.Users.AnonymousUserToken = arvadostest.AnonymousToken
s.testServer = &server{Config: cfg}
- err := s.testServer.Start()
+ err = s.testServer.Start()
c.Assert(err, check.Equals, nil)
}
)
func (s *UnitSuite) TestStatus(c *check.C) {
- h := handler{Config: DefaultConfig()}
+ h := handler{Config: newConfig(s.Config)}
u, _ := url.Parse("http://keep-web.example/status.json")
req := &http.Request{
Method: "GET",
+++ /dev/null
-// Copyright (C) The Arvados Authors. All rights reserved.
-//
-// SPDX-License-Identifier: AGPL-3.0
-
-package main
-
-import (
- "encoding/json"
- "flag"
- "fmt"
- "os"
-)
-
-func usage() {
- c := DefaultConfig()
- c.AnonymousTokens = []string{"xxxxxxxxxxxxxxxxxxxxxxx"}
- c.Client.APIHost = "zzzzz.arvadosapi.com:443"
- exampleConfigFile, err := json.MarshalIndent(c, " ", " ")
- if err != nil {
- panic(err)
- }
- fmt.Fprintf(os.Stderr, `
-
-Keep-web provides read-only HTTP access to files stored in Keep; see
-https://godoc.org/github.com/curoverse/arvados/services/keep-web and
-http://doc.arvados.org/install/install-keep-web.html
-
-Usage: keep-web -config path/to/keep-web.yml
-
-Options:
-`)
- flag.PrintDefaults()
- fmt.Fprintf(os.Stderr, `
-Example config file:
- %s
-
-Client.APIHost:
-
- Address (or address:port) of the Arvados API endpoint.
-
-Client.AuthToken:
-
- Unused. Normally empty, or omitted entirely.
-
-Client.Insecure:
-
- True if your Arvados API endpoint uses an unverifiable SSL/TLS
- certificate.
-
-Listen:
-
- Local port to listen on. Can be "address", "address:port", or
- ":port", where "address" is a host IP address or name and "port"
- is a port number or name.
-
-AnonymousTokens:
-
- Array of tokens to try when a client does not provide a token.
-
-AttachmentOnlyHost:
-
- Accept credentials, and add "Content-Disposition: attachment"
- response headers, for requests at this hostname:port.
-
- This prohibits inline display, which makes it possible to serve
- untrusted and non-public content from a single origin, i.e.,
- without wildcard DNS or SSL.
-
-TrustAllContent:
-
- Serve non-public content from a single origin. Dangerous: read
- docs before using!
-
-Cache.TTL:
-
- Maximum time to cache manifests and permission checks.
-
-Cache.UUIDTTL:
-
- Maximum time to cache collection state.
-
-Cache.MaxCollectionEntries:
-
- Maximum number of collection cache entries.
-
-Cache.MaxCollectionBytes:
-
- Approximate memory limit for collection cache.
-
-Cache.MaxPermissionEntries:
-
- Maximum number of permission cache entries.
-
-Cache.MaxUUIDEntries:
-
- Maximum number of UUID cache entries.
-
-`, exampleConfigFile)
-}
${uuid_prefix}:
ManagementToken: $management_token
Services:
+ RailsAPI:
+ InternalURLs:
+ "http://localhost:${services[api]}": {}
Workbench1:
ExternalURL: "https://$localip:${services[workbench]}"
Workbench2:
GitHTTP:
ExternalURL: "http://$localip:${services[arv-git-httpd]}/"
WebDAV:
+ InternalURLs:
+ "http://localhost:${services[keep-web]}/": {}
+ ExternalURL: "https://$localip:${services[keep-web-ssl]}/"
+ WebDAVDownload:
+ InternalURLs:
+ "http://localhost:${services[keep-web]}/": {}
ExternalURL: "https://$localip:${services[keep-web-ssl]}/"
Composer:
ExternalURL: "http://$localip:${services[composer]}"
Controller:
ExternalURL: "https://$localip:${services[controller-ssl]}"
- NodeProfiles: # to be deprecated in favor of "Services" section
- "*":
- arvados-controller:
- Listen: ":${services[controller]}" # choose a port
- arvados-api-server:
- Listen: ":${services[api]}" # must match Rails server port in your Nginx config
+ InternalURLs:
+ "http://localhost:${services[controller]}": {}
PostgreSQL:
ConnectionPool: 32 # max concurrent connections per arvados server daemon
Connection:
Collections:
BlobSigningKey: $blob_signing_key
DefaultReplication: 1
+ TrustAllContent: true
Login:
ProviderAppSecret: $sso_app_secret
ProviderAppID: arvados-server
AutoSetupNewUsers: true
AutoSetupNewUsersWithVmUUID: $vm_uuid
AutoSetupNewUsersWithRepository: true
+ AnonymousUserToken: $(cat /var/lib/arvados/superuser_token)
Workbench:
SecretKeyBase: $workbench_secret_key_base
ArvadosDocsite: http://$localip:${services[doc]}/
exit
fi
-export ARVADOS_API_HOST=$localip:${services[controller-ssl]}
-export ARVADOS_API_HOST_INSECURE=1
-export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
-
-exec /usr/local/bin/keep-web -trust-all-content -listen=:${services[keep-web]}
+exec /usr/local/bin/keep-web
projects / arvados.git / commitdiff