The legacy keepproxy config (loaded from @/etc/arvados/keepproxy/keepproxy.yml@ or a different location specified via -legacy-keepproxy-config command line argument) takes precedence over the centralized config. After you migrate everything from the legacy config to the centralized config, you should delete @/etc/arvados/keepproxy/keepproxy.yml@ and stop using the -legacy-keepproxy-config argument.
+h2(#arv-git-httpd). arv-git-httpd
+
+The legacy arv-git-httpd config (loaded from @/etc/arvados/git-httpd/git-httpd.yml@ or a different location specified via -legacy-git-httpd-config command line argument) takes precedence over the centralized config. After you migrate everything from the legacy config to the centralized config, you should delete @/etc/arvados/git-httpd/git-httpd.yml@ and stop using the -legacy-git-httpd-config argument.
+
+
h2. arvados-controller
Already uses centralized config exclusively. No migration needed.
h3(#master). development master (as of 2019-08-12)
+h4. Arv-git-httpd configuration migration
+
+(feature "#14712":https://dev.arvados.org/issues/14712 ) The arv-git-httpd package can now be configured using the centralized configuration file at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available. Please see "arv-git-httpd's config migration guide":{{site.baseurl}}/admin/config-migration.html#arv-git-httpd for more details.
+
h4. Keep-web dropped support on command line flags configuration
As we're migrating to a central cluster configuration file, the already deprecated way of getting configurations via environment variables and command line flags isn't valid anymore. Current keep-web supports both the now legacy @keep-web.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check "keep-web's install guide":{{site.baseurl}}/install/install-keep-web.html for more details.
h3. Configure the API server to advertise the correct SSH URLs
-In your API server's @application.yml@ file, add the following entry:
+Edit the cluster config at @/etc/arvados/config.yml@ and set @Services.GitSSH.ExternalURL@. Replace @uuid_prefix@ with your cluster id.
<notextile>
-<pre><code>git_repo_ssh_base: "git@git.<span class="userinput">uuid_prefix.your.domain</span>:"
+<pre><code>Clusters:
+ <span class="userinput">uuid_prefix</span>:
+ Services:
+ GitSSH:
+ ExternalURL: <span class="userinput">git@git.uuid_prefix.your.domain:</span>
</code></pre>
</notextile>
{% include 'notebox_end' %}
-Create the configuration file @/etc/arvados/git-httpd/git-httpd.yml@. Run @arvados-git-httpd -h@ to learn more about configuration entries.
+Edit the cluster config at @/etc/arvados/config.yml@ and set the following values. Replace @uuid_prefix@ with your cluster id.
<notextile>
-<pre><code>Client:
- APIHost: <b>uuid_prefix.your.domain</b>
- Insecure: false
-GitCommand: /var/lib/arvados/git/gitolite/src/gitolite-shell
-GitoliteHome: /var/lib/arvados/git
-Listen: :9001
-RepoRoot: /var/lib/arvados/git/repositories
+<pre><code>Clusters:
+ <span class="userinput">uuid_prefix</span>:
+ Services:
+ GitHTTP:
+ ExternalURL: <span class="userinput">https://git.uuid_prefix.your.domain/</span>
+ InternalURLs:
+ <span class="userinput">"http://localhost:9001": {}</span>
+ Git:
+ GitCommand: <span class="userinput">/var/lib/arvados/git/gitolite/src/gitolite-shell</span>
+ GitoliteHome: <span class="userinput">/var/lib/arvados/git</span>
+ Repositories: <span class="userinput">/var/lib/arvados/git/repositories</span>
</code></pre>
</notextile>
+Make sure to include the trailing slash for @Services.GitHTTP.ExternalURL@.
+
Restart the systemd service to ensure the new configuration is used.
+
<notextile>
<pre><code>~$ <span class="userinput">sudo systemctl restart arvados-git-httpd</span>
</code></pre>
</code></pre>
</notextile>
-h3. Configure the API server to advertise the correct HTTPS URLs
-
-In your API server's @application.yml@ file, add the following entry:
-
-<notextile>
-<pre><code>git_repo_https_base: https://git.<span class="userinput">uuid_prefix.your.domain</span>/
-</code></pre>
-</notextile>
-
-Make sure to include the trailing slash.
-
h2. Restart Nginx
Restart Nginx to make the Nginx and API server configuration changes take effect.
h3. Update the cluster config
-Edit the cluster config at @/etc/arvados/config.yml@ and set @Services.Keepproxy.ExternalURL@ and @Services.Keepproxy.InternalURLs@. Replace @zzzzz@ with your cluster id.
+Edit the cluster config at @/etc/arvados/config.yml@ and set @Services.Keepproxy.ExternalURL@ and @Services.Keepproxy.InternalURLs@. Replace @uuid_prefix@ with your cluster id.
<notextile>
<pre><code>Clusters:
<span class="userinput">uuid_prefix</span>:
Services:
- <span class="userinput">Keepproxy:
- ExternalURL: https://keep.uuid_prefix.your.domain
+ Keepproxy:
+ ExternalURL: <span class="userinput">https://keep.uuid_prefix.your.domain</span>
InternalURLs:
- "http://localhost:25107": {}
+ <span class="userinput">"http://localhost:25107": {}</span>
</span></code></pre>
</notextile>
ProviderAppID: ""
Git:
+ # Path to git or gitolite-shell executable. Each authenticated
+ # request will execute this program with the single argument "http-backend"
+ GitCommand: /usr/bin/git
+
+ # Path to Gitolite's home directory. If a non-empty path is given,
+ # the CGI environment will be set up to support the use of
+ # gitolite-shell as a GitCommand: for example, if GitoliteHome is
+ # "/gh", then the CGI environment will have GITOLITE_HTTP_HOME=/gh,
+ # PATH=$PATH:/gh/bin, and GL_BYPASS_ACCESS_CHECKS=1.
+ GitoliteHome: ""
+
# Git repositories must be readable by api server, or you won't be
# able to submit crunch jobs. To pass the test suites, put a clone
# of the arvados tree in {git_repositories_dir}/arvados.git or
cfg.Clusters[cluster.ClusterID] = *cluster
return nil
}
+
+const defaultGitHttpdConfigPath = "/etc/arvados/git-httpd/git-httpd.yml"
+
+type oldGitHttpdConfig struct {
+ Client *arvados.Client
+ Listen string
+ GitCommand string
+ GitoliteHome string
+ RepoRoot string
+ ManagementToken string
+}
+
+func (ldr *Loader) loadOldGitHttpdConfig(cfg *arvados.Config) error {
+ if ldr.GitHttpdPath == "" {
+ return nil
+ }
+ var oc oldGitHttpdConfig
+ err := ldr.loadOldConfigHelper("arv-git-httpd", ldr.GitHttpdPath, &oc)
+ if os.IsNotExist(err) && ldr.GitHttpdPath == defaultGitHttpdConfigPath {
+ return nil
+ } else if err != nil {
+ return err
+ }
+
+ cluster, err := cfg.GetCluster("")
+ if err != nil {
+ return err
+ }
+
+ loadOldClientConfig(cluster, oc.Client)
+
+ cluster.Services.GitHTTP.InternalURLs[arvados.URL{Host: oc.Listen}] = arvados.ServiceInstance{}
+ cluster.TLS.Insecure = oc.Client.Insecure
+ cluster.ManagementToken = oc.ManagementToken
+ cluster.Git.GitCommand = oc.GitCommand
+ cluster.Git.GitoliteHome = oc.GitoliteHome
+ cluster.Git.Repositories = oc.RepoRoot
+
+ cfg.Clusters[cluster.ClusterID] = *cluster
+ return nil
+}
}
`, debugLog, param)
}
+
+func (s *LoadSuite) TestLegacyArvGitHttpdConfig(c *check.C) {
+ content := []byte(`
+{
+ "Client": {
+ "Scheme": "",
+ "APIHost": "example.com",
+ "AuthToken": "abcdefg",
+ },
+ "Listen": ":9000",
+ "GitCommand": "/test/git",
+ "GitoliteHome": "/test/gitolite",
+ "RepoRoot": "/test/reporoot",
+ "ManagementToken": "xyzzy"
+}
+`)
+ f := "-legacy-git-httpd-config"
+ cluster, err := testLoadLegacyConfig(content, f, c)
+
+ c.Check(err, check.IsNil)
+ c.Check(cluster, check.NotNil)
+ c.Check(cluster.Services.Controller.ExternalURL, check.Equals, arvados.URL{Scheme: "https", Host: "example.com"})
+ c.Check(cluster.SystemRootToken, check.Equals, "abcdefg")
+ c.Check(cluster.ManagementToken, check.Equals, "xyzzy")
+ c.Check(cluster.Git.GitCommand, check.Equals, "/test/git")
+ c.Check(cluster.Git.GitoliteHome, check.Equals, "/test/gitolite")
+ c.Check(cluster.Git.Repositories, check.Equals, "/test/reporoot")
+ c.Check(cluster.Services.Keepproxy.InternalURLs[arvados.URL{Host: ":9000"}], check.Equals, arvados.ServiceInstance{})
+}
ProviderAppID: ""
Git:
+ # Path to git or gitolite-shell executable. Each authenticated
+ # request will execute this program with the single argument "http-backend"
+ GitCommand: /usr/bin/git
+
+ # Path to Gitolite's home directory. If a non-empty path is given,
+ # the CGI environment will be set up to support the use of
+ # gitolite-shell as a GitCommand: for example, if GitoliteHome is
+ # "/gh", then the CGI environment will have GITOLITE_HTTP_HOME=/gh,
+ # PATH=$PATH:/gh/bin, and GL_BYPASS_ACCESS_CHECKS=1.
+ GitoliteHome: ""
+
# Git repositories must be readable by api server, or you won't be
# able to submit crunch jobs. To pass the test suites, put a clone
# of the arvados tree in {git_repositories_dir}/arvados.git or
CrunchDispatchSlurmPath string
WebsocketPath string
KeepproxyPath string
+ GitHttpdPath string
configdata []byte
}
flagset.StringVar(&ldr.CrunchDispatchSlurmPath, "legacy-crunch-dispatch-slurm-config", defaultCrunchDispatchSlurmConfigPath, "Legacy crunch-dispatch-slurm configuration `file`")
flagset.StringVar(&ldr.WebsocketPath, "legacy-ws-config", defaultWebsocketConfigPath, "Legacy arvados-ws configuration `file`")
flagset.StringVar(&ldr.KeepproxyPath, "legacy-keepproxy-config", defaultKeepproxyConfigPath, "Legacy keepproxy configuration `file`")
+ flagset.StringVar(&ldr.GitHttpdPath, "legacy-git-httpd-config", defaultGitHttpdConfigPath, "Legacy arv-git-httpd configuration `file`")
flagset.BoolVar(&ldr.SkipLegacy, "skip-legacy", false, "Don't load legacy config files")
}
if legacyConfigArg != "-legacy-keepproxy-config" {
ldr.KeepproxyPath = ""
}
+ if legacyConfigArg != "-legacy-git-httpd-config" {
+ ldr.GitHttpdPath = ""
+ }
return munged
}
ldr.loadOldCrunchDispatchSlurmConfig(&cfg),
ldr.loadOldWebsocketConfig(&cfg),
ldr.loadOldKeepproxyConfig(&cfg),
+ ldr.loadOldGitHttpdConfig(&cfg),
} {
if err != nil {
return nil, err
WebDAVCache WebDAVCacheConfig
}
Git struct {
+ GitCommand string
+ GitoliteHome string
Repositories string
}
Login struct {
from arvados.collection import CollectionReader
HEX_RE = re.compile(r'^[0-9a-fA-F]+$')
+CR_UNCOMMITTED = 'Uncommitted'
+CR_COMMITTED = 'Committed'
+CR_FINAL = 'Final'
keep_locator_pattern = re.compile(r'[0-9a-f]{32}\+\d+(\+\S+)*')
signed_locator_pattern = re.compile(r'[0-9a-f]{32}\+\d+(\+\S+)*\+A\S+(\+\S+)*')
return
stop_arv_git_httpd()
- gitdir = os.path.join(SERVICES_SRC_DIR, 'api', 'tmp', 'git')
gitport = internal_port_from_config("GitHTTP")
env = os.environ.copy()
env.pop('ARVADOS_API_TOKEN', None)
logf = open(_logfilename('arv-git-httpd'), 'a')
- agh = subprocess.Popen(
- ['arv-git-httpd',
- '-repo-root='+gitdir+'/test',
- '-management-token=e687950a23c3a9bceec28c6223a06c79',
- '-address=:'+str(gitport)],
+ agh = subprocess.Popen(['arv-git-httpd'],
env=env, stdin=open('/dev/null'), stdout=logf, stderr=logf)
with open(_pidfile('arv-git-httpd'), 'w') as f:
f.write(str(agh.pid))
},
"Collections": {
"TrustAllContent": True
+ },
+ "Git": {
+ "Repositories": "%s/test" % os.path.join(SERVICES_SRC_DIR, 'api', 'tmp', 'git')
}
}
}
Description=Arvados git server
Documentation=https://doc.arvados.org/
After=network.target
-AssertPathExists=/etc/arvados/git-httpd/git-httpd.yml
# systemd==229 (ubuntu:xenial) obeys StartLimitInterval in the [Unit] section
StartLimitInterval=0
"sync"
"time"
+ "git.curoverse.com/arvados.git/sdk/go/arvados"
"git.curoverse.com/arvados.git/sdk/go/arvadosclient"
"git.curoverse.com/arvados.git/sdk/go/auth"
"git.curoverse.com/arvados.git/sdk/go/httpserver"
type authHandler struct {
handler http.Handler
clientPool *arvadosclient.ClientPool
+ cluster *arvados.Cluster
setupOnce sync.Once
}
func (h *authHandler) setup() {
- ac, err := arvadosclient.New(&theConfig.Client)
+ client, err := arvados.NewClientFromConfig(h.cluster)
if err != nil {
log.Fatal(err)
}
+
+ ac, err := arvadosclient.New(client)
+ if err != nil {
+ log.Fatalf("Error setting up arvados client prototype %v", err)
+ }
+
h.clientPool = &arvadosclient.ClientPool{Prototype: ac}
}
"/" + repoName + "/.git",
}
for _, dir := range tryDirs {
- if fileInfo, err := os.Stat(theConfig.RepoRoot + dir); err != nil {
+ if fileInfo, err := os.Stat(h.cluster.Git.Repositories + dir); err != nil {
if !os.IsNotExist(err) {
statusCode, statusText = http.StatusInternalServerError, err.Error()
return
}
if rewrittenPath == "" {
log.Println("WARNING:", repoUUID,
- "git directory not found in", theConfig.RepoRoot, tryDirs)
+ "git directory not found in", h.cluster.Git.Repositories, tryDirs)
// We say "content not found" to disambiguate from the
// earlier "API says that repo does not exist" error.
statusCode, statusText = http.StatusNotFound, "content not found"
"path/filepath"
"strings"
+ "git.curoverse.com/arvados.git/lib/config"
"git.curoverse.com/arvados.git/sdk/go/arvados"
"git.curoverse.com/arvados.git/sdk/go/arvadostest"
check "gopkg.in/check.v1"
var _ = check.Suite(&AuthHandlerSuite{})
-type AuthHandlerSuite struct{}
+type AuthHandlerSuite struct {
+ cluster *arvados.Cluster
+}
func (s *AuthHandlerSuite) SetUpSuite(c *check.C) {
arvadostest.StartAPI()
arvadostest.ResetEnv()
repoRoot, err := filepath.Abs("../api/tmp/git/test")
c.Assert(err, check.IsNil)
- theConfig = &Config{
- Client: arvados.Client{
- APIHost: arvadostest.APIHost(),
- Insecure: true,
- },
- Listen: ":0",
- GitCommand: "/usr/bin/git",
- RepoRoot: repoRoot,
- ManagementToken: arvadostest.ManagementToken,
- }
+
+ cfg, err := config.NewLoader(nil, nil).Load()
+ c.Assert(err, check.Equals, nil)
+ s.cluster, err = cfg.GetCluster("")
+ c.Assert(err, check.Equals, nil)
+
+ s.cluster.Services.GitHTTP.InternalURLs = map[arvados.URL]arvados.ServiceInstance{arvados.URL{Host: "localhost:0"}: arvados.ServiceInstance{}}
+ s.cluster.TLS.Insecure = true
+ s.cluster.Git.GitCommand = "/usr/bin/git"
+ s.cluster.Git.Repositories = repoRoot
}
func (s *AuthHandlerSuite) TestPermission(c *check.C) {
h := &authHandler{handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
log.Printf("%v", r.URL)
io.WriteString(w, r.URL.Path)
- })}
+ }), cluster: s.cluster}
baseURL, err := url.Parse("http://git.example/")
c.Assert(err, check.IsNil)
for _, trial := range []struct {
}
func (s *AuthHandlerSuite) TestCORS(c *check.C) {
- h := &authHandler{}
+ h := &authHandler{cluster: s.cluster}
// CORS preflight
resp := httptest.NewRecorder()
"net/http"
"net/http/cgi"
"os"
+
+ "git.curoverse.com/arvados.git/sdk/go/arvados"
)
// gitHandler is an http.Handler that invokes git-http-backend (or
cgi.Handler
}
-func newGitHandler() http.Handler {
+func newGitHandler(cluster *arvados.Cluster) http.Handler {
const glBypass = "GL_BYPASS_ACCESS_CHECKS"
const glHome = "GITOLITE_HTTP_HOME"
var env []string
path := os.Getenv("PATH")
- if theConfig.GitoliteHome != "" {
+ if cluster.Git.GitoliteHome != "" {
env = append(env,
- glHome+"="+theConfig.GitoliteHome,
+ glHome+"="+cluster.Git.GitoliteHome,
glBypass+"=1")
- path = path + ":" + theConfig.GitoliteHome + "/bin"
+ path = path + ":" + cluster.Git.GitoliteHome + "/bin"
} else if home, bypass := os.Getenv(glHome), os.Getenv(glBypass); home != "" || bypass != "" {
env = append(env, glHome+"="+home, glBypass+"="+bypass)
log.Printf("DEPRECATED: Passing through %s and %s environment variables. Use GitoliteHome configuration instead.", glHome, glBypass)
}
+
+ var listen arvados.URL
+ for listen = range cluster.Services.GitHTTP.InternalURLs {
+ break
+ }
env = append(env,
- "GIT_PROJECT_ROOT="+theConfig.RepoRoot,
+ "GIT_PROJECT_ROOT="+cluster.Git.Repositories,
"GIT_HTTP_EXPORT_ALL=",
- "SERVER_ADDR="+theConfig.Listen,
+ "SERVER_ADDR="+listen.Host,
"PATH="+path)
return &gitHandler{
Handler: cgi.Handler{
- Path: theConfig.GitCommand,
- Dir: theConfig.RepoRoot,
+ Path: cluster.Git.GitCommand,
+ Dir: cluster.Git.Repositories,
Env: env,
Args: []string{"http-backend"},
},
"net/url"
"regexp"
+ "git.curoverse.com/arvados.git/lib/config"
+ "git.curoverse.com/arvados.git/sdk/go/arvados"
check "gopkg.in/check.v1"
)
var _ = check.Suite(&GitHandlerSuite{})
-type GitHandlerSuite struct{}
+type GitHandlerSuite struct {
+ cluster *arvados.Cluster
+}
-func (s *GitHandlerSuite) TestEnvVars(c *check.C) {
- theConfig = defaultConfig()
- theConfig.RepoRoot = "/"
- theConfig.GitoliteHome = "/test/ghh"
+func (s *GitHandlerSuite) SetUpTest(c *check.C) {
+ cfg, err := config.NewLoader(nil, nil).Load()
+ c.Assert(err, check.Equals, nil)
+ s.cluster, err = cfg.GetCluster("")
+ c.Assert(err, check.Equals, nil)
+ s.cluster.Services.GitHTTP.InternalURLs = map[arvados.URL]arvados.ServiceInstance{arvados.URL{Host: "localhost:80"}: arvados.ServiceInstance{}}
+ s.cluster.Git.GitoliteHome = "/test/ghh"
+ s.cluster.Git.Repositories = "/"
+}
+
+func (s *GitHandlerSuite) TestEnvVars(c *check.C) {
u, err := url.Parse("git.zzzzz.arvadosapi.com/test")
c.Check(err, check.Equals, nil)
resp := httptest.NewRecorder()
URL: u,
RemoteAddr: "[::1]:12345",
}
- h := newGitHandler()
+ h := newGitHandler(s.cluster)
h.(*gitHandler).Path = "/bin/sh"
h.(*gitHandler).Args = []string{"-c", "printf 'Content-Type: text/plain\r\n\r\n'; env"}
c.Check(body, check.Matches, `(?ms).*^GL_BYPASS_ACCESS_CHECKS=1$.*`)
c.Check(body, check.Matches, `(?ms).*^REMOTE_HOST=::1$.*`)
c.Check(body, check.Matches, `(?ms).*^REMOTE_PORT=12345$.*`)
- c.Check(body, check.Matches, `(?ms).*^SERVER_ADDR=`+regexp.QuoteMeta(theConfig.Listen)+`$.*`)
+ c.Check(body, check.Matches, `(?ms).*^SERVER_ADDR=`+regexp.QuoteMeta("localhost:80")+`$.*`)
}
func (s *GitHandlerSuite) TestCGIErrorOnSplitHostPortError(c *check.C) {
URL: u,
RemoteAddr: "test.bad.address.missing.port",
}
- h := newGitHandler()
+ h := newGitHandler(s.cluster)
h.ServeHTTP(resp, req)
c.Check(resp.Code, check.Equals, http.StatusInternalServerError)
c.Check(resp.Body.String(), check.Equals, "")
"os/exec"
"strings"
+ "git.curoverse.com/arvados.git/lib/config"
"git.curoverse.com/arvados.git/sdk/go/arvados"
- "git.curoverse.com/arvados.git/sdk/go/arvadostest"
check "gopkg.in/check.v1"
)
runGitolite("gitolite", "setup", "--admin", "root")
s.tmpRepoRoot = s.gitoliteHome + "/repositories"
- s.Config = &Config{
- Client: arvados.Client{
- APIHost: arvadostest.APIHost(),
- Insecure: true,
- },
- Listen: "localhost:0",
- GitCommand: "/usr/share/gitolite3/gitolite-shell",
- GitoliteHome: s.gitoliteHome,
- RepoRoot: s.tmpRepoRoot,
- }
+
+ cfg, err := config.NewLoader(nil, nil).Load()
+ c.Assert(err, check.Equals, nil)
+ s.cluster, err = cfg.GetCluster("")
+ c.Assert(err, check.Equals, nil)
+
+ s.cluster.Services.GitHTTP.InternalURLs = map[arvados.URL]arvados.ServiceInstance{arvados.URL{Host: "localhost:0"}: arvados.ServiceInstance{}}
+ s.cluster.TLS.Insecure = true
+ s.cluster.Git.GitCommand = "/usr/share/gitolite3/gitolite-shell"
+ s.cluster.Git.GitoliteHome = s.gitoliteHome
+ s.cluster.Git.Repositories = s.tmpRepoRoot
+
s.IntegrationSuite.SetUpTest(c)
// Install the gitolite hooks in the bare repo we made in
"strings"
"testing"
+ "git.curoverse.com/arvados.git/lib/config"
"git.curoverse.com/arvados.git/sdk/go/arvados"
"git.curoverse.com/arvados.git/sdk/go/arvadostest"
check "gopkg.in/check.v1"
tmpRepoRoot string
tmpWorkdir string
testServer *server
- Config *Config
+ cluster *arvados.Cluster
}
func (s *IntegrationSuite) SetUpSuite(c *check.C) {
func (s *IntegrationSuite) SetUpTest(c *check.C) {
arvadostest.ResetEnv()
- s.testServer = &server{}
+
var err error
if s.tmpRepoRoot == "" {
s.tmpRepoRoot, err = ioutil.TempDir("", "arv-git-httpd")
_, err = exec.Command("sh", "-c", "cd "+s.tmpWorkdir+" && echo work >work && git add work && git -c user.name=Foo -c user.email=Foo commit -am 'workdir: test'").CombinedOutput()
c.Assert(err, check.Equals, nil)
+ if s.cluster == nil {
+ cfg, err := config.NewLoader(nil, nil).Load()
+ c.Assert(err, check.Equals, nil)
+ s.cluster, err = cfg.GetCluster("")
+ c.Assert(err, check.Equals, nil)
+
+ s.cluster.Services.GitHTTP.InternalURLs = map[arvados.URL]arvados.ServiceInstance{arvados.URL{Host: "localhost:0"}: arvados.ServiceInstance{}}
+ s.cluster.TLS.Insecure = true
+ s.cluster.Git.GitCommand = "/usr/bin/git"
+ s.cluster.Git.Repositories = s.tmpRepoRoot
+ s.cluster.ManagementToken = arvadostest.ManagementToken
+ }
+
+ s.testServer = &server{cluster: s.cluster}
+ err = s.testServer.Start()
+ c.Assert(err, check.Equals, nil)
+
_, err = exec.Command("git", "config",
"--file", s.tmpWorkdir+"/.git/config",
"credential.http://"+s.testServer.Addr+"/.helper",
"none").Output()
c.Assert(err, check.Equals, nil)
- if s.Config == nil {
- s.Config = &Config{
- Client: arvados.Client{
- APIHost: arvadostest.APIHost(),
- Insecure: true,
- },
- Listen: "localhost:0",
- GitCommand: "/usr/bin/git",
- RepoRoot: s.tmpRepoRoot,
- ManagementToken: arvadostest.ManagementToken,
- }
- }
-
// Clear ARVADOS_API_* env vars before starting up the server,
// to make sure arv-git-httpd doesn't use them or complain
// about them being missing.
os.Unsetenv("ARVADOS_API_HOST")
os.Unsetenv("ARVADOS_API_HOST_INSECURE")
os.Unsetenv("ARVADOS_API_TOKEN")
-
- theConfig = s.Config
- err = s.testServer.Start()
- c.Assert(err, check.Equals, nil)
}
func (s *IntegrationSuite) TearDownTest(c *check.C) {
}
s.tmpWorkdir = ""
- s.Config = nil
-
- theConfig = defaultConfig()
+ s.cluster = nil
}
func (s *IntegrationSuite) RunGit(c *check.C, token, gitCmd, repo string, args ...string) error {
package main
import (
- "encoding/json"
"flag"
"fmt"
- "log"
"os"
- "regexp"
- "git.curoverse.com/arvados.git/sdk/go/arvados"
- "git.curoverse.com/arvados.git/sdk/go/config"
+ "git.curoverse.com/arvados.git/lib/config"
"github.com/coreos/go-systemd/daemon"
+ "github.com/ghodss/yaml"
+ log "github.com/sirupsen/logrus"
)
var version = "dev"
-// Server configuration
-type Config struct {
- Client arvados.Client
- Listen string
- GitCommand string
- RepoRoot string
- GitoliteHome string
- ManagementToken string
-}
-
-var theConfig = defaultConfig()
-
-func defaultConfig() *Config {
- return &Config{
- Listen: ":80",
- GitCommand: "/usr/bin/git",
- RepoRoot: "/var/lib/arvados/git/repositories",
- }
-}
-
func main() {
- const defaultCfgPath = "/etc/arvados/git-httpd/git-httpd.yml"
- const deprecated = " (DEPRECATED -- use config file instead)"
- flag.StringVar(&theConfig.Listen, "address", theConfig.Listen,
- "Address to listen on, \"host:port\" or \":port\"."+deprecated)
- flag.StringVar(&theConfig.GitCommand, "git-command", theConfig.GitCommand,
- "Path to git or gitolite-shell executable. Each authenticated request will execute this program with a single argument, \"http-backend\"."+deprecated)
- flag.StringVar(&theConfig.RepoRoot, "repo-root", theConfig.RepoRoot,
- "Path to git repositories."+deprecated)
- flag.StringVar(&theConfig.GitoliteHome, "gitolite-home", theConfig.GitoliteHome,
- "Value for GITOLITE_HTTP_HOME environment variable. If not empty, GL_BYPASS_ACCESS_CHECKS=1 will also be set."+deprecated)
+ logger := log.New()
+ log.SetFormatter(&log.JSONFormatter{
+ TimestampFormat: "2006-01-02T15:04:05.000000000Z07:00",
+ })
- cfgPath := flag.String("config", defaultCfgPath, "Configuration file `path`.")
- dumpConfig := flag.Bool("dump-config", false, "write current configuration to stdout and exit (useful for migrating from command line flags to config file)")
- getVersion := flag.Bool("version", false, "print version information and exit.")
+ flags := flag.NewFlagSet(os.Args[0], flag.ExitOnError)
+ loader := config.NewLoader(os.Stdin, logger)
+ loader.SetupFlags(flags)
- flag.StringVar(&theConfig.ManagementToken, "management-token", theConfig.ManagementToken,
- "Authorization token to be included in all health check requests.")
+ dumpConfig := flags.Bool("dump-config", false, "write current configuration to stdout and exit (useful for migrating from command line flags to config file)")
+ getVersion := flags.Bool("version", false, "print version information and exit.")
- flag.Usage = usage
- flag.Parse()
+ args := loader.MungeLegacyConfigArgs(logger, os.Args[1:], "-legacy-git-httpd-config")
+ flags.Parse(args)
- // Print version information if requested
if *getVersion {
fmt.Printf("arv-git-httpd %s\n", version)
return
}
- err := config.LoadFile(theConfig, *cfgPath)
+ cfg, err := loader.Load()
if err != nil {
- h := os.Getenv("ARVADOS_API_HOST")
- if h == "" || !os.IsNotExist(err) || *cfgPath != defaultCfgPath {
- log.Fatal(err)
- }
- log.Print("DEPRECATED: No config file found, but ARVADOS_API_HOST environment variable is set. Please use a config file instead.")
- theConfig.Client.APIHost = h
- if regexp.MustCompile("^(?i:1|yes|true)$").MatchString(os.Getenv("ARVADOS_API_HOST_INSECURE")) {
- theConfig.Client.Insecure = true
- }
- if j, err := json.MarshalIndent(theConfig, "", " "); err == nil {
- log.Print("Current configuration:\n", string(j))
- }
+ log.Fatal(err)
+ }
+
+ cluster, err := cfg.GetCluster("")
+ if err != nil {
+ log.Fatal(err)
}
if *dumpConfig {
- log.Fatal(config.DumpAndExit(theConfig))
+ out, err := yaml.Marshal(cfg)
+ if err != nil {
+ log.Fatal(err)
+ }
+ _, err = os.Stdout.Write(out)
+ if err != nil {
+ log.Fatal(err)
+ }
+ return
}
- srv := &server{}
+ srv := &server{cluster: cluster}
if err := srv.Start(); err != nil {
log.Fatal(err)
}
}
log.Printf("arv-git-httpd %s started", version)
log.Println("Listening at", srv.Addr)
- log.Println("Repository root", theConfig.RepoRoot)
+ log.Println("Repository root", cluster.Git.Repositories)
if err := srv.Wait(); err != nil {
log.Fatal(err)
}
import (
"net/http"
+ "git.curoverse.com/arvados.git/sdk/go/arvados"
"git.curoverse.com/arvados.git/sdk/go/health"
"git.curoverse.com/arvados.git/sdk/go/httpserver"
)
type server struct {
httpserver.Server
+ cluster *arvados.Cluster
}
func (srv *server) Start() error {
mux := http.NewServeMux()
- mux.Handle("/", &authHandler{handler: newGitHandler()})
+ mux.Handle("/", &authHandler{handler: newGitHandler(srv.cluster), cluster: srv.cluster})
mux.Handle("/_health/", &health.Handler{
- Token: theConfig.ManagementToken,
+ Token: srv.cluster.ManagementToken,
Prefix: "/_health/",
})
+
+ var listen arvados.URL
+ for listen = range srv.cluster.Services.GitHTTP.InternalURLs {
+ break
+ }
+
srv.Handler = mux
- srv.Addr = theConfig.Listen
+ srv.Addr = listen.Host
return srv.Server.Start()
}
+++ /dev/null
-// Copyright (C) The Arvados Authors. All rights reserved.
-//
-// SPDX-License-Identifier: AGPL-3.0
-
-// arvados-git-httpd provides authenticated access to Arvados-hosted
-// git repositories.
-//
-// See http://doc.arvados.org/install/install-arv-git-httpd.html.
-package main
-
-import (
- "flag"
- "fmt"
- "os"
-
- "github.com/ghodss/yaml"
-)
-
-func usage() {
- c := defaultConfig()
- c.Client.APIHost = "zzzzz.arvadosapi.com:443"
- exampleConfigFile, err := yaml.Marshal(c)
- if err != nil {
- panic(err)
- }
- fmt.Fprintf(os.Stderr, `
-
-arvados-git-httpd provides authenticated access to Arvados-hosted git
-repositories.
-
-See http://doc.arvados.org/install/install-arv-git-httpd.html.
-
-Usage: arvados-git-httpd [-config path/to/arvados/git-httpd.yml]
-
-Options:
-`)
- flag.PrintDefaults()
- fmt.Fprintf(os.Stderr, `
-Example config file:
-
-%s
-
-Client.APIHost:
-
- Address (or address:port) of the Arvados API endpoint.
-
-Client.AuthToken:
-
- Unused. Normally empty, or omitted entirely.
-
-Client.Insecure:
-
- True if your Arvados API endpoint uses an unverifiable SSL/TLS
- certificate.
-
-GitCommand:
-
- Path to git or gitolite-shell executable. Each authenticated
- request will execute this program with the single argument
- "http-backend".
-
-GitoliteHome:
-
- Path to Gitolite's home directory. If a non-empty path is given,
- the CGI environment will be set up to support the use of
- gitolite-shell as a GitCommand: for example, if GitoliteHome is
- "/gh", then the CGI environment will have GITOLITE_HTTP_HOME=/gh,
- PATH=$PATH:/gh/bin, and GL_BYPASS_ACCESS_CHECKS=1.
-
-Listen:
-
- Local port to listen on. Can be "address:port" or ":port", where
- "address" is a host IP address or name and "port" is a port number
- or name.
-
-RepoRoot:
-
- Path to git repositories.
-
-`, exampleConfigFile)
-}
GitSSH:
ExternalURL: "ssh://git@$localip:"
GitHTTP:
- ExternalURL: "http://$localip:${services[arv-git-httpd]}/"
+ InternalURLs:
+ "http://localhost:${services[arv-git-httpd]}/": {}
+ ExternalURL: "https://$localip:${services[arv-git-httpd-ssl]}/"
WebDAV:
InternalURLs:
"http://localhost:${services[keep-web]}/": {}
Workbench:
SecretKeyBase: $workbench_secret_key_base
ArvadosDocsite: http://$localip:${services[doc]}/
+ Git:
+ GitCommand: /usr/share/gitolite3/gitolite-shell
+ GitoliteHome: /var/lib/arvados/git
+ Repositories: /var/lib/arvados/git/repositories
EOF
/usr/local/lib/arvbox/yml_override.py /var/lib/arvados/cluster_config.yml
[controller-ssl]=8000
[sso]=8900
[composer]=4200
+ [arv-git-httpd-ssl]=9000
[arv-git-httpd]=9001
[keep-web]=9003
[keep-web-ssl]=9002
export PATH="$PATH:/var/lib/arvados/git/bin"
cd ~git
-exec /usr/local/bin/arv-git-httpd \
- -address=:${services[arv-git-httpd]} \
- -git-command=/usr/share/gitolite3/gitolite-shell \
- -gitolite-home=/var/lib/arvados/git \
- -repo-root=/var/lib/arvados/git/repositories
+exec /usr/local/bin/arv-git-httpd
}
}
+ upstream arvados-git-httpd {
+ server localhost:${services[arv-git-httpd]};
+ }
+ server {
+ listen *:${services[arv-git-httpd-ssl]} ssl default_server;
+ server_name arvados-git-httpd;
+ proxy_connect_timeout 90s;
+ proxy_read_timeout 300s;
+
+ ssl on;
+ ssl_certificate "${server_cert}";
+ ssl_certificate_key "${server_cert_key}";
+ client_max_body_size 50m;
+
+ location / {
+ proxy_pass http://arvados-git-httpd;
+ proxy_set_header Host \$http_host;
+ proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+ proxy_set_header X-Forwarded-Proto https;
+ proxy_redirect off;
+ }
+ }
+
}
EOF
projects / arvados.git / commitdiff