X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/db36b8ad5184c46c0bcbb42a384f298a6c45dee5..51169558811743fcd2e21e4b9397f9fb2d4766b0:/doc/install/install-arv-git-httpd.html.textile.liquid diff --git a/doc/install/install-arv-git-httpd.html.textile.liquid b/doc/install/install-arv-git-httpd.html.textile.liquid index 85f643f1f8..1c31dc4d6e 100644 --- a/doc/install/install-arv-git-httpd.html.textile.liquid +++ b/doc/install/install-arv-git-httpd.html.textile.liquid @@ -1,39 +1,260 @@ --- layout: default navsection: installguide -title: Install Git server +title: Install the Git server ... -This installation guide assumes you are on a 64 bit Debian or Ubuntu system. +Arvados allows users to create their own private and public git repositories, and clone/push them using SSH and HTTPS. -The arv-git-httpd server provides HTTP access to hosted git repositories, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support. +The git hosting setup involves three components. +* The "arvados-git-sync.rb" script polls the API server for the current list of repositories, creates bare repositories, and updates the local permission cache used by gitolite. +* Gitolite provides SSH access. +* arvados-git-http provides HTTPS access. + +It is not strictly necessary to deploy _both_ SSH and HTTPS access, but we recommend deploying both: +* SSH is a more appropriate way to authenticate from a user's workstation because it does not require managing tokens on the client side; +* HTTPS is a more appropriate way to authenticate from a shell VM because it does not depend on SSH agent forwarding (SSH clients' agent forwarding features tend to behave as if the remote machine is fully trusted). + +The HTTPS instructions given below will not work if you skip the SSH setup steps. + +h2. Set up DNS By convention, we use the following hostname for the git service: -
-table(table table-bordered table-condensed). -|git.@uuid_prefix@.your.domain| -
+ +
git.uuid_prefix.your.domain
+
+ + +{% include 'notebox_begin' %} +Here, we show how to install the git hosting services *on the same host as your API server.* Using a different host is not yet fully supported. On this page we will refer to it as your git server. +{% include 'notebox_end' %} + +DNS and network configuration should be set up so port 443 reaches your HTTPS proxy, and port 22 reaches the OpenSSH service on your git server. + +h2. Generate an API token + +Use the following command to generate an API token, changing *@webserver-user@* to the user of the web server process. This is typically *@www-data@* on Debian systems by default, other systems may use different defaults such the name of the web server software (for example, *@nginx@*). + +Using RVM: + + +
gitserver:~$ cd /var/www/arvados-api/current
+gitserver:/var/www/arvados-api/current$ sudo -u webserver-user RAILS_ENV=production `which rvm-exec` default bundle exec ./script/create_superuser_token.rb
+zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
+
+ + +Not using RVM: + + +
gitserver:~$ cd /var/www/arvados-api/current
+gitserver:/var/www/arvados-api/current$ sudo -u webserver-user RAILS_ENV=production bundle exec ./script/create_superuser_token.rb
+zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
+
+ + +Copy that token; you'll need it in a minute. + +h2. Install git and other dependencies + +On Debian-based systems: + + +
gitserver:~$ sudo apt-get install git openssh-server
+
+ + +On Red Hat-based systems: + + +
gitserver:~$ sudo yum install git perl-Data-Dumper openssh-server
+
+ + +{% include 'install_git' %} + +h2. Create a "git" user and a storage directory + +Gitolite and some additional scripts will be installed in @/var/lib/arvados/git@, which means hosted repository data will be stored in @/var/lib/arvados/git/repositories@. If you choose to install gitolite in a different location, make sure to update the @git_repositories_dir@ entry in your API server's @application.yml@ file accordingly: for example, if you install gitolite at @/data/gitolite@ then your @git_repositories_dir@ will be @/data/gitolite/repositories@. + +A new UNIX account called "git" will own the files. This makes git URLs look familiar to users (git@[...]:username/reponame.git). + +On Debian- or Red Hat-based systems: + + +
gitserver:~$ sudo mkdir -p /var/lib/arvados/git
+gitserver:~$ sudo useradd --comment git --home-dir /var/lib/arvados/git git
+gitserver:~$ sudo chown -R git:git ~git
+
+ + +The git user needs its own SSH key. (It must be able to run ssh git@localhost from scripts.) + + +
gitserver:~$ sudo -u git -i bash
+git@gitserver:~$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
+git@gitserver:~$ cp .ssh/id_rsa.pub .ssh/authorized_keys
+git@gitserver:~$ ssh -o stricthostkeychecking=no localhost cat .ssh/id_rsa.pub
+Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7aBIDAAgMQN16Pg6eHmvc+D+6TljwCGr4YGUBphSdVb25UyBCeAEgzqRiqy0IjQR2BLtSirXr+1SJAcQfBgI/jwR7FG+YIzJ4ND9JFEfcpq20FvWnMMQ6XD3y3xrZ1/h/RdBNwy4QCqjiXuxDpDB7VNP9/oeAzoATPZGhqjPfNS+RRVEQpC6BzZdsR+S838E53URguBOf9yrPwdHvosZn7VC0akeWQerHqaBIpSfDMtaM4+9s1Gdsz0iP85rtj/6U/K/XOuv2CZsuVZZ52nu3soHnEX2nx2IaXMS3L8Z+lfOXB2T6EaJgXF7Z9ME5K1tx9TSNTRcYCiKztXLNLSbp git@gitserver
+git@gitserver:~$ rm .ssh/authorized_keys
+
+ + +h2. Install gitolite + +Check "https://github.com/sitaramc/gitolite/tags":https://github.com/sitaramc/gitolite/tags for the latest stable version. This guide was tested with @v3.6.3@. _Versions below 3.0 are missing some features needed by Arvados, and should not be used._ + +Download and install the version you selected. + + +
git@gitserver:~$ echo 'PATH=$HOME/bin:$PATH' >.profile
+git@gitserver:~$ source .profile
+git@gitserver:~$ git clone --branch v3.6.3 git://github.com/sitaramc/gitolite
+...
+Note: checking out '5d24ae666bfd2fa9093d67c840eb8d686992083f'.
+...
+git@gitserver:~$ mkdir bin
+git@gitserver:~$ gitolite/install -ln ~git/bin
+git@gitserver:~$ bin/gitolite setup -pk .ssh/id_rsa.pub
+Initialized empty Git repository in /var/lib/arvados/git/repositories/gitolite-admin.git/
+Initialized empty Git repository in /var/lib/arvados/git/repositories/testing.git/
+WARNING: /var/lib/arvados/git/.ssh/authorized_keys missing; creating a new one
+    (this is normal on a brand new install)
+
+ + +_If this didn't go well, more detail about installing gitolite, and information about how it works, can be found on the "gitolite home page":http://gitolite.com/._ + +Clone the gitolite-admin repository. The arvados-git-sync.rb script works by editing the files in this working directory and pushing them to gitolite. Here we make sure "git push" won't produce any errors or warnings. + + +
git@gitserver:~$ git clone git@localhost:gitolite-admin
+Cloning into 'gitolite-admin'...
+remote: Counting objects: 6, done.
+remote: Compressing objects: 100% (4/4), done.
+remote: Total 6 (delta 0), reused 0 (delta 0)
+Receiving objects: 100% (6/6), done.
+Checking connectivity... done.
+git@gitserver:~$ cd gitolite-admin
+git@gitserver:~/gitolite-admin$ git config user.email arvados
+git@gitserver:~/gitolite-admin$ git config user.name arvados
+git@gitserver:~/gitolite-admin$ git config push.default simple
+git@gitserver:~/gitolite-admin$ git push
+Everything up-to-date
+
+ + +h3. Configure gitolite + +Configure gitolite to look up a repository name like @username/reponame.git@ and find the appropriate bare repository storage directory. + +Add the following lines to the top of @~git/.gitolite.rc@: + + +
my $repo_aliases;
+my $aliases_src = "$ENV{HOME}/.gitolite/arvadosaliases.pl";
+if ($ENV{HOME} && (-e $aliases_src)) {
+    $repo_aliases = do $aliases_src;
+}
+$repo_aliases ||= {};
+
+ + +Add the following lines inside the section that begins @%RC = (@: -This hostname should resolve from anywhere on the internet. + +
    REPO_ALIASES => $repo_aliases,
+
+ + +Inside that section, adjust the 'UMASK' setting to @022@, to ensure the API server has permission to read repositories: + + +
    UMASK => 022,
+
+ + +Uncomment the 'Alias' line in the section that begins @ENABLE => [@: + + +
            # access a repo by another (possibly legacy) name
+            'Alias',
+
+ + +h2. Configure git synchronization + +Create a configuration file @/var/www/arvados-api/current/config/arvados-clients.yml@ using the following template, filling in the appropriate values for your system. +* For @arvados_api_token@, use the token you generated above. +* For @gitolite_arvados_git_user_key@, provide the public key you generated above, i.e., the contents of @~git/.ssh/id_rsa.pub@. + + +
production:
+  gitolite_url: /var/lib/arvados/git/repositories/gitolite-admin.git
+  gitolite_tmp: /var/lib/arvados/git
+  arvados_api_host: uuid_prefix.example.com
+  arvados_api_token: "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"
+  arvados_api_host_insecure: false
+  gitolite_arvados_git_user_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7aBIDAAgMQN16Pg6eHmvc+D+6TljwCGr4YGUBphSdVb25UyBCeAEgzqRiqy0IjQR2BLtSirXr+1SJAcQfBgI/jwR7FG+YIzJ4ND9JFEfcpq20FvWnMMQ6XD3y3xrZ1/h/RdBNwy4QCqjiXuxDpDB7VNP9/oeAzoATPZGhqjPfNS+RRVEQpC6BzZdsR+S838E53URguBOf9yrPwdHvosZn7VC0akeWQerHqaBIpSfDMtaM4+9s1Gdsz0iP85rtj/6U/K/XOuv2CZsuVZZ52nu3soHnEX2nx2IaXMS3L8Z+lfOXB2T6EaJgXF7Z9ME5K1tx9TSNTRcYCiKztXLNLSbp git@gitserver"
+
+ + +h3. Enable the synchronization script + +The API server package includes a script that retrieves the current set of repository names and permissions from the API, writes them to @arvadosaliases.pl@ in a format usable by gitolite, and triggers gitolite hooks which create new empty repositories if needed. This script should run every 2 to 5 minutes. + +If you are using RVM, create @/etc/cron.d/arvados-git-sync@ with the following content: + + +
*/5 * * * * git cd /var/www/arvados-api/current && /usr/local/rvm/bin/rvm-exec default bundle exec script/arvados-git-sync.rb production
+
+ + +Otherwise, create @/etc/cron.d/arvados-git-sync@ with the following content: + + +
*/5 * * * * git cd /var/www/arvados-api/current && bundle exec script/arvados-git-sync.rb production
+
+ + +h3. Configure the API server to advertise the correct SSH URLs + +In your API server's @application.yml@ file, add the following entry: + + +
git_repo_ssh_base: "git@git.uuid_prefix.your.domain:"
+
+ + +Make sure to include the trailing colon. + +h2. Install the arvados-git-httpd package + +This is needed only for HTTPS access. + +The arvados-git-httpd package provides HTTP access, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support. -h2. Install arv-git-httpd +On Debian-based systems: + + +
~$ sudo apt-get install git arvados-git-httpd
+
+ -First add the Arvados apt repository, and then install the arv-git-httpd package. +On Red Hat-based systems: -
~$ echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
-~$ sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
-~$ sudo /usr/bin/apt-get update
-~$ sudo /usr/bin/apt-get install arv-git-httpd
+
~$ sudo yum install git arvados-git-httpd
 

 
 
-Verify that @arv-git-httpd@ and @git-http-backend@ are functional:
+Verify that @arvados-git-httpd@ and @git-http-backend@ can be run:
 
 
-
~$ arv-git-httpd -h
-Usage of arv-git-httpd:
+
~$ arvados-git-httpd -h
+Usage of arvados-git-httpd:
   -address="0.0.0.0:80": Address to listen on, "host:port".
   -git-command="/usr/bin/git": Path to git executable. Each authenticated request will execute this program with a single argument, "http-backend".
   -repo-root="/path/to/cwd": Path to git repositories.
@@ -47,27 +268,99 @@ fatal: No REQUEST_METHOD from server
 

 
 
-We recommend running @arv-git-httpd@ under "runit":https://packages.debian.org/search?keywords=runit or something similar.
+h3. Enable arvados-git-httpd
+
+On Debian-based systems, install runit:
+
+
+
~$ sudo apt-get install runit
+

+
+
+On Red Hat-based systems, "install runit from source":http://smarden.org/runit/install.html or use an alternative daemon supervisor.
+
+Configure runit to run arvados-git-httpd, making sure to update the API host to match your site:
+
+
+
~$ cd /etc/sv
+/etc/sv$ sudo mkdir arvados-git-httpd; cd arvados-git-httpd
+/etc/sv/arvados-git-httpd$ sudo mkdir log
+/etc/sv/arvados-git-httpd$ sudo sh -c 'cat >log/run' <<'EOF'
+#!/bin/sh
+mkdir -p main
+chown git:git main
+exec chpst -u git:git svlogd -tt main
+EOF
+/etc/sv/arvados-git-httpd$ sudo sh -c 'cat >run' <<'EOF'
+#!/bin/sh
+export ARVADOS_API_HOST=uuid_prefix.your.domain
+export GITOLITE_HTTP_HOME=/var/lib/arvados/git
+export GL_BYPASS_ACCESS_CHECKS=1
+export PATH="$PATH:/var/lib/arvados/git/bin"
+exec chpst -u git:git arvados-git-httpd -address=:9001 -git-command=/var/lib/arvados/git/gitolite/src/gitolite-shell -repo-root=/var/lib/arvados/git/repositories 2>&1
+EOF
+/etc/sv/arvados-git-httpd$ sudo chmod +x run log/run
+

+
+
+If you are using a different daemon supervisor, or if you want to test the daemon in a terminal window, an equivalent shell command to run arvados-git-httpd is:
+
+
+
sudo -u git \
+  ARVADOS_API_HOST=uuid_prefix.your.domain \
+  GITOLITE_HTTP_HOME=/var/lib/arvados/git \
+  GL_BYPASS_ACCESS_CHECKS=1 \
+  PATH="$PATH:/var/lib/arvados/git/bin" \
+  arvados-git-httpd -address=:9001 -git-command=/var/lib/arvados/git/gitolite/src/gitolite-shell -repo-root=/var/lib/arvados/git/repositories 2>&1
+

+
+
+h3. Set up a reverse proxy to provide SSL service
 
-Your @run@ script should look something like this:
+The arvados-git-httpd service will be accessible from anywhere on the internet, so we recommend using SSL.
+
+This is best achieved by putting a reverse proxy with SSL support in front of arvados-git-httpd, running on port 443 and passing requests to @arvados-git-httpd@ on port 9001 (or whichever port you used in your run script).
+
+Add the following configuration to the @http@ section of your Nginx configuration:
 
 
-
export ARVADOS_API_HOST=uuid_prefix.your.domain
-exec sudo -u git arv-git-httpd -address=:9001 -git-command="$(which git)" -repo-root=/var/lib/arvados/git 2>&1
+

+upstream arvados-git-httpd {
+  server                  127.0.0.1:9001;
+}
+server {
+  listen                  [your public IP address]:443 ssl;
+  server_name             git.uuid_prefix.your.domain;
+  proxy_connect_timeout   90s;
+  proxy_read_timeout      300s;
+
+  ssl on;
+  ssl_certificate         /YOUR/PATH/TO/cert.pem;
+  ssl_certificate_key     /YOUR/PATH/TO/cert.key;
+
+  location  / {
+    proxy_pass            http://arvados-git-httpd;
+  }
+}
 

 
 
-h3. Set up a reverse proxy with SSL support
+h3. Configure the API server to advertise the correct HTTPS URLs
 
-The arv-git-httpd service will be accessible from anywhere on the internet, so we recommend using SSL for transport encryption.
+In your API server's @application.yml@ file, add the following entry:
+
+
+
git_repo_http_base: https://git.uuid_prefix.your.domain/
+

+
 
-This is best achieved by putting a reverse proxy with SSL support in front of arv-git-httpd, running on port 443 and passing requests to arv-git-httpd on port 9001 (or whatever port you chose in your run script).
+Make sure to include the trailing slash.
 
-h3. Tell the API server about the arv-git-httpd service
+h2. Restart Nginx
 
-In your API server's config/application.yml file, add the following entry:
+Restart Nginx to make the Nginx and API server configuration changes take effect.
 
 
-
git_http_base: git.uuid_prefix.your.domain
+
gitserver:~$ sudo nginx -s reload