3 navsection: installguide
4 title: Install the Git server
7 Arvados allows users to create their own private and public git repositories, and clone/push them using SSH and HTTPS.
9 The git hosting setup involves three components.
10 * The "arvados-git-sync.rb" script polls the API server for the current list of repositories, creates bare repositories, and updates the local permission cache used by gitolite.
11 * Gitolite provides SSH access.
12 * arvados-git-http provides HTTPS access.
14 It is not strictly necessary to deploy _both_ SSH and HTTPS access, but we recommend deploying both:
15 * SSH is a more appropriate way to authenticate from a user's workstation because it does not require managing tokens on the client side;
16 * HTTPS is a more appropriate way to authenticate from a shell VM because it does not depend on SSH agent forwarding (SSH clients' agent forwarding features tend to behave as if the remote machine is fully trusted).
18 The HTTPS instructions given below will not work if you skip the SSH setup steps.
22 By convention, we use the following hostname for the git service:
25 <pre><code>git.<span class="userinput">uuid_prefix</span>.your.domain
29 {% include 'notebox_begin' %}
30 Here, we show how to install the git hosting services *on the same host as your API server.* Using a different host is not yet fully supported. On this page we will refer to it as your git server.
31 {% include 'notebox_end' %}
33 DNS and network configuration should be set up so port 443 reaches your HTTPS proxy, and port 22 reaches the OpenSSH service on your git server.
35 h2. Generate an API token
37 On the API server, if you are using RVM:
40 <pre><code>gitserver:~$ <span class="userinput">cd /var/www/arvados-api/current</span>
41 gitserver:/var/www/arvados-api/current$ <span class="userinput">sudo -u www-data RAILS_ENV=production `which rvm-exec` default bundle exec ./script/create_superuser_token.rb</span>
42 zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
46 If you are not using RVM:
49 <pre><code>gitserver:~$ <span class="userinput">cd /var/www/arvados-api/current</span>
50 gitserver:/var/www/arvados-api/current$ <span class="userinput">sudo -u www-data RAILS_ENV=production bundle exec ./script/create_superuser_token.rb</span>
51 zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
55 Copy that token; you'll need it in a minute.
57 h2. Install git and other dependencies
59 On Debian-based systems:
62 <pre><code>gitserver:~$ <span class="userinput">sudo apt-get install git openssh-server</span>
66 On Red Hat-based systems:
69 <pre><code>gitserver:~$ <span class="userinput">sudo yum install git perl-Data-Dumper openssh-server</span>
73 h2. Create a "git" user and a storage directory
75 Gitolite and some additional scripts will be installed in @/var/lib/arvados/git@, which means hosted repository data will be stored in @/var/lib/arvados/git/repositories@. If you choose to install gitolite in a different location, make sure to update the @git_repositories_dir@ entry in your API server's @application.yml@ file accordingly: for example, if you install gitolite at @/data/gitolite@ then your @git_repositories_dir@ will be @/data/gitolite/repositories@.
77 A new UNIX account called "git" will own the files. This makes git URLs look familiar to users (<code>git@[...]:username/reponame.git</code>).
79 On Debian- or Red Hat-based systems:
82 <pre><code>gitserver:~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git</span>
83 gitserver:~$ <span class="userinput">sudo useradd --comment git --home-dir /var/lib/arvados/git git</span>
84 gitserver:~$ <span class="userinput">sudo chown -R git:git ~git</span>
88 The git user needs its own SSH key. (It must be able to run <code>ssh git@localhost</code> from scripts.)
91 <pre><code>gitserver:~$ <span class="userinput">sudo -u git -i bash</span>
92 git@gitserver:~$ <span class="userinput">ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa</span>
93 git@gitserver:~$ <span class="userinput">cp .ssh/id_rsa.pub .ssh/authorized_keys</span>
94 git@gitserver:~$ <span class="userinput">ssh -o stricthostkeychecking=no localhost cat .ssh/id_rsa.pub</span>
95 Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
96 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7aBIDAAgMQN16Pg6eHmvc+D+6TljwCGr4YGUBphSdVb25UyBCeAEgzqRiqy0IjQR2BLtSirXr+1SJAcQfBgI/jwR7FG+YIzJ4ND9JFEfcpq20FvWnMMQ6XD3y3xrZ1/h/RdBNwy4QCqjiXuxDpDB7VNP9/oeAzoATPZGhqjPfNS+RRVEQpC6BzZdsR+S838E53URguBOf9yrPwdHvosZn7VC0akeWQerHqaBIpSfDMtaM4+9s1Gdsz0iP85rtj/6U/K/XOuv2CZsuVZZ52nu3soHnEX2nx2IaXMS3L8Z+lfOXB2T6EaJgXF7Z9ME5K1tx9TSNTRcYCiKztXLNLSbp git@gitserver
97 git@gitserver:~$ <span class="userinput">rm .ssh/authorized_keys</span>
103 Check "https://github.com/sitaramc/gitolite/tags":https://github.com/sitaramc/gitolite/tags for the latest stable version. This guide was tested with @v3.6.3@. _Versions below 3.0 are missing some features needed by Arvados, and should not be used._
105 Download and install the version you selected.
108 <pre><code>git@gitserver:~$ <span class="userinput">echo 'PATH=$HOME/bin:$PATH' >.profile</span>
109 git@gitserver:~$ <span class="userinput">source .profile</span>
110 git@gitserver:~$ <span class="userinput">git clone --branch <b>v3.6.3</b> git://github.com/sitaramc/gitolite</span>
112 Note: checking out '5d24ae666bfd2fa9093d67c840eb8d686992083f'.
114 git@gitserver:~$ <span class="userinput">mkdir bin</span>
115 git@gitserver:~$ <span class="userinput">gitolite/install -ln ~git/bin</span>
116 git@gitserver:~$ <span class="userinput">bin/gitolite setup -pk .ssh/id_rsa.pub</span>
117 Initialized empty Git repository in /var/lib/arvados/git/repositories/gitolite-admin.git/
118 Initialized empty Git repository in /var/lib/arvados/git/repositories/testing.git/
119 WARNING: /var/lib/arvados/git/.ssh/authorized_keys missing; creating a new one
120 (this is normal on a brand new install)
124 _If this didn't go well, more detail about installing gitolite, and information about how it works, can be found on the "gitolite home page":http://gitolite.com/._
126 Clone the gitolite-admin repository. The arvados-git-sync.rb script works by editing the files in this working directory and pushing them to gitolite. Here we make sure "git push" won't produce any errors or warnings.
129 <pre><code>git@gitserver:~$ <span class="userinput">git clone git@localhost:gitolite-admin</span>
130 Cloning into 'gitolite-admin'...
131 remote: Counting objects: 6, done.
132 remote: Compressing objects: 100% (4/4), done.
133 remote: Total 6 (delta 0), reused 0 (delta 0)
134 Receiving objects: 100% (6/6), done.
135 Checking connectivity... done.
136 git@gitserver:~$ <span class="userinput">cd gitolite-admin</span>
137 git@gitserver:~/gitolite-admin$ <span class="userinput">git config user.email arvados</span>
138 git@gitserver:~/gitolite-admin$ <span class="userinput">git config user.name arvados</span>
139 git@gitserver:~/gitolite-admin$ <span class="userinput">git config push.default simple</span>
140 git@gitserver:~/gitolite-admin$ <span class="userinput">git push</span>
141 Everything up-to-date
145 h3. Configure gitolite
147 Configure gitolite to look up a repository name like @username/reponame.git@ and find the appropriate bare repository storage directory.
149 Add the following lines to the top of @~git/.gitolite.rc@:
152 <pre><code><span class="userinput">my $repo_aliases;
153 my $aliases_src = "$ENV{HOME}/.gitolite/arvadosaliases.pl";
154 if ($ENV{HOME} && (-e $aliases_src)) {
155 $repo_aliases = do $aliases_src;
157 $repo_aliases ||= {};
161 Add the following lines inside the section that begins @%RC = (@:
164 <pre><code><span class="userinput"> REPO_ALIASES => $repo_aliases,
168 Inside that section, adjust the 'UMASK' setting to @022@, to ensure the API server has permission to read repositories:
171 <pre><code> UMASK => <span class="userinput">022</span>,
175 Uncomment the 'Alias' line in the section that begins @ENABLE => [@:
178 <pre><code><span class="userinput"> # access a repo by another (possibly legacy) name
183 h2. Configure git synchronization
185 Create a configuration file @/var/www/arvados-api/current/config/arvados-clients.yml@ using the following template, filling in the appropriate values for your system.
186 * For @arvados_api_token@, use the token you generated above.
187 * For @gitolite_arvados_git_user_key@, provide the public key you generated above, i.e., the contents of @~git/.ssh/id_rsa.pub@.
190 <pre><code>production:
191 gitolite_url: /var/lib/arvados/git/repositories/gitolite-admin.git
192 gitolite_tmp: /var/lib/arvados/git
193 arvados_api_host: <span class="userinput">uuid_prefix.example.com</span>
194 arvados_api_token: "<span class="userinput">zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz</span>"
195 arvados_api_host_insecure: <span class="userinput">false</span>
196 gitolite_arvados_git_user_key: "<span class="userinput">ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7aBIDAAgMQN16Pg6eHmvc+D+6TljwCGr4YGUBphSdVb25UyBCeAEgzqRiqy0IjQR2BLtSirXr+1SJAcQfBgI/jwR7FG+YIzJ4ND9JFEfcpq20FvWnMMQ6XD3y3xrZ1/h/RdBNwy4QCqjiXuxDpDB7VNP9/oeAzoATPZGhqjPfNS+RRVEQpC6BzZdsR+S838E53URguBOf9yrPwdHvosZn7VC0akeWQerHqaBIpSfDMtaM4+9s1Gdsz0iP85rtj/6U/K/XOuv2CZsuVZZ52nu3soHnEX2nx2IaXMS3L8Z+lfOXB2T6EaJgXF7Z9ME5K1tx9TSNTRcYCiKztXLNLSbp git@gitserver</span>"
200 h3. Enable the synchronization script
202 The API server package includes a script that retrieves the current set of repository names and permissions from the API, writes them to @arvadosaliases.pl@ in a format usable by gitolite, and triggers gitolite hooks which create new empty repositories if needed. This script should run every 2 to 5 minutes.
204 If you are using RVM, create @/etc/cron.d/arvados-git-sync@ with the following content:
207 <pre><code><span class="userinput">*/5 * * * * git cd /var/www/arvados-api/current && /usr/local/rvm/bin/rvm-exec default bundle exec script/arvados-git-sync.rb production</span>
211 Otherwise, create @/etc/cron.d/arvados-git-sync@ with the following content:
214 <pre><code><span class="userinput">*/5 * * * * git cd /var/www/arvados-api/current && bundle exec script/arvados-git-sync.rb production</span>
218 h3. Configure the API server to advertise the correct SSH URLs
220 In your API server's @application.yml@ file, add the following entry:
223 <pre><code>git_repo_ssh_base: "git@git.<span class="userinput">uuid_prefix.your.domain</span>:"
227 Make sure to include the trailing colon.
229 h2. Install the arvados-git-httpd package
231 This is needed only for HTTPS access.
233 The arvados-git-httpd package provides HTTP access, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support.
235 On Debian-based systems:
238 <pre><code>~$ <span class="userinput">sudo apt-get install git arvados-git-httpd</span>
242 On Red Hat-based systems:
245 <pre><code>~$ <span class="userinput">sudo yum install git arvados-git-httpd</span>
249 Verify that @arvados-git-httpd@ and @git-http-backend@ can be run:
252 <pre><code>~$ <span class="userinput">arvados-git-httpd -h</span>
253 Usage of arvados-git-httpd:
254 -address="0.0.0.0:80": Address to listen on, "host:port".
255 -git-command="/usr/bin/git": Path to git executable. Each authenticated request will execute this program with a single argument, "http-backend".
256 -repo-root="/path/to/cwd": Path to git repositories.
257 ~$ <span class="userinput">git http-backend</span>
258 Status: 500 Internal Server Error
259 Expires: Fri, 01 Jan 1980 00:00:00 GMT
261 Cache-Control: no-cache, max-age=0, must-revalidate
263 fatal: No REQUEST_METHOD from server
267 h3. Enable arvados-git-httpd
269 On Debian-based systems, install runit:
272 <pre><code>~$ <span class="userinput">sudo apt-get install runit</span>
276 On Red Hat-based systems, "install runit from source":http://smarden.org/runit/install.html or use an alternative daemon supervisor.
278 Configure runit to run arvados-git-httpd, making sure to update the API host to match your site:
281 <pre><code>~$ <span class="userinput">cd /etc/sv</span>
282 /etc/sv$ <span class="userinput">sudo mkdir arvados-git-httpd; cd arvados-git-httpd</span>
283 /etc/sv/arvados-git-httpd$ <span class="userinput">sudo mkdir log</span>
284 /etc/sv/arvados-git-httpd$ <span class="userinput">sudo sh -c 'cat >log/run' <<'EOF'
288 exec chpst -u git:git svlogd -tt main
290 /etc/sv/arvados-git-httpd$ <span class="userinput">sudo sh -c 'cat >run' <<'EOF'
292 export ARVADOS_API_HOST=<b>uuid_prefix.your.domain</b>
293 export GITOLITE_HTTP_HOME=/var/lib/arvados/git
294 export PATH="$PATH:/var/lib/arvados/git/bin"
295 exec chpst -u git:git arvados-git-httpd -address=:9001 -git-command="$(which git)" -repo-root=<b>/var/lib/arvados/git/repositories</b> 2>&1
297 /etc/sv/arvados-git-httpd$ <span class="userinput">sudo chmod +x run log/run</span>
301 If you are using a different daemon supervisor, or if you want to test the daemon in a terminal window, an equivalent shell command to run arvados-git-httpd is:
304 <pre><code>sudo -u git \
305 ARVADOS_API_HOST=<span class="userinput">uuid_prefix.your.domain</span> \
306 GITOLITE_HTTP_HOME=/var/lib/arvados/git \
307 PATH="$PATH:/var/lib/arvados/git/bin" \
308 arvados-git-httpd -address=:9001 -git-command="$(which git)" -repo-root=<span class="userinput">/var/lib/arvados/git/repositories</span> 2>&1
312 h3. Set up a reverse proxy to provide SSL service
314 The arvados-git-httpd service will be accessible from anywhere on the internet, so we recommend using SSL.
316 This is best achieved by putting a reverse proxy with SSL support in front of arvados-git-httpd, running on port 443 and passing requests to @arvados-git-httpd@ on port 9001 (or whichever port you used in your run script).
318 Add the following configuration to the @http@ section of your Nginx configuration:
322 upstream arvados-git-httpd {
323 server 127.0.0.1:<span class="userinput">9001</span>;
326 listen <span class="userinput">[your public IP address]</span>:443 ssl;
327 server_name git.<span class="userinput">uuid_prefix.your.domain</span>;
328 proxy_connect_timeout 90s;
329 proxy_read_timeout 300s;
332 ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
333 ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
336 proxy_pass http://arvados-git-httpd;
342 h3. Configure the API server to advertise the correct HTTPS URLs
344 In your API server's @application.yml@ file, add the following entry:
347 <pre><code>git_repo_http_base: https://git.<span class="userinput">uuid_prefix.your.domain</span>/
351 Make sure to include the trailing slash.
355 Restart Nginx to make the Nginx and API server configuration changes take effect.
358 <pre><code>gitserver:~$ <span class="userinput">sudo nginx -s reload</span>