## Quick start
-Curoverse maintains an Arvados public cloud demo at
-[https://cloud.curoverse.com](https://cloud.curoverse.com). A Google account
-is required to log in.
+Veritas Genetics maintains a public installation of Arvados for evaluation and trial use, the [Arvados Playground](https://playground.arvados.org). A Google account is required to log in.
To try out Arvados on your local workstation, you can use Arvbox, which
provides Arvados components pre-installed in a Docker container (requires
+++ /dev/null
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-<div class="alert alert-block alert-info">
- <button type="button" class="close" data-dismiss="alert">×</button>
- <h4>Note!</h4>
- <p>The SSO server codebase currently uses OpenID 2.0 to talk to Google's authentication service. Google <a href="https://developers.google.com/accounts/docs/OpenID2">has deprecated that protocol</a>. This means that new clients will not be allowed to talk to Google's authentication services anymore over OpenID 2.0, and they will phase out the use of OpenID 2.0 completely in the coming monts. We are working on upgrading the SSO server codebase to a newer protocol. That work should be complete by the end of November 2014. In the mean time, anyone is free to use the existing Curoverse SSO server for any local Arvados installation. Instructions to do so are provided on the "API server":install-api-server.html page.</p>
- <p><strong>Recommendation: skip this step</strong></p>
-</div>
---
layout: default
navsection: installguide
-title: Copy pipeline from Curoverse cloud
+title: Copy pipeline from the Arvados Playground
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This tutorial describes how to find and copy a publicly shared pipeline from Curoverse cloud. Please note that you can use similar steps to copy any template you can access from Curoverse cloud to your cluster.
+This tutorial describes how to find and copy a publicly shared pipeline from the Arvados Playground. Please note that you can use similar steps to copy any template you can access from the Arvados Playground to your cluster.
-h3. Access a public pipeline in Curoverse cloud using Workbench
+h3. Access a public pipeline in the Arvados Playground using Workbench
-Curoverse cloud provides access to some public data, which can be used to experience Arvados in action. Let's access a public pipeline and copy it to your cluster, so that you can run it in your environment.
+the Arvados Playground provides access to some public data, which can be used to experience Arvados in action. Let's access a public pipeline and copy it to your cluster, so that you can run it in your environment.
-Start by visiting the "*Curoverse public projects page*":https://cloud.curoverse.com/projects/public. This page lists all the publicly accessible projects in this arvados installation. Click on one of these projects to open it. We will use "*lobSTR v.3 (Public)*":https://cloud.curoverse.com/projects/qr1hi-j7d0g-up6qgpqz5ie2vfq as the example in this tutorial.
+Start by visiting the "*Arvados Playground public projects page*":https://playground.arvados.org/projects/public. This page lists all the publicly accessible projects in this arvados installation. Click on one of these projects to open it. We will use "*lobSTR v.3 (Public)*":https://playground.arvados.org/projects/qr1hi-j7d0g-up6qgpqz5ie2vfq as the example in this tutorial.
-Once in the "*lobSTR v.3 (Public)*":https://cloud.curoverse.com/projects/qr1hi-j7d0g-up6qgpqz5ie2vfq project, click on the *Pipeline templates* tab. In the pipeline templates tab, you will see a template named *lobSTR v.3*. Click on the <span class="fa fa-lg fa-gears"></span> *Show* button to the left of this name. This will take to you to the "*lobSTR v.3*":https://cloud.curoverse.com/pipeline_templates/qr1hi-p5p6p-9pkaxt6qjnkxhhu template page.
+Once in the "*lobSTR v.3 (Public)*":https://playground.arvados.org/projects/qr1hi-j7d0g-up6qgpqz5ie2vfq project, click on the *Pipeline templates* tab. In the pipeline templates tab, you will see a template named *lobSTR v.3*. Click on the <span class="fa fa-lg fa-gears"></span> *Show* button to the left of this name. This will take to you to the "*lobSTR v.3*":https://playground.arvados.org/pipeline_templates/qr1hi-p5p6p-9pkaxt6qjnkxhhu template page.
Once in this page, you can take the *uuid* of this template from the address bar, which is *qr1hi-p5p6p-9pkaxt6qjnkxhhu*. Next, we will copy this template to your Arvados instance.
-h3. Copying a pipeline template from Curoverse cloud to your cluster
+h3. Copying a pipeline template from the Arvados Playground to your cluster
-As described above, navigate to the publicly shared pipeline template "*lobSTR v.3*":https://cloud.curoverse.com/pipeline_templates/qr1hi-p5p6p-9pkaxt6qjnkxhhu using Curoverse Workbench. We will now copy this template with uuid *qr1hi-p5p6p-9pkaxt6qjnkxhhu* to your cluster.
+As described above, navigate to the publicly shared pipeline template "*lobSTR v.3*":https://playground.arvados.org/pipeline_templates/qr1hi-p5p6p-9pkaxt6qjnkxhhu on the Arvados Playground. We will now copy this template with uuid *qr1hi-p5p6p-9pkaxt6qjnkxhhu* to your cluster.
{% include 'tutorial_expectations' %}
We will use the Arvados *arv-copy* command to copy this template to your cluster. In order to use arv-copy, first you need to setup the source and destination cluster configuration files. Here, *qr1hi* would be the source cluster and your Arvados instance would be the *dst_cluster*.
-During this setup, if you have an account in Curoverse cloud, you can use "your access token":#using-your-token to create the source configuration file. If you do not have an account in Curoverse cloud, you can use the "anonymous access token":#using-anonymous-token for the source cluster configuration.
+During this setup, if you have an account in the Arvados Playground, you can use "your access token":#using-your-token to create the source configuration file. If you do not have an account in the Arvados Playground, you can use the "anonymous access token":#using-anonymous-token for the source cluster configuration.
h4(#using-anonymous-token). *Configuring source and destination setup files using anonymous access token*
h4(#using-your-token). *Configuring source and destination setup files using personal access token*
-If you already have an account in Curoverse cloud, you can follow the instructions in the "*Using arv-copy*":http://doc.arvados.org/user/topics/arv-copy.html user guide to get your *Current token* for source and destination clusters, and use them to create the source *qr1hi.conf* and dst_cluster.conf configuration files.
+If you already have an account in the Arvados Playground, you can follow the instructions in the "*Using arv-copy*":http://doc.arvados.org/user/topics/arv-copy.html user guide to get your *Current token* for source and destination clusters, and use them to create the source *qr1hi.conf* and dst_cluster.conf configuration files.
You can now copy the pipeline template from *qr1hi* to *your cluster* with or without recursion. Replace *dst_cluster* with the *uuid_prefix* of your cluster.
|"Arvados-in-a-box":arvbox.html (arvbox)|Easy|no|yes|no|yes|yes|
|"Arvados on Kubernetes":arvados-on-kubernetes.html|Easy ^1^|yes|yes ^2^|no ^2^|no|yes|
|"Manual installation":install-manual-prerequisites.html|Complicated|yes|yes|yes|no|no|
-|"Cloud demo":https://cloud.curoverse.com by Veritas Genetics|N/A ^3^|yes|yes|no|no|no|
-|"Cluster Operation Subscription":https://curoverse.com/products by Veritas Genetics|N/A ^3^|yes|yes|yes|yes|yes|
+|"Arvados Playground":https://playground.arvados.org hosted by Veritas Genetics|N/A ^3^|yes|yes|no|no|no|
+|"Cluster Operation Subscription":https://curoverse.com/products supported by Veritas Genetics|N/A ^3^|yes|yes|yes|yes|yes|
</div>
* ^1^ Assumes a Kubernetes cluster is available
h2. LobSTR v3
-In this quickstart guide, we'll run an existing pipeline with pre-existing data. Step-by-step instructions are shown below. You can follow along using your own local install or by using Curoverse's <a href="http://lp.curoverse.com/beta-signup/">hosted version of Arvados</a> (in public beta, any Google account can be used to login).
+In this quickstart guide, we'll run an existing pipeline with pre-existing data. Step-by-step instructions are shown below. You can follow along using your own local install or by using the <a href="https://playground.arvados.org/">Arvados Playground</a> (any Google account can be used to log in).
(For more information about this pipeline, see our <a href="https://dev.arvados.org/projects/arvados/wiki/LobSTR_tutorial">detailed lobSTR guide</a>).
</code></pre>
</notextile>
-The tutorial data is hosted on "https://cloud.curoverse.com":https://cloud.curoverse.com (also referred to by the identifier *qr1hi*). If you are using a different Arvados instance, you may need to copy the data to your own instance. The easiest way to do this is with "arv-copy":{{site.baseurl}}/user/topics/arv-copy.html (this requires signing up for a free cloud.curoverse.com account).
+The tutorial data is hosted on "https://playground.arvados.org":https://playground.arvados.org (also referred to by the identifier *qr1hi*). If you are using a different Arvados instance, you may need to copy the data to your own instance. The easiest way to do this is with "arv-copy":{{site.baseurl}}/user/topics/arv-copy.html (this requires signing up for a free playground.arvados.org account).
<notextile>
<pre><code>~$ <span class="userinput">arv-copy --src qr1hi --dst settings 2463fa9efeb75e099685528b3b9071e0+438</span>
</code></pre>
</notextile>
-If you do not wish to create an account on "https://cloud.curoverse.com":https://cloud.curoverse.com, you may download the files anonymously and upload them to your local Arvados instance:
+If you do not wish to create an account on "https://playground.arvados.org":https://playground.arvados.org, you may download the files anonymously and upload them to your local Arvados instance:
-"https://cloud.curoverse.com/collections/2463fa9efeb75e099685528b3b9071e0+438":https://cloud.curoverse.com/collections/2463fa9efeb75e099685528b3b9071e0+438
+"https://playground.arvados.org/collections/2463fa9efeb75e099685528b3b9071e0+438":https://playground.arvados.org/collections/2463fa9efeb75e099685528b3b9071e0+438
-"https://cloud.curoverse.com/collections/ae480c5099b81e17267b7445e35b4bc7+180":https://cloud.curoverse.com/collections/ae480c5099b81e17267b7445e35b4bc7+180
+"https://playground.arvados.org/collections/ae480c5099b81e17267b7445e35b4bc7+180":https://playground.arvados.org/collections/ae480c5099b81e17267b7445e35b4bc7+180
-"https://cloud.curoverse.com/collections/655c6cd07550151b210961ed1d3852cf+57":https://cloud.curoverse.com/collections/655c6cd07550151b210961ed1d3852cf+57
+"https://playground.arvados.org/collections/655c6cd07550151b210961ed1d3852cf+57":https://playground.arvados.org/collections/655c6cd07550151b210961ed1d3852cf+57
h2. Submitting a workflow to an Arvados cluster
@arv-copy@ allows users to copy collections and pipeline templates from one cluster to another. By default, @arv-copy@ will recursively go through a template and copy all dependencies associated with the object.
-For example, let's copy from the <a href="https://cloud.curoverse.com/">cloud instance *qr1hi*</a> to *dst_cluster*. The names *qr1hi* and *dst_cluster* are interchangable with any cluster name. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *qr1hi*-4zz18-tci4vn4fa95w0zx, the cluster name is qr1hi.
+For example, let's copy from the <a href="https://playground.arvados.org/">Arvados playground</a>, also known as *qr1hi*, to *dst_cluster*. The names *qr1hi* and *dst_cluster* are interchangable with any cluster name. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *qr1hi*-4zz18-tci4vn4fa95w0zx, the cluster name is qr1hi.
In order to communicate with both clusters, you must create custom configuration files for each cluster. In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Current token*. Copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ in both of your clusters. Then, create two configuration files, one for each cluster. The names of the files must have the format of *uuid_prefix.conf*. In our example, let's make two files, one for *qr1hi* and one for *dst_cluster*. From your *Current token* page in *qr1hi* and *dst_cluster*, copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@.
First, select the uuid of the collection you want to copy from the source cluster. The uuid can be found in the collection display page in the collection summary area (top left box), or from the URL bar (the part after @collections/...@)
-Now copy the collection from *qr1hi* to *dst_cluster*. We will use the uuid @qr1hi-4zz18-tci4vn4fa95w0zx@ as an example. You can find this collection in the <a href="https://cloud.curoverse.com/collections/qr1hi-4zz18-tci4vn4fa95w0zx">lobSTR v.3 project on cloud.curoverse.com</a>.
+Now copy the collection from *qr1hi* to *dst_cluster*. We will use the uuid @qr1hi-4zz18-tci4vn4fa95w0zx@ as an example. You can find this collection in the <a href="https://playground.arvados.org/collections/qr1hi-4zz18-tci4vn4fa95w0zx">lobSTR v.3 project on playground.arvados.org</a>.
<notextile>
<pre><code>~$ <span class="userinput">arv-copy --src qr1hi --dst dst_cluster qr1hi-4zz18-tci4vn4fa95w0zx</span>
qr1hi-4zz18-tci4vn4fa95w0zx: 6.1M / 6.1M 100.0%
<pre><code>~$ <span class="userinput">git rev-parse HEAD</span></code></pre>
</notextile>
-* @"docker_image"@ : The docker image hash used is found on the "Collection page":https://cloud.curoverse.com/collections/qr1hi-4zz18-dov6im679g3jr1n as the *Content address*.
+* @"docker_image"@ : The docker image hash used is found on the "Collection page":https://playground.arvados.org/collections/qr1hi-4zz18-dov6im679g3jr1n as the *Content address*.
h2. Running your pipeline
<pre><code>~$ <span class="userinput">git rev-parse HEAD</span></code></pre>
</notextile>
-* @"docker_image"@ : This specifies the "Docker":https://www.docker.com/ runtime environment where jobs run their scripts. Docker version control is similar to git, and you can commit and push changes to your images. You must re-use the docker image hash from the previous run to use the same image. It can be found on the "Collection page":https://cloud.curoverse.com/collections/qr1hi-4zz18-dov6im679g3jr1n as the *Content address* or the *docker_image_locator* in a job's metadata.
+* @"docker_image"@ : This specifies the "Docker":https://www.docker.com/ runtime environment where jobs run their scripts. Docker version control is similar to git, and you can commit and push changes to your images. You must re-use the docker image hash from the previous run to use the same image. It can be found on the "Collection page":https://playground.arvados.org/collections/qr1hi-4zz18-dov6im679g3jr1n as the *Content address* or the *docker_image_locator* in a job's metadata.
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package httpserver
+
+import (
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "git.curoverse.com/arvados.git/sdk/go/stats"
+ "github.com/Sirupsen/logrus"
+ "github.com/gogo/protobuf/jsonpb"
+ "github.com/prometheus/client_golang/prometheus"
+ "github.com/prometheus/client_golang/prometheus/promhttp"
+)
+
+type Handler interface {
+ http.Handler
+
+ // Returns an http.Handler that serves the Handler's metrics
+ // data at /metrics and /metrics.json, and passes other
+ // requests through to next.
+ ServeAPI(next http.Handler) http.Handler
+}
+
+type metrics struct {
+ next http.Handler
+ logger *logrus.Logger
+ registry *prometheus.Registry
+ reqDuration *prometheus.SummaryVec
+ timeToStatus *prometheus.SummaryVec
+ exportProm http.Handler
+}
+
+func (*metrics) Levels() []logrus.Level {
+ return logrus.AllLevels
+}
+
+// Fire implements logrus.Hook in order to collect data points from
+// request logs.
+func (m *metrics) Fire(ent *logrus.Entry) error {
+ if tts, ok := ent.Data["timeToStatus"].(stats.Duration); !ok {
+ } else if method, ok := ent.Data["reqMethod"].(string); !ok {
+ } else if code, ok := ent.Data["respStatusCode"].(int); !ok {
+ } else {
+ m.timeToStatus.WithLabelValues(strconv.Itoa(code), strings.ToLower(method)).Observe(time.Duration(tts).Seconds())
+ }
+ return nil
+}
+
+func (m *metrics) exportJSON(w http.ResponseWriter, req *http.Request) {
+ jm := jsonpb.Marshaler{Indent: " "}
+ mfs, _ := m.registry.Gather()
+ w.Write([]byte{'['})
+ for i, mf := range mfs {
+ if i > 0 {
+ w.Write([]byte{','})
+ }
+ jm.Marshal(w, mf)
+ }
+ w.Write([]byte{']'})
+}
+
+// ServeHTTP implements http.Handler.
+func (m *metrics) ServeHTTP(w http.ResponseWriter, req *http.Request) {
+ m.next.ServeHTTP(w, req)
+}
+
+// ServeAPI returns a new http.Handler that serves current data at
+// metrics API endpoints (currently "GET /metrics(.json)?") and passes
+// other requests through to next.
+//
+// Typical example:
+//
+// m := Instrument(...)
+// srv := http.Server{Handler: m.ServeAPI(m)}
+func (m *metrics) ServeAPI(next http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+ switch {
+ case req.Method != "GET" && req.Method != "HEAD":
+ next.ServeHTTP(w, req)
+ case req.URL.Path == "/metrics.json":
+ m.exportJSON(w, req)
+ case req.URL.Path == "/metrics":
+ m.exportProm.ServeHTTP(w, req)
+ default:
+ next.ServeHTTP(w, req)
+ }
+ })
+}
+
+// Instrument returns a new Handler that passes requests through to
+// the next handler in the stack, and tracks metrics of those
+// requests.
+//
+// For the metrics to be accurate, the caller must ensure every
+// request passed to the Handler also passes through
+// LogRequests(logger, ...), and vice versa.
+func Instrument(logger *logrus.Logger, next http.Handler) Handler {
+ if logger == nil {
+ logger = logrus.StandardLogger()
+ }
+ reqDuration := prometheus.NewSummaryVec(prometheus.SummaryOpts{
+ Name: "request_duration_seconds",
+ Help: "Summary of request duration.",
+ }, []string{"code", "method"})
+ timeToStatus := prometheus.NewSummaryVec(prometheus.SummaryOpts{
+ Name: "time_to_status_seconds",
+ Help: "Summary of request TTFB.",
+ }, []string{"code", "method"})
+ registry := prometheus.NewRegistry()
+ registry.MustRegister(timeToStatus)
+ registry.MustRegister(reqDuration)
+ m := &metrics{
+ next: promhttp.InstrumentHandlerDuration(reqDuration, next),
+ logger: logger,
+ registry: registry,
+ reqDuration: reqDuration,
+ timeToStatus: timeToStatus,
+ exportProm: promhttp.HandlerFor(registry, promhttp.HandlerOpts{
+ ErrorLog: logger,
+ }),
+ }
+ m.logger.AddHook(m)
+ return m
+}
// avoids redirecting requests to keep-web if they depend on
// TrustAllContent being enabled.
//
+// Metrics
+//
+// Keep-web exposes request metrics in Prometheus text-based format at
+// /metrics. The same information is also available as JSON at
+// /metrics.json.
+//
package main
type handler struct {
Config *Config
+ MetricsAPI http.Handler
clientPool *arvadosclient.ClientPool
setupOnce sync.Once
healthHandler http.Handler
} else if r.URL.Path == "/status.json" {
h.serveStatus(w, r)
return
+ } else if strings.HasPrefix(r.URL.Path, "/metrics") {
+ h.MetricsAPI.ServeHTTP(w, r)
+ return
} else if siteFSDir[pathParts[0]] {
useSiteFS = true
} else if len(pathParts) >= 1 && strings.HasPrefix(pathParts[0], "c=") {
func (s *UnitSuite) TestCORSPreflight(c *check.C) {
h := handler{Config: DefaultConfig()}
- u, _ := url.Parse("http://keep-web.example/c=" + arvadostest.FooCollection + "/foo")
+ u := mustParseURL("http://keep-web.example/c=" + arvadostest.FooCollection + "/foo")
req := &http.Request{
Method: "OPTIONS",
Host: u.Host,
"http://" + bogusID + ".keep-web/t=" + token + "/" + bogusID + "/foo",
} {
c.Log(trial)
- u, err := url.Parse(trial)
- c.Assert(err, check.IsNil)
+ u := mustParseURL(trial)
req := &http.Request{
Method: "GET",
Host: u.Host,
package main
import (
+ "net/http"
+
"git.curoverse.com/arvados.git/sdk/go/httpserver"
)
}
func (srv *server) Start() error {
- srv.Handler = httpserver.AddRequestIDs(httpserver.LogRequests(nil, &handler{Config: srv.Config}))
+ h := &handler{Config: srv.Config}
+ mh := httpserver.Instrument(nil, httpserver.AddRequestIDs(httpserver.LogRequests(nil, h)))
+ h.MetricsAPI = mh.ServeAPI(http.NotFoundHandler())
+ srv.Handler = mh
srv.Addr = srv.Config.Listen
return srv.Server.Start()
}
import (
"crypto/md5"
+ "encoding/json"
"fmt"
"io"
"io/ioutil"
"net"
+ "net/http"
"os"
"os/exec"
"strings"
return
}
+func (s *IntegrationSuite) TestMetrics(c *check.C) {
+ origin := "http://" + s.testServer.Addr
+ req, _ := http.NewRequest("GET", origin+"/notfound", nil)
+ _, err := http.DefaultClient.Do(req)
+ c.Assert(err, check.IsNil)
+ req, _ = http.NewRequest("GET", origin+"/by_id/", nil)
+ req.Header.Set("Authorization", "Bearer "+arvadostest.ActiveToken)
+ resp, err := http.DefaultClient.Do(req)
+ c.Assert(err, check.IsNil)
+ c.Check(resp.StatusCode, check.Equals, http.StatusOK)
+ req, _ = http.NewRequest("GET", origin+"/foo", nil)
+ req.Host = arvadostest.FooCollection + ".example.com"
+ req.Header.Set("Authorization", "Bearer "+arvadostest.ActiveToken)
+ resp, err = http.DefaultClient.Do(req)
+ c.Assert(err, check.IsNil)
+ c.Check(resp.StatusCode, check.Equals, http.StatusOK)
+ buf, _ := ioutil.ReadAll(resp.Body)
+ c.Check(buf, check.DeepEquals, []byte("foo"))
+
+ req, _ = http.NewRequest("GET", origin+"/metrics.json", nil)
+ resp, err = http.DefaultClient.Do(req)
+ c.Assert(err, check.IsNil)
+ c.Check(resp.StatusCode, check.Equals, http.StatusOK)
+ type summary struct {
+ SampleCount string `json:"sample_count"`
+ SampleSum float64 `json:"sample_sum"`
+ Quantile []struct {
+ Quantile float64
+ Value float64
+ }
+ }
+ var ents []struct {
+ Name string
+ Help string
+ Type string
+ Metric []struct {
+ Label []struct {
+ Name string
+ Value string
+ }
+ Summary summary
+ }
+ }
+ json.NewDecoder(resp.Body).Decode(&ents)
+ flat := map[string]summary{}
+ for _, e := range ents {
+ for _, m := range e.Metric {
+ labels := map[string]string{}
+ for _, lbl := range m.Label {
+ labels[lbl.Name] = lbl.Value
+ }
+ flat[e.Name+"/"+labels["method"]+"/"+labels["code"]] = m.Summary
+ }
+ }
+ c.Check(flat["request_duration_seconds/get/200"].SampleSum, check.Not(check.Equals), 0)
+ c.Check(flat["request_duration_seconds/get/200"].SampleCount, check.Equals, "2")
+ c.Check(flat["request_duration_seconds/get/404"].SampleCount, check.Equals, "1")
+ c.Check(flat["time_to_status_seconds/get/404"].SampleCount, check.Equals, "1")
+
+ // If the Host header indicates a collection, /metrics.json
+ // refers to a file in the collection -- the metrics handler
+ // must not intercept that route.
+ req, _ = http.NewRequest("GET", origin+"/metrics.json", nil)
+ req.Host = strings.Replace(arvadostest.FooCollectionPDH, "+", "-", -1) + ".example.com"
+ req.Header.Set("Authorization", "Bearer "+arvadostest.ActiveToken)
+ resp, err = http.DefaultClient.Do(req)
+ c.Assert(err, check.IsNil)
+ c.Check(resp.StatusCode, check.Equals, http.StatusNotFound)
+}
+
func (s *IntegrationSuite) SetUpSuite(c *check.C) {
arvadostest.StartAPI()
arvadostest.StartKeep(2, true)
"encoding/json"
"fmt"
"io/ioutil"
- "net/http"
- "strconv"
"strings"
"time"
"git.curoverse.com/arvados.git/sdk/go/arvados"
- "git.curoverse.com/arvados.git/sdk/go/stats"
"github.com/Sirupsen/logrus"
- "github.com/golang/protobuf/jsonpb"
- "github.com/prometheus/client_golang/prometheus"
- "github.com/prometheus/client_golang/prometheus/promhttp"
)
type Config struct {
ManagementToken string `doc: The secret key that must be provided by monitoring services
wishing to access the health check endpoint (/_health).`
-
- metrics
}
var (
return nil
}
-type metrics struct {
- registry *prometheus.Registry
- reqDuration *prometheus.SummaryVec
- timeToStatus *prometheus.SummaryVec
- exportProm http.Handler
-}
-
-func (*metrics) Levels() []logrus.Level {
- return logrus.AllLevels
-}
-
-func (m *metrics) Fire(ent *logrus.Entry) error {
- if tts, ok := ent.Data["timeToStatus"].(stats.Duration); !ok {
- } else if method, ok := ent.Data["reqMethod"].(string); !ok {
- } else if code, ok := ent.Data["respStatusCode"].(int); !ok {
- } else {
- m.timeToStatus.WithLabelValues(strconv.Itoa(code), strings.ToLower(method)).Observe(time.Duration(tts).Seconds())
- }
- return nil
-}
-
-func (m *metrics) setup() {
- m.registry = prometheus.NewRegistry()
- m.timeToStatus = prometheus.NewSummaryVec(prometheus.SummaryOpts{
- Name: "time_to_status_seconds",
- Help: "Summary of request TTFB.",
- }, []string{"code", "method"})
- m.reqDuration = prometheus.NewSummaryVec(prometheus.SummaryOpts{
- Name: "request_duration_seconds",
- Help: "Summary of request duration.",
- }, []string{"code", "method"})
- m.registry.MustRegister(m.timeToStatus)
- m.registry.MustRegister(m.reqDuration)
- m.exportProm = promhttp.HandlerFor(m.registry, promhttp.HandlerOpts{
- ErrorLog: log,
- })
- log.AddHook(m)
-}
-
-func (m *metrics) exportJSON(w http.ResponseWriter, req *http.Request) {
- jm := jsonpb.Marshaler{Indent: " "}
- mfs, _ := m.registry.Gather()
- w.Write([]byte{'['})
- for i, mf := range mfs {
- if i > 0 {
- w.Write([]byte{','})
- }
- jm.Marshal(w, mf)
- }
- w.Write([]byte{']'})
-}
-
-func (m *metrics) Instrument(next http.Handler) http.Handler {
- return promhttp.InstrumentHandlerDuration(m.reqDuration, next)
-}
-
// VolumeTypes is built up by init() funcs in the source files that
// define the volume types.
var VolumeTypes = []func() VolumeWithExamples{}
// 400 Bad Request.
rtr.NotFoundHandler = http.HandlerFunc(BadRequestHandler)
- theConfig.metrics.setup()
-
rtr.limiter = httpserver.NewRequestLimiter(theConfig.MaxRequests, rtr)
- mux := http.NewServeMux()
- mux.Handle("/", theConfig.metrics.Instrument(
- httpserver.AddRequestIDs(httpserver.LogRequests(nil, rtr.limiter))))
- mux.HandleFunc("/metrics.json", theConfig.metrics.exportJSON)
- mux.Handle("/metrics", theConfig.metrics.exportProm)
-
- return mux
+ stack := httpserver.Instrument(nil,
+ httpserver.AddRequestIDs(httpserver.LogRequests(nil, rtr.limiter)))
+ return stack.ServeAPI(stack)
}
// BadRequestHandler is a HandleFunc to address bad requests.
"revision": "7a0fa49edf48165190530c675167e2f319a05268",
"revisionTime": "2018-06-25T08:58:08Z"
},
+ {
+ "checksumSHA1": "8UEp6v0Dczw/SlasE0DivB0mAHA=",
+ "path": "github.com/gogo/protobuf/jsonpb",
+ "revision": "30cf7ac33676b5786e78c746683f0d4cd64fa75b",
+ "revisionTime": "2018-05-09T16:24:41Z"
+ },
{
"checksumSHA1": "wn2shNJMwRZpvuvkf1s7h0wvqHI=",
"path": "github.com/gogo/protobuf/proto",
"revisionTime": "2018-01-04T10:21:28Z"
},
{
- "checksumSHA1": "iVfdaLxIDjfk2KLP8dCMIbsxZZM=",
- "path": "github.com/golang/protobuf/jsonpb",
- "revision": "1e59b77b52bf8e4b449a57e6f79f21226d571845",
- "revisionTime": "2017-11-13T18:07:20Z"
+ "checksumSHA1": "HPVQZu059/Rfw2bAWM538bVTcUc=",
+ "path": "github.com/gogo/protobuf/sortkeys",
+ "revision": "30cf7ac33676b5786e78c746683f0d4cd64fa75b",
+ "revisionTime": "2018-05-09T16:24:41Z"
},
{
- "checksumSHA1": "yqF125xVSkmfLpIVGrLlfE05IUk=",
- "path": "github.com/golang/protobuf/proto",
- "revision": "1e59b77b52bf8e4b449a57e6f79f21226d571845",
- "revisionTime": "2017-11-13T18:07:20Z"
+ "checksumSHA1": "SkxU1+wPGUJyLyQENrZtr2/OUBs=",
+ "path": "github.com/gogo/protobuf/types",
+ "revision": "30cf7ac33676b5786e78c746683f0d4cd64fa75b",
+ "revisionTime": "2018-05-09T16:24:41Z"
},
{
- "checksumSHA1": "Ylq6kq3KWBy6mu68oyEwenhNMdg=",
- "path": "github.com/golang/protobuf/ptypes/struct",
+ "checksumSHA1": "yqF125xVSkmfLpIVGrLlfE05IUk=",
+ "path": "github.com/golang/protobuf/proto",
"revision": "1e59b77b52bf8e4b449a57e6f79f21226d571845",
"revisionTime": "2017-11-13T18:07:20Z"
},
projects / arvados.git / commitdiff