SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Metrics endpoints are found at @/status.json@ on many Arvados services. The purpose of metrics are to provide statistics about the operation of a service, suitable for diagnosing how well a service is performing under load.
+Some Arvados services publish Prometheus/OpenMetrics-compatible metrics at @/metrics@. Metrics can help you understand how components perform under load, find performance bottlenecks, and detect and diagnose problems.
-To access metrics endpoints, services must be configured with a "management token":management-token.html .
+To access metrics endpoints, services must be configured with a "management token":management-token.html. When accessing a metrics endpoint, prefix the management token with @"Bearer "@ and supply it in the @Authorization@ request header.
-h2. Keepstore
-
-h3. Root
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Volumes| array of "volumeStatusEnt":#volumeStatusEnt ||
-|BufferPool| "PoolStatus":#PoolStatus ||
-|PullQueue| "WorkQueueStatus":#WorkQueueStatus ||
-|TrashQueue| "WorkQueueStatus":#WorkQueueStatus ||
-|RequestsCurrent| int ||
-|RequestsMax| int ||
-|Version| string ||
-
-h3(#volumeStatusEnt). volumeStatusEnt
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Label| string||
-|Status| "VolumeStatus":#VolumeStatus ||
-|VolumeStats| "ioStats":#ioStats ||
-
-h3(#VolumeStatus). VolumeStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|MountPoint| string||
-|DeviceNum| uint64||
-|BytesFree| uint64||
-|BytesUsed| uint64||
-
-h3(#ioStats). ioStats
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Errors| uint64||
-|Ops| uint64||
-|CompareOps| uint64||
-|GetOps| uint64||
-|PutOps| uint64||
-|TouchOps| uint64||
-|InBytes| uint64||
-|OutBytes| uint64||
-
-h3(#PoolStatus). PoolStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|BytesAllocatedCumulative| uint64||
-|BuffersMax| int||
-|BuffersInUse| int||
-
-h3(#WorkQueueStatus). WorkQueueStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|InProgress| int||
-|Queued| int||
-
-h3. Example response
-
-<pre>
-{
- "Volumes": [
- {
- "Label": "[UnixVolume /var/lib/arvados/keep0]",
- "Status": {
- "MountPoint": "/var/lib/arvados/keep0",
- "DeviceNum": 65029,
- "BytesFree": 222532972544,
- "BytesUsed": 435456679936
- },
- "InternalStats": {
- "Errors": 0,
- "InBytes": 1111,
- "OutBytes": 0,
- "OpenOps": 1,
- "StatOps": 4,
- "FlockOps": 0,
- "UtimesOps": 0,
- "CreateOps": 0,
- "RenameOps": 0,
- "UnlinkOps": 0,
- "ReaddirOps": 0
- }
- }
- ],
- "BufferPool": {
- "BytesAllocatedCumulative": 67108864,
- "BuffersMax": 20,
- "BuffersInUse": 0
- },
- "PullQueue": {
- "InProgress": 0,
- "Queued": 0
- },
- "TrashQueue": {
- "InProgress": 0,
- "Queued": 0
- },
- "RequestsCurrent": 1,
- "RequestsMax": 40,
- "Version": "dev"
-}
+<pre>curl -sfH "Authorization: Bearer your_management_token_goes_here" "https://0.0.0.0:25107/metrics"
</pre>
-h2. Node manager
-
-The node manager status end point provides a snapshot of internal status at the time of the most recent wishlist update.
+The plain text export format includes "help" messages with a description of each reported metric.
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|nodes_booting|int|Number of nodes in booting state|
-|nodes_unpaired|int|Number of nodes in unpaired state|
-|nodes_busy|int|Number of nodes in busy state|
-|nodes_idle|int|Number of nodes in idle state|
-|nodes_fail|int|Number of nodes in fail state|
-|nodes_down|int|Number of nodes in down state|
-|nodes_shutdown|int|Number of nodes in shutdown state|
-|nodes_wish|int|Number of nodes in the current wishlist|
-|node_quota|int|Current node count ceiling due to cloud quota limits|
-|config_max_nodes|int|Configured max node count|
+When configuring Prometheus, use a @bearer_token@ or @bearer_token_file@ option to authenticate requests.
-h3. Example
-
-<pre>
-{
- "actor_exceptions": 0,
- "idle_times": {
- "compute1": 0,
- "compute3": 0,
- "compute2": 0,
- "compute4": 0
- },
- "create_node_errors": 0,
- "destroy_node_errors": 0,
- "nodes_idle": 0,
- "config_max_nodes": 8,
- "list_nodes_errors": 0,
- "node_quota": 8,
- "Version": "1.1.4.20180719160944",
- "nodes_wish": 0,
- "nodes_unpaired": 0,
- "nodes_busy": 4,
- "boot_failures": 0
-}
+<pre>scrape_configs:
+ - job_name: keepstore
+ bearer_token: your_management_token_goes_here
+ static_configs:
+ - targets:
+ - "keep0.ClusterID.example.com:25107"
</pre>
+
+table(table table-bordered table-condensed table-hover).
+|_. Component|_. Metrics endpoint|
+|arvados-api-server||
+|arvados-controller|✓|
+|arvados-dispatch-cloud|✓|
+|arvados-git-httpd||
+|arvados-ws|✓|
+|composer||
+|keepproxy||
+|keepstore|✓|
+|keep-balance|✓|
+|keep-web|✓|
+|workbench1||
+|workbench2||