--- /dev/null
+Arvados Code of Conduct
+=======================
+
+The Arvados Project is dedicated to providing a harassment-free experience for
+everyone. We do not tolerate harassment of participants in any form.
+
+This code of conduct applies to all Arvados Project spaces both online and off:
+Gitter chat, Redmine issues, wiki, mailing lists, forums, video chats, and any other
+Arvados spaces. Anyone who violates this code of conduct may be sanctioned or
+expelled from these spaces at the discretion of the Arvados Team.
+
+Some Arvados Project spaces may have additional rules in place, which will be
+made clearly available to participants. Participants are responsible for
+knowing and abiding by these rules.
+
+Harassment includes, but is not limited to:
+
+ - Offensive comments related to gender, gender identity and expression, sexual
+orientation, disability, mental illness, neuro(a)typicality, physical
+appearance, body size, age, race, or religion.
+ - Unwelcome comments regarding a person’s lifestyle choices and practices,
+including those related to food, health, parenting, drugs, and employment.
+ - Deliberate misgendering or use of [dead](https://www.quora.com/What-is-deadnaming/answer/Nancy-C-Walker)
+or rejected names.
+ - Gratuitous or off-topic sexual images or behaviour in spaces where they’re not
+appropriate.
+ - Physical contact and simulated physical contact (eg, textual descriptions like
+“\*hug\*” or “\*backrub\*”) without consent or after a request to stop.
+ - Threats of violence.
+ - Incitement of violence towards any individual, including encouraging a person
+to commit suicide or to engage in self-harm.
+ - Deliberate intimidation.
+ - Stalking or following.
+ - Harassing photography or recording, including logging online activity for
+harassment purposes.
+ - Sustained disruption of discussion.
+ - Unwelcome sexual attention.
+ - Pattern of inappropriate social contact, such as requesting/assuming
+inappropriate levels of intimacy with others
+ - Continued one-on-one communication after requests to cease.
+ - Deliberate “outing” of any aspect of a person’s identity without their consent
+except as necessary to protect vulnerable people from intentional abuse.
+ - Publication of non-harassing private communication.
+
+The Arvados Project prioritizes marginalized people’s safety over privileged
+people’s comfort. The Arvados Leadership Team will not act on complaints regarding:
+
+ - ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’
+ - Reasonable communication of boundaries, such as “leave me alone,” “go away,” or
+“I’m not discussing this with you.”
+ - Communicating in a [tone](http://geekfeminism.wikia.com/wiki/Tone_argument)
+you don’t find congenial
+
+Reporting
+---------
+
+If you are being harassed by a member of the Arvados Project, notice that someone
+else is being harassed, or have any other concerns, please contact the Arvados
+Project Team at contact@arvados.org. If person who is harassing
+you is on the team, they will recuse themselves from handling your incident. We
+will respond as promptly as we can.
+
+This code of conduct applies to Arvados Project spaces, but if you are being
+harassed by a member of Arvados Project outside our spaces, we still want to
+know about it. We will take all good-faith reports of harassment by Arvados Project
+members, especially the Arvados Team, seriously. This includes harassment
+outside our spaces and harassment that took place at any point in time. The
+abuse team reserves the right to exclude people from the Arvados Project based on
+their past behavior, including behavior outside Arvados Project spaces and
+behavior towards people who are not in the Arvados Project.
+
+In order to protect volunteers from abuse and burnout, we reserve the right to
+reject any report we believe to have been made in bad faith. Reports intended
+to silence legitimate criticism may be deleted without response.
+
+We will respect confidentiality requests for the purpose of protecting victims
+of abuse. At our discretion, we may publicly name a person about whom we’ve
+received harassment complaints, or privately warn third parties about them, if
+we believe that doing so will increase the safety of Arvados Project members or
+the general public. We will not name harassment victims without their
+affirmative consent.
+
+Consequences
+------------
+
+Participants asked to stop any harassing behavior are expected to comply
+immediately.
+
+If a participant engages in harassing behavior, the Arvados Team may
+take any action they deem appropriate, up to and including expulsion from all
+Arvados Project spaces and identification of the participant as a harasser to other
+Arvados Project members or the general public.
+
+This anti-harassment policy is based on the [example policy from the Geek
+Feminism wiki](http://geekfeminism.wikia.com/wiki/Community_anti-harassment/Policy),
+created by the Geek Feminism community.
--- /dev/null
+[comment]: # (Copyright © The Arvados Authors. All rights reserved.)
+[comment]: # ()
+[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)
+
+# Contributing
+
+Arvados is free software, which means it is free for all to use, learn
+from, and improve. We encourage contributions from the community that
+improve Arvados for everyone. Some examples of contributions are bug
+reports, bug fixes, new features, and scripts or documentation that help
+with using, administering, or installing Arvados. We also love to
+hear about Arvados success stories.
+
+Those interested in contributing should begin by joining the [Arvados community
+channel](https://gitter.im/arvados/community) and telling us about your interest.
+
+Contributers should also create an account at https://dev.arvados.org
+to be able to create and comment on bug tracker issues. The
+Arvados public bug tracker is located at
+https://dev.arvados.org/projects/arvados/issues .
+
+Contributers may also be interested in the [development road map](https://dev.arvados.org/issues/gantt?utf8=%E2%9C%93&set_filter=1&gantt=1&f%5B%5D=project_id&op%5Bproject_id%5D=%3D&v%5Bproject_id%5D%5B%5D=49&f%5B%5D=&zoom=1).
+
+# Development
+
+Git repositories for primary development are located at
+https://git.arvados.org/ and can also be browsed at
+https://dev.arvados.org/projects/arvados/repository . Every push to
+the master branch is also mirrored to Github at
+https://github.com/arvados/arvados .
+
+Visit [Hacking Arvados](https://dev.arvados.org/projects/arvados/wiki/Hacking) for
+detailed information about setting up an Arvados development
+environment, development process, coding standards, and notes about specific components.
+
+If you wish to build the Arvados documentation from a local git clone, see
+[doc/README.textile](doc/README.textile) for instructions.
+
+# Pull requests
+
+The preferred method for making contributions is through Github pull requests.
+
+This is the general contribution process:
+
+1. Fork the Arvados repository using the Github "Fork" button
+2. Clone your fork, make your changes, commit to your fork.
+3. Every commit message must have a DCO sign-off and every file must have a SPDX license (see below).
+4. Add yourself to the [AUTHORS](AUTHORS) file
+5. When your fork is ready, through Github, Create a Pull Request against `arvados:master`
+6. Notify the core team about your pull request through the [Arvados development
+channel](https://gitter.im/arvados/development) or by other means.
+7. A member of the core team will review the pull request. They may have questions or comments, or request changes.
+8. When the contribution is ready, a member of the core team will
+merge the pull request into the master branch, which will
+automatically resolve the pull request.
+
+The Arvados project does not require a contributor agreement in advance, but does require each commit message include a [Developer Certificate of Origin](https://dev.arvados.org/projects/arvados/wiki/Developer_Certificate_Of_Origin). Please ensure *every git commit message* includes `Arvados-DCO-1.1-Signed-off-by`. If you have already made commits without it, fix them with `git commit --amend` or `git rebase`.
+
+The Developer Certificate of Origin line looks like this:
+
+```
+Arvados-DCO-1.1-Signed-off-by: Joe Smith <joe.smith@example.com>
+```
+
+New files must also include `SPDX-License-Identifier` at the top with one of the three Arvados open source licenses. See [COPYING](COPYING) for details.
+
+# Continuous integration
+
+Continuous integration is hosted at https://ci.arvados.org/
+
+Currently, external contributers cannot trigger builds. We are investigating integration with Github pull requests for the future.
+
+[![Build Status](https://ci.arvados.org/buildStatus/icon?job=run-tests)](https://ci.arvados.org/job/run-tests/)
+
+[![Go Report Card](https://goreportcard.com/badge/github.com/arvados/arvados)](https://goreportcard.com/report/github.com/arvados/arvados)
AGPL-3.0: agpl-3.0.txt
Apache-2.0: apache-2.0.txt
CC-BY-SA-3.0: cc-by-sa-3.0.txt
+
+As a general rule, code in the sdk/ directory is licensed Apache-2.0,
+documentation in the doc/ directory is licensed CC-BY-SA-3.0, and
+everything else is licensed AGPL-3.0.
\ No newline at end of file
[comment]: # ()
[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)
-[Arvados](https://arvados.org) is a free software distributed computing platform
-for bioinformatics, data science, and high throughput analysis of massive data
-sets. Arvados supports a variety of cloud, cluster and HPC environments.
+[![Join the chat at https://gitter.im/arvados/community](https://badges.gitter.im/arvados/community.svg)](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) | [Installing Arvados](https://doc.arvados.org/install/index.html) | [Installing Client SDKs](https://doc.arvados.org/sdk/index.html) | [Report a bug](https://dev.arvados.org/projects/arvados/issues/new) | [Development and Contributing](CONTRIBUTING.md)
-Arvados consists of:
+<img align="right" src="doc/images/dax.png" height="240px">
-* *Keep*: a petabyte-scale content-addressed distributed storage system for managing and
- storing collections of files, accessible via HTTP and FUSE mount.
+[Arvados](https://arvados.org) is an open source platform for
+managing, processing, and sharing genomic and other large scientific
+and biomedical data. With Arvados, bioinformaticians run and scale
+compute-intensive workflows, developers create biomedical
+applications, and IT administrators manage large compute and storage
+resources.
-* *Crunch*: a Docker-based cluster and HPC workflow engine designed providing
- strong versioning, reproducibilty, and provenance of computations.
+The key components of Arvados are:
-* Related services and components including a web workbench for managing files
- and compute jobs, REST APIs, SDKs, and other tools.
+* *Keep*: Keep is the Arvados storage system for managing and storing large
+collections of files. Keep combines content addressing and a
+distributed storage architecture resulting in both high reliability
+and high throughput. Every file stored in Keep can be accurately
+verified every time it is retrieved. Keep supports the creation of
+collections as a flexible way to define data sets without having to
+re-organize or needlessly copy data. Keep works on a wide range of
+underlying filesystems and object stores.
-## Quick start
+* *Crunch*: Crunch is the orchestration system for running [Common Workflow Language](https://www.commonwl.org) workflows. It is
+designed to maintain data provenance and workflow
+reproducibility. Crunch automatically tracks data inputs and outputs
+through Keep and executes workflow processes in Docker containers. In
+a cloud environment, Crunch optimizes costs by scaling compute on demand.
+
+* *Workbench*: The Workbench web application allows users to interactively access
+Arvados functionality. It is especially helpful for querying and
+browsing data, visualizing provenance, and tracking the progress of
+workflows.
+
+* *Command Line tools*: The command line interface (CLI) provides convenient access to Arvados
+functionality in the Arvados platform from the command line.
+
+* *API and SDKs*: Arvados is designed to be integrated with existing infrastructure. All
+the services in Arvados are accessed through a RESTful API. SDKs are
+available for Python, Go, R, Perl, Ruby, and Java.
+
+# Quick start
To try out Arvados on your local workstation, you can use Arvbox, which
provides Arvados components pre-installed in a Docker container (requires
configure Arvbox to be accessible over a network and for other options see
http://doc.arvados.org/install/arvbox.html for details.
-## Documentation
+# Documentation
-Complete documentation, including a User Guide, Installation documentation and
-API documentation is available at http://doc.arvados.org/
+Complete documentation, including the [User Guide](https://doc.arvados.org/user/index.html), [Installation documentation](https://doc.arvados.org/install/index.html), [Administrator documentation](https://doc.arvados.org/admin/index.html) and
+[API documentation](https://doc.arvados.org/api/index.html) is available at http://doc.arvados.org/
If you wish to build the Arvados documentation from a local git clone, see
-doc/README.textile for instructions.
+[doc/README.textile](doc/README.textile) for instructions.
-## Community
+# Community
[![Join the chat at https://gitter.im/arvados/community](https://badges.gitter.im/arvados/community.svg)](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
-The [arvados community channel](https://gitter.im/arvados/community)
+The [Arvados community channel](https://gitter.im/arvados/community)
channel at [gitter.im](https://gitter.im) is available for live
discussion and support.
-The
-[Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados)
-is a forum for general discussion, questions, and news about Arvados
-development. The
-[Arvados developer mailing list](http://lists.arvados.org/mailman/listinfo/arvados-dev)
-is a forum for more technical discussion, intended for developers and
-contributors to Arvados.
+The [Arvados developement channel](https://gitter.im/arvados/development)
+channel at [gitter.im](https://gitter.im) is used to coordinate development.
-## Development
+The [Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados)
+is used to announce new versions and other news.
-[![Build Status](https://ci.arvados.org/buildStatus/icon?job=run-tests)](https://ci.arvados.org/job/run-tests/)
-[![Go Report Card](https://goreportcard.com/badge/github.com/arvados/arvados)](https://goreportcard.com/report/github.com/arvados/arvados)
+All participants are expected to abide by the [Arvados Code of Conduct](CODE_OF_CONDUCT.md).
-The Arvados public bug tracker is located at https://dev.arvados.org/projects/arvados/issues
+# Reporting bugs
-Continuous integration is hosted at https://ci.arvados.org/
+[Report a bug](https://dev.arvados.org/projects/arvados/issues/new) on [dev.arvados.org](https://dev.arvados.org).
-Instructions for setting up a development environment and working on specific
-components can be found on the
-["Hacking Arvados" page of the Arvados wiki](https://dev.arvados.org/projects/arvados/wiki/Hacking).
+# Development and Contributing
-## Contributing
+See [CONTRIBUTING](CONTRIBUTING.md) for information about Arvados development and how to contribute to the Arvados project.
-When making a pull request, please ensure *every git commit message* includes a one-line [Developer Certificate of Origin](https://dev.arvados.org/projects/arvados/wiki/Developer_Certificate_Of_Origin). If you have already made commits without it, fix them with `git commit --amend` or `git rebase`.
+The [development road map](https://dev.arvados.org/issues/gantt?utf8=%E2%9C%93&set_filter=1&gantt=1&f%5B%5D=project_id&op%5Bproject_id%5D=%3D&v%5Bproject_id%5D%5B%5D=49&f%5B%5D=&zoom=1) outlines some of the project priorities over the next twelve months.
-## Licensing
+# Licensing
-Arvados is Free Software. See COPYING for information about Arvados Free
-Software licenses.
+Arvados is Free Software. See [COPYING](COPYING) for information about the open source licenses used in Arvados.
RUN git clone --depth 1 git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle
# The version of setuptools that comes with CentOS is way too old
-RUN pip install --upgrade setuptools
+RUN pip install --upgrade 'setuptools<45'
ENV WORKSPACE /arvados
CMD ["scl", "enable", "rh-python36", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos7"]
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip unzip python3-venv python3-dev
# Install virtualenv
-RUN /usr/bin/pip install virtualenv
+RUN /usr/bin/pip install 'virtualenv<20'
# Install RVM
ADD generated/mpapis.asc /tmp/
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip unzip python3-venv python3-dev
# Install virtualenv
-RUN /usr/bin/pip install virtualenv
+RUN /usr/bin/pip install 'virtualenv<20'
# Install RVM
ADD generated/mpapis.asc /tmp/
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev libgnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip tzdata python3-venv python3-dev
# Install virtualenv
-RUN /usr/bin/pip install virtualenv
+RUN /usr/bin/pip install 'virtualenv<20'
# Install RVM
ADD generated/mpapis.asc /tmp/
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-pip libcurl4-gnutls-dev libgnutls28-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip tzdata python3-venv python3-dev
# Install virtualenv
-RUN /usr/bin/pip install virtualenv
+RUN /usr/bin/pip install 'virtualenv<20'
# Install RVM
ADD generated/mpapis.asc /tmp/
rm -rf dist/*
# Get the latest setuptools
- if ! $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools; then
+ if ! $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'; then
echo "Error, unable to upgrade setuptools with"
- echo " $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools"
+ echo " $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'"
exit 1
fi
# filter a useless warning (when building the cwltest package) from the stderr output
fi
echo "pip version: `build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip --version`"
- if ! build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools; then
+ if ! build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'; then
echo "Error, unable to upgrade setuptools with"
- echo " build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools"
+ echo " build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'"
exit 1
fi
echo "setuptools version: `build/usr/share/$python/dist/$PYTHON_PKG/bin/$python -c 'import setuptools; print(setuptools.__version__)'`"
<script>
function dismissAnnotateNotify() {
- window.sessionStorage.setItem("dismiss-annotate-notify", "true");
+ window.localStorage.setItem("dismiss-annotate-notify", "true");
$('#annotate-notify').attr('style', "display: none;");
}
- if (window.sessionStorage.getItem("dismiss-annotate-notify") === "true") {
+ if (window.localStorage.getItem("dismiss-annotate-notify") === "true") {
dismissAnnotateNotify();
} else {
$('#annotate-notify').attr('style', "display: inline-block;");
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Some Arvados services publish Prometheus/OpenMetrics-compatible metrics at @/metrics@, and some provide additional runtime status at @/status.json@. Metrics can help you understand how components perform under load, find performance bottlenecks, and detect and diagnose problems.
+Some Arvados services publish Prometheus/OpenMetrics-compatible metrics at @/metrics@. Metrics can help you understand how components perform under load, find performance bottlenecks, and detect and diagnose problems.
To access metrics endpoints, services must be configured with a "management token":management-token.html. When accessing a metrics endpoint, prefix the management token with @"Bearer "@ and supply it in the @Authorization@ request header.
-<pre>curl -sfH "Authorization: Bearer your_management_token_goes_here" "https://0.0.0.0:25107/status.json"
+<pre>curl -sfH "Authorization: Bearer your_management_token_goes_here" "https://0.0.0.0:25107/metrics"
</pre>
-h2. Keep-web
+The plain text export format includes "help" messages with a description of each reported metric.
-Keep-web exports metrics at @/metrics@ -- e.g., @https://collections.zzzzz.arvadosapi.com/metrics@.
+When configuring Prometheus, use a @bearer_token@ or @bearer_token_file@ option to authenticate requests.
-table(table table-bordered table-condensed).
-|_. Name|_. Type|_. Description|
-|request_duration_seconds|summary|elapsed time between receiving a request and sending the last byte of the response body (segmented by HTTP request method and response status code)|
-|time_to_status_seconds|summary|elapsed time between receiving a request and sending the HTTP response status code (segmented by HTTP request method and response status code)|
-
-Metrics in the @arvados_keepweb_collectioncache@ namespace report keep-web's internal cache of Arvados collection metadata.
-
-table(table table-bordered table-condensed).
-|_. Name|_. Type|_. Description|
-|arvados_keepweb_collectioncache_requests|counter|cache lookups|
-|arvados_keepweb_collectioncache_api_calls|counter|outgoing API calls|
-|arvados_keepweb_collectioncache_permission_hits|counter|collection-to-permission cache hits|
-|arvados_keepweb_collectioncache_pdh_hits|counter|UUID-to-PDH cache hits|
-|arvados_keepweb_collectioncache_hits|counter|PDH-to-manifest cache hits|
-|arvados_keepweb_collectioncache_cached_manifests|gauge|number of collections in the cache|
-|arvados_keepweb_collectioncache_cached_manifest_bytes|gauge|memory consumed by cached collection manifests|
-
-h2. Keepstore
-
-Keepstore exports metrics at @/status.json@ -- e.g., @http://keep0.zzzzz.arvadosapi.com:25107/status.json@.
-
-h3. Root
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Volumes| array of "volumeStatusEnt":#volumeStatusEnt ||
-|BufferPool| "PoolStatus":#PoolStatus ||
-|PullQueue| "WorkQueueStatus":#WorkQueueStatus ||
-|TrashQueue| "WorkQueueStatus":#WorkQueueStatus ||
-|RequestsCurrent| int ||
-|RequestsMax| int ||
-|Version| string ||
-
-h3(#volumeStatusEnt). volumeStatusEnt
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Label| string||
-|Status| "VolumeStatus":#VolumeStatus ||
-|VolumeStats| "ioStats":#ioStats ||
-
-h3(#VolumeStatus). VolumeStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|MountPoint| string||
-|DeviceNum| uint64||
-|BytesFree| uint64||
-|BytesUsed| uint64||
-
-h3(#ioStats). ioStats
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Errors| uint64||
-|Ops| uint64||
-|CompareOps| uint64||
-|GetOps| uint64||
-|PutOps| uint64||
-|TouchOps| uint64||
-|InBytes| uint64||
-|OutBytes| uint64||
-
-h3(#PoolStatus). PoolStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|BytesAllocatedCumulative| uint64||
-|BuffersMax| int||
-|BuffersInUse| int||
-
-h3(#WorkQueueStatus). WorkQueueStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|InProgress| int||
-|Queued| int||
-
-h3. Example response
-
-<pre>
-{
- "Volumes": [
- {
- "Label": "[UnixVolume /var/lib/arvados/keep0]",
- "Status": {
- "MountPoint": "/var/lib/arvados/keep0",
- "DeviceNum": 65029,
- "BytesFree": 222532972544,
- "BytesUsed": 435456679936
- },
- "InternalStats": {
- "Errors": 0,
- "InBytes": 1111,
- "OutBytes": 0,
- "OpenOps": 1,
- "StatOps": 4,
- "FlockOps": 0,
- "UtimesOps": 0,
- "CreateOps": 0,
- "RenameOps": 0,
- "UnlinkOps": 0,
- "ReaddirOps": 0
- }
- }
- ],
- "BufferPool": {
- "BytesAllocatedCumulative": 67108864,
- "BuffersMax": 20,
- "BuffersInUse": 0
- },
- "PullQueue": {
- "InProgress": 0,
- "Queued": 0
- },
- "TrashQueue": {
- "InProgress": 0,
- "Queued": 0
- },
- "RequestsCurrent": 1,
- "RequestsMax": 40,
- "Version": "dev"
-}
+<pre>scrape_configs:
+ - job_name: keepstore
+ bearer_token: your_management_token_goes_here
+ static_configs:
+ - targets:
+ - "keep0.ClusterID.example.com:25107"
</pre>
-h2. Keep-balance
-
-Keep-balance exports metrics at @/metrics@ -- e.g., @http://keep.zzzzz.arvadosapi.com:9005/metrics@.
-
-table(table table-bordered table-condensed).
-|_. Name|_. Type|_. Description|
-|arvados_keep_total_{replicas,blocks,bytes}|gauge|stored data (stored in backend volumes, whether referenced or not)|
-|arvados_keep_garbage_{replicas,blocks,bytes}|gauge|garbage data (unreferenced, and old enough to trash)|
-|arvados_keep_transient_{replicas,blocks,bytes}|gauge|transient data (unreferenced, but too new to trash)|
-|arvados_keep_overreplicated_{replicas,blocks,bytes}|gauge|overreplicated data (more replicas exist than are needed)|
-|arvados_keep_underreplicated_{replicas,blocks,bytes}|gauge|underreplicated data (fewer replicas exist than are needed)|
-|arvados_keep_lost_{replicas,blocks,bytes}|gauge|lost data (referenced by collections, but not found on any backend volume)|
-|arvados_keep_dedup_block_ratio|gauge|deduplication ratio (block references in collections ÷ distinct blocks referenced)|
-|arvados_keep_dedup_byte_ratio|gauge|deduplication ratio (block references in collections ÷ distinct blocks referenced, weighted by block size)|
-|arvados_keepbalance_get_state_seconds|summary|time to get all collections and keepstore volume indexes for one iteration|
-|arvados_keepbalance_changeset_compute_seconds|summary|time to compute changesets for one iteration|
-|arvados_keepbalance_send_pull_list_seconds|summary|time to send pull lists to all keepstore servers for one iteration|
-|arvados_keepbalance_send_trash_list_seconds|summary|time to send trash lists to all keepstore servers for one iteration|
-|arvados_keepbalance_sweep_seconds|summary|time to complete one iteration|
-
-Each @arvados_keep_@ storage state statistic above is presented as a set of three metrics:
-
-table(table table-bordered table-condensed).
-|*_blocks|distinct block hashes|
-|*_bytes|bytes stored on backend volumes|
-|*_replicas|objects/files stored on backend volumes|
+table(table table-bordered table-condensed table-hover).
+|_. Component|_. Metrics endpoint|
+|arvados-api-server||
+|arvados-controller|✓|
+|arvados-dispatch-cloud|✓|
+|arvados-git-httpd||
+|arvados-node-manager||
+|arvados-ws||
+|composer||
+|keepproxy||
+|keepstore|✓|
+|keep-balance|✓|
+|keep-web|✓|
+|sso-provider||
+|workbench1||
+|workbench2||
h2. Node manager
-The node manager status end point provides a snapshot of internal status at the time of the most recent wishlist update.
+The node manager does not export prometheus-style metrics, but its @/status.json@ endpoint provides a snapshot of internal status at the time of the most recent wishlist update.
+
+<pre>curl -sfH "Authorization: Bearer your_management_token_goes_here" "http://0.0.0.0:8989/status.json"
+</pre>
table(table table-bordered table-condensed).
|_. Attribute|_. Type|_. Description|
import (
"context"
+ "crypto/tls"
"encoding/json"
+ "io/ioutil"
"net/http"
"net/http/httptest"
"net/url"
c.Check(user.UUID, check.Equals, arvadostest.ActiveUserUUID)
c.Check(user.Authorization.TokenV2(), check.Equals, auth.TokenV2())
}
+
+func (s *HandlerSuite) CheckObjectType(c *check.C, url string, token string, skippedFields map[string]bool) {
+ var proxied, direct map[string]interface{}
+ var err error
+
+ // Get collection from controller
+ req := httptest.NewRequest("GET", url, nil)
+ req.Header.Set("Authorization", "Bearer "+token)
+ resp := httptest.NewRecorder()
+ s.handler.ServeHTTP(resp, req)
+ c.Assert(resp.Code, check.Equals, http.StatusOK,
+ check.Commentf("Wasn't able to get data from the controller at %q", url))
+ err = json.Unmarshal(resp.Body.Bytes(), &proxied)
+ c.Check(err, check.Equals, nil)
+
+ // Get collection directly from RailsAPI
+ client := &http.Client{
+ Transport: &http.Transport{
+ TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
+ },
+ }
+ resp2, err := client.Get(s.cluster.Services.RailsAPI.ExternalURL.String() + url + "/?api_token=" + token)
+ c.Check(err, check.Equals, nil)
+ defer resp2.Body.Close()
+ db, err := ioutil.ReadAll(resp2.Body)
+ c.Check(err, check.Equals, nil)
+ err = json.Unmarshal(db, &direct)
+ c.Check(err, check.Equals, nil)
+
+ // Check that all RailsAPI provided keys exist on the controller response.
+ for k := range direct {
+ if _, ok := skippedFields[k]; ok {
+ continue
+ } else if val, ok := proxied[k]; ok {
+ if direct["kind"] == "arvados#collection" && k == "manifest_text" {
+ // Tokens differ from request to request
+ c.Check(strings.Split(val.(string), "+A")[0], check.Equals, strings.Split(direct[k].(string), "+A")[0])
+ } else {
+ c.Check(val, check.DeepEquals, direct[k],
+ check.Commentf("RailsAPI %s key %q's value %q differs from controller's %q.", direct["kind"], k, direct[k], val))
+ }
+ } else {
+ c.Errorf("%s's key %q missing on controller's response.", direct["kind"], k)
+ }
+ }
+}
+
+func (s *HandlerSuite) TestGetObjects(c *check.C) {
+ // Get the 1st keep service's uuid from the running test server.
+ req := httptest.NewRequest("GET", "/arvados/v1/keep_services/", nil)
+ req.Header.Set("Authorization", "Bearer "+arvadostest.AdminToken)
+ resp := httptest.NewRecorder()
+ s.handler.ServeHTTP(resp, req)
+ c.Assert(resp.Code, check.Equals, http.StatusOK)
+ var ksList arvados.KeepServiceList
+ json.Unmarshal(resp.Body.Bytes(), &ksList)
+ c.Assert(len(ksList.Items), check.Not(check.Equals), 0)
+ ksUUID := ksList.Items[0].UUID
+
+ testCases := map[string]map[string]bool{
+ "api_clients/" + arvadostest.TrustedWorkbenchAPIClientUUID: nil,
+ "api_client_authorizations/" + arvadostest.AdminTokenUUID: nil,
+ "authorized_keys/" + arvadostest.AdminAuthorizedKeysUUID: nil,
+ "collections/" + arvadostest.CollectionWithUniqueWordsUUID: map[string]bool{"href": true},
+ "containers/" + arvadostest.RunningContainerUUID: nil,
+ "container_requests/" + arvadostest.QueuedContainerRequestUUID: nil,
+ "groups/" + arvadostest.AProjectUUID: nil,
+ "keep_services/" + ksUUID: nil,
+ "links/" + arvadostest.ActiveUserCanReadAllUsersLinkUUID: nil,
+ "logs/" + arvadostest.CrunchstatForRunningJobLogUUID: nil,
+ "nodes/" + arvadostest.IdleNodeUUID: nil,
+ "repositories/" + arvadostest.ArvadosRepoUUID: nil,
+ "users/" + arvadostest.ActiveUserUUID: map[string]bool{"href": true},
+ "virtual_machines/" + arvadostest.TestVMUUID: nil,
+ "workflows/" + arvadostest.WorkflowWithDefinitionYAMLUUID: nil,
+ }
+ for url, skippedFields := range testCases {
+ s.CheckObjectType(c, "/arvados/v1/"+url, arvadostest.AdminToken, skippedFields)
+ }
+}
for req in job_reqs:
tool.requirements.append(req)
- def arv_executor(self, tool, job_order, runtimeContext, logger=None):
+ def arv_executor(self, updated_tool, job_order, runtimeContext, logger=None):
self.debug = runtimeContext.debug
- tool.visit(self.check_features)
+ updated_tool.visit(self.check_features)
self.project_uuid = runtimeContext.project_uuid
self.pipeline = None
raise Exception("--submit-request-uuid requires containers API, but using '{}' api".format(self.work_api))
if not runtimeContext.name:
- runtimeContext.name = self.name = tool.tool.get("label") or tool.metadata.get("label") or os.path.basename(tool.tool["id"])
+ runtimeContext.name = self.name = updated_tool.tool.get("label") or updated_tool.metadata.get("label") or os.path.basename(updated_tool.tool["id"])
# Upload local file references in the job order.
job_order = upload_job_order(self, "%s input" % runtimeContext.name,
- tool, job_order)
+ updated_tool, job_order)
+
+ # the last clause means: if it is a command line tool, and we
+ # are going to wait for the result, and always_submit_runner
+ # is false, then we don't submit a runner process.
submitting = (runtimeContext.update_workflow or
runtimeContext.create_workflow or
(runtimeContext.submit and not
- (tool.tool["class"] == "CommandLineTool" and
+ (updated_tool.tool["class"] == "CommandLineTool" and
runtimeContext.wait and
not runtimeContext.always_submit_runner)))
if submitting:
# Document may have been auto-updated. Reload the original
# document with updating disabled because we want to
- # submit the original document, not the auto-updated one.
- tool = load_tool(tool.tool["id"], loadingContext)
+ # submit the document with its original CWL version, not
+ # the auto-updated one.
+ tool = load_tool(updated_tool.tool["id"], loadingContext)
+ else:
+ tool = updated_tool
# Upload direct dependencies of workflow steps, get back mapping of files to keep references.
# Also uploads docker images.
if runtimeContext.submit:
# Submit a runner job to run the workflow for us.
if self.work_api == "containers":
- if tool.tool["class"] == "CommandLineTool" and runtimeContext.wait and (not runtimeContext.always_submit_runner):
- runtimeContext.runnerjob = tool.tool["id"]
+ if submitting:
+ tool = RunnerContainer(self, updated_tool,
+ tool, loadingContext, runtimeContext.enable_reuse,
+ self.output_name,
+ self.output_tags,
+ submit_runner_ram=runtimeContext.submit_runner_ram,
+ name=runtimeContext.name,
+ on_error=runtimeContext.on_error,
+ submit_runner_image=runtimeContext.submit_runner_image,
+ intermediate_output_ttl=runtimeContext.intermediate_output_ttl,
+ merged_map=merged_map,
+ priority=runtimeContext.priority,
+ secret_store=self.secret_store,
+ collection_cache_size=runtimeContext.collection_cache_size,
+ collection_cache_is_default=self.should_estimate_cache_size)
else:
- tool = RunnerContainer(self, tool, loadingContext, runtimeContext.enable_reuse,
- self.output_name,
- self.output_tags,
- submit_runner_ram=runtimeContext.submit_runner_ram,
- name=runtimeContext.name,
- on_error=runtimeContext.on_error,
- submit_runner_image=runtimeContext.submit_runner_image,
- intermediate_output_ttl=runtimeContext.intermediate_output_ttl,
- merged_map=merged_map,
- priority=runtimeContext.priority,
- secret_store=self.secret_store,
- collection_cache_size=runtimeContext.collection_cache_size,
- collection_cache_is_default=self.should_estimate_cache_size)
+ runtimeContext.runnerjob = tool.tool["id"]
if runtimeContext.cwl_runner_job is not None:
self.uuid = runtimeContext.cwl_runner_job.get('uuid')
"""Base class for runner processes, which submit an instance of
arvados-cwl-runner and wait for the final result."""
- def __init__(self, runner, tool, loadingContext, enable_reuse,
+ def __init__(self, runner, updated_tool,
+ tool, loadingContext, enable_reuse,
output_name, output_tags, submit_runner_ram=0,
name=None, on_error=None, submit_runner_image=None,
intermediate_output_ttl=0, merged_map=None,
collection_cache_is_default=True):
loadingContext = loadingContext.copy()
- loadingContext.metadata = loadingContext.metadata.copy()
- loadingContext.metadata["cwlVersion"] = INTERNAL_VERSION
+ loadingContext.metadata = updated_tool.metadata.copy()
- super(Runner, self).__init__(tool.tool, loadingContext)
+ super(Runner, self).__init__(updated_tool.tool, loadingContext)
self.arvrunner = runner
self.embedded_tool = tool
],
test_suite='tests',
tests_require=[
- 'mock>=1.0',
+ 'mock>=1.0,<4',
'subprocess32>=3.5.1',
],
zip_safe=True,
stubs.expect_container_request_uuid + '\n')
self.assertEqual(exited, 0)
+
+ @stubs
+ def test_submit_container_tool(self, stubs):
+ # test for issue #16139
+ exited = arvados_cwl.main(
+ ["--submit", "--no-wait", "--api=containers", "--debug",
+ "tests/tool/tool_with_sf.cwl", "tests/tool/tool_with_sf.yml"],
+ stubs.capture_stdout, sys.stderr, api_client=stubs.api, keep_client=stubs.keep_client)
+
+ self.assertEqual(stubs.capture_stdout.getvalue(),
+ stubs.expect_container_request_uuid + '\n')
+ self.assertEqual(exited, 0)
+
@stubs
def test_submit_container_no_reuse(self, stubs):
exited = arvados_cwl.main(
--- /dev/null
+clipper clupper
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Test case for arvados-cwl-runner
+#
+# Used to test whether scanning a tool file for dependencies (e.g. default
+# value blub.txt) and uploading to Keep works as intended.
+
+class: CommandLineTool
+cwlVersion: v1.0
+requirements:
+ - class: DockerRequirement
+ dockerPull: debian:8
+inputs:
+ - id: x
+ type: File
+ secondaryFiles:
+ - .cat
+ inputBinding:
+ valueFrom: $(self.path).cat
+ position: 1
+outputs: []
+baseCommand: cat
--- /dev/null
+x:
+ class: File
+ location: blub.txt
ReplicationDesired *int `json:"replication_desired"`
StorageClassesDesired []string `json:"storage_classes_desired"`
StorageClassesConfirmed []string `json:"storage_classes_confirmed"`
- StorageClassesConfirmedAt time.Time `json:"storage_classes_confirmed_at"`
+ StorageClassesConfirmedAt *time.Time `json:"storage_classes_confirmed_at"`
DeleteAt *time.Time `json:"delete_at"`
IsTrashed bool `json:"is_trashed"`
Properties map[string]interface{} `json:"properties"`
WritableBy []string `json:"writable_by,omitempty"`
+ FileCount int `json:"file_count"`
+ FileSizeTotal int64 `json:"file_size_total"`
+ Version int `json:"version"`
+ PreserveVersion bool `json:"preserve_version"`
+ CurrentVersionUUID string `json:"current_version_uuid"`
+ Description string `json:"description"`
}
func (c Collection) resourceName() string {
ActiveTokenUUID = "zzzzz-gj3su-077z32aux8dg2s1"
ActiveTokenV2 = "v2/zzzzz-gj3su-077z32aux8dg2s1/3kg6k6lzmp9kj5cpkcoxie963cmvjahbt2fod9zru30k1jqdmi"
AdminToken = "4axaw8zxe0qm22wa6urpp5nskcne8z88cvbupv653y1njyi05h"
+ AdminTokenUUID = "zzzzz-gj3su-027z32aux8dg2s1"
AnonymousToken = "4kg6k6lzmp9kj4cpkcoxie964cmvjahbt4fod9zru44k4jqdmi"
DataManagerToken = "320mkve8qkswstz7ff61glpk3mhgghmg67wmic7elw4z41pke1"
SystemRootToken = "systemusertesttoken1234567890aoeuidhtnsqjkxbmwvzpy"
WorkflowWithDefinitionYAMLUUID = "zzzzz-7fd4e-validworkfloyml"
CollectionReplicationDesired2Confirmed2UUID = "zzzzz-4zz18-434zv1tnnf2rygp"
+
+ ActiveUserCanReadAllUsersLinkUUID = "zzzzz-o0j2j-ctbysaduejxfrs5"
+
+ TrustedWorkbenchAPIClientUUID = "zzzzz-ozdt8-teyxzyd8qllg11h"
+
+ AdminAuthorizedKeysUUID = "zzzzz-fngyi-12nc9ov4osp8nae"
+
+ CrunchstatForRunningJobLogUUID = "zzzzz-57u5n-tmymyrojrbtnxh1"
+
+ IdleNodeUUID = "zzzzz-7ekkf-2z3mc76g2q73aio"
+
+ TestVMUUID = "zzzzz-2x53u-382brsig8rp3064"
+
+ CollectionWithUniqueWordsUUID = "zzzzz-4zz18-mnt690klmb51aud"
)
// PathologicalManifest : A valid manifest designed to test
'Programming Language :: Python :: 3',
],
test_suite='tests',
- tests_require=['pbr<1.7.0', 'mock>=1.0', 'PyYAML'],
+ tests_require=['pbr<1.7.0', 'mock>=1.0,<4', 'PyYAML'],
zip_safe=False
)
# (same order as Container#handle_completed). Locking always
# reloads the Container and ContainerRequest records.
c = Container.find_by_uuid(container_uuid)
- c.lock!
+ c.lock! if !c.nil?
self.lock!
- if container_uuid != c.uuid
+ if !c.nil? && container_uuid != c.uuid
# After locking, we've noticed a race, the container_uuid is
# different than the container record we just loaded. This
# can happen if Container#handle_completed scheduled a new
redo
end
- if state == Committed && c.final?
- # The current container is
- act_as_system_user do
- leave_modified_by_user_alone do
- finalize!
+ if !c.nil?
+ if state == Committed && c.final?
+ # The current container is
+ act_as_system_user do
+ leave_modified_by_user_alone do
+ finalize!
+ end
end
end
+ elsif state == Committed
+ # Behave as if the container is cancelled
+ update_attributes!(state: Final)
end
return true
end
# finished/cancelled.
def finalize!
container = Container.find_by_uuid(container_uuid)
- update_collections(container: container)
-
- if container.state == Container::Complete
- log_col = Collection.where(portable_data_hash: container.log).first
- if log_col
- # Need to save collection
- completed_coll = Collection.new(
- owner_uuid: self.owner_uuid,
- name: "Container log for container #{container_uuid}",
- properties: {
- 'type' => 'log',
- 'container_request' => self.uuid,
- 'container_uuid' => container_uuid,
- },
- portable_data_hash: log_col.portable_data_hash,
- manifest_text: log_col.manifest_text)
- completed_coll.save_with_unique_name!
+ if !container.nil?
+ update_collections(container: container)
+
+ if container.state == Container::Complete
+ log_col = Collection.where(portable_data_hash: container.log).first
+ if log_col
+ # Need to save collection
+ completed_coll = Collection.new(
+ owner_uuid: self.owner_uuid,
+ name: "Container log for container #{container_uuid}",
+ properties: {
+ 'type' => 'log',
+ 'container_request' => self.uuid,
+ 'container_uuid' => container_uuid,
+ },
+ portable_data_hash: log_col.portable_data_hash,
+ manifest_text: log_col.manifest_text)
+ completed_coll.save_with_unique_name!
+ end
end
end
-
update_attributes!(state: Final)
end
collections.each do |out_type|
pdh = container.send(out_type)
next if pdh.nil?
+ c = Collection.where(portable_data_hash: pdh).first
+ next if c.nil?
+ manifest = c.manifest_text
+
coll_name = "Container #{out_type} for request #{uuid}"
trash_at = nil
if out_type == 'output'
trash_at = db_current_time + self.output_ttl
end
end
- manifest = Collection.where(portable_data_hash: pdh).first.manifest_text
coll_uuid = self.send(out_type + '_uuid')
coll = coll_uuid.nil? ? nil : Collection.where(uuid: coll_uuid).first
if self.container_count_changed?
errors.add :container_count, "cannot be updated directly."
return false
- else
- self.container_count += 1
- if self.container_uuid_was
- old_container = Container.find_by_uuid(self.container_uuid_was)
- old_logs = Collection.where(portable_data_hash: old_container.log).first
- if old_logs
- log_coll = self.log_uuid.nil? ? nil : Collection.where(uuid: self.log_uuid).first
- if self.log_uuid.nil?
- log_coll = Collection.new(
- owner_uuid: self.owner_uuid,
- name: coll_name = "Container log for request #{uuid}",
- manifest_text: "")
- end
+ end
- # copy logs from old container into CR's log collection
- src = Arv::Collection.new(old_logs.manifest_text)
- dst = Arv::Collection.new(log_coll.manifest_text)
- dst.cp_r("./", "log for container #{old_container.uuid}", src)
- manifest = dst.manifest_text
-
- log_coll.assign_attributes(
- portable_data_hash: Digest::MD5.hexdigest(manifest) + '+' + manifest.bytesize.to_s,
- manifest_text: manifest)
- log_coll.save_with_unique_name!
- self.log_uuid = log_coll.uuid
- end
- end
+ self.container_count += 1
+ return if self.container_uuid_was.nil?
+
+ old_container = Container.find_by_uuid(self.container_uuid_was)
+ return if old_container.nil?
+
+ old_logs = Collection.where(portable_data_hash: old_container.log).first
+ return if old_logs.nil?
+
+ log_coll = self.log_uuid.nil? ? nil : Collection.where(uuid: self.log_uuid).first
+ if self.log_uuid.nil?
+ log_coll = Collection.new(
+ owner_uuid: self.owner_uuid,
+ name: coll_name = "Container log for request #{uuid}",
+ manifest_text: "")
end
+
+ # copy logs from old container into CR's log collection
+ src = Arv::Collection.new(old_logs.manifest_text)
+ dst = Arv::Collection.new(log_coll.manifest_text)
+ dst.cp_r("./", "log for container #{old_container.uuid}", src)
+ manifest = dst.manifest_text
+
+ log_coll.assign_attributes(
+ portable_data_hash: Digest::MD5.hexdigest(manifest) + '+' + manifest.bytesize.to_s,
+ manifest_text: manifest)
+ log_coll.save_with_unique_name!
+ self.log_uuid = log_coll.uuid
end
end
owner_uuid: zzzzz-tpzed-000000000000000
uuid: zzzzz-tpzed-xurymjxw79nv3jz
email: active-user@arvados.local
+ modified_by_client_uuid: zzzzz-ozdt8-teyxzyd8qllg11h
+ modified_by_user_uuid: zzzzz-tpzed-xurymjxw79nv3jz
first_name: Active
last_name: User
identity_url: https://active-user.openid.local
is_active: true
is_admin: false
+ modified_at: 2015-03-26 12:34:56.789000000 Z
username: active
prefs:
profile:
assert_equal log.owner_uuid, project.uuid, "Container log should be copied to #{project.uuid}"
end
+ # This tests bug report #16144
+ test "Request is finalized when its container is completed even when log & output don't exist" do
+ set_user_from_auth :active
+ project = groups(:private)
+ cr = create_minimal_req!(owner_uuid: project.uuid,
+ priority: 1,
+ state: "Committed")
+ assert_equal users(:active).uuid, cr.modified_by_user_uuid
+
+ output_pdh = '1f4b0bc7583c2a7f9102c395f4ffc5e3+45'
+ log_pdh = 'fa7aeb5140e2848d39b416daeef4ffc5+45'
+
+ c = act_as_system_user do
+ c = Container.find_by_uuid(cr.container_uuid)
+ c.update_attributes!(state: Container::Locked)
+ c.update_attributes!(state: Container::Running,
+ output: output_pdh,
+ log: log_pdh)
+ c
+ end
+
+ cr.reload
+ assert_equal "Committed", cr.state
+
+ act_as_system_user do
+ Collection.where(portable_data_hash: output_pdh).delete_all
+ Collection.where(portable_data_hash: log_pdh).delete_all
+ c.update_attributes!(state: Container::Complete)
+ end
+
+ cr.reload
+ assert_equal "Final", cr.state
+ end
+
+ # This tests bug report #16144
+ test "Can destroy CR even if its container doesn't exist" do
+ set_user_from_auth :active
+ project = groups(:private)
+ cr = create_minimal_req!(owner_uuid: project.uuid,
+ priority: 1,
+ state: "Committed")
+ assert_equal users(:active).uuid, cr.modified_by_user_uuid
+
+ c = act_as_system_user do
+ c = Container.find_by_uuid(cr.container_uuid)
+ c.update_attributes!(state: Container::Locked)
+ c.update_attributes!(state: Container::Running)
+ c
+ end
+
+ cr.reload
+ assert_equal "Committed", cr.state
+
+ cr_uuid = cr.uuid
+ act_as_system_user do
+ Container.find_by_uuid(cr.container_uuid).destroy
+ cr.destroy
+ end
+ assert_nil ContainerRequest.find_by_uuid(cr_uuid)
+ end
+
test "Container makes container request, then is cancelled" do
set_user_from_auth :active
cr = create_minimal_req!(priority: 5, state: "Committed", container_count_max: 1)
"flag"
"fmt"
"io"
+ "net/http"
"os"
"git.arvados.org/arvados.git/lib/config"
options.Dumper = dumper
}
- // Only pass along the version flag, which gets handled in RunCommand
+ // Drop our custom args that would be rejected by the generic
+ // service.Command
args = nil
+ dropFlag := map[string]bool{
+ "once": true,
+ "commit-pulls": true,
+ "commit-trash": true,
+ "dump": true,
+ }
flags.Visit(func(f *flag.Flag) {
- if f.Name == "version" {
+ if !dropFlag[f.Name] {
args = append(args, "-"+f.Name, f.Value.String())
}
})
}
srv := &Server{
+ Handler: http.NotFoundHandler(),
Cluster: cluster,
ArvClient: ac,
RunOptions: options,
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: AGPL-3.0
+
+package main
+
+import (
+ "bytes"
+ "io/ioutil"
+ "net"
+ "net/http"
+ "time"
+
+ check "gopkg.in/check.v1"
+)
+
+var _ = check.Suite(&mainSuite{})
+
+type mainSuite struct{}
+
+func (s *mainSuite) TestVersionFlag(c *check.C) {
+ var stdout, stderr bytes.Buffer
+ runCommand("keep-balance", []string{"-version"}, nil, &stdout, &stderr)
+ c.Check(stderr.String(), check.Equals, "")
+ c.Log(stdout.String())
+}
+
+func (s *mainSuite) TestHTTPServer(c *check.C) {
+ ln, err := net.Listen("tcp", ":0")
+ if err != nil {
+ c.Fatal(err)
+ }
+ _, p, err := net.SplitHostPort(ln.Addr().String())
+ ln.Close()
+ config := "Clusters:\n zzzzz:\n ManagementToken: abcdefg\n Services: {Keepbalance: {InternalURLs: {'http://localhost:" + p + "/': {}}}}\n"
+
+ var stdout bytes.Buffer
+ go runCommand("keep-balance", []string{"-config", "-"}, bytes.NewBufferString(config), &stdout, &stdout)
+ done := make(chan struct{})
+ go func() {
+ defer close(done)
+ for {
+ time.Sleep(time.Second / 10)
+ req, err := http.NewRequest(http.MethodGet, "http://:"+p+"/metrics", nil)
+ if err != nil {
+ c.Fatal(err)
+ return
+ }
+ req.Header.Set("Authorization", "Bearer abcdefg")
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ c.Logf("error %s", err)
+ continue
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ c.Logf("http status %d", resp.StatusCode)
+ continue
+ }
+ buf, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ c.Logf("read body: %s", err)
+ continue
+ }
+ c.Check(string(buf), check.Matches, `(?ms).*arvados_keepbalance_sweep_seconds_sum.*`)
+ return
+ }
+ }()
+ select {
+ case <-done:
+ case <-time.After(time.Second):
+ c.Log(stdout.String())
+ c.Fatal("timeout")
+ }
+
+ // Check non-metrics URL that gets passed through to us from
+ // service.Command
+ req, err := http.NewRequest(http.MethodGet, "http://:"+p+"/not-metrics", nil)
+ c.Assert(err, check.IsNil)
+ resp, err := http.DefaultClient.Do(req)
+ c.Check(err, check.IsNil)
+ defer resp.Body.Close()
+ c.Check(resp.StatusCode, check.Equals, http.StatusNotFound)
+}