sdk/python/tests/fed-migrate/*.cwlex
doc/install/*.xlsx
sdk/cwl/tests/wf/hello.txt
-sdk/cwl/tests/wf/indir1/hello2.txt
\ No newline at end of file
+sdk/cwl/tests/wf/indir1/hello2.txt
+sdk/cwl/tests/chipseq/data/Genomes/*
\ No newline at end of file
The Controller exposes a subset of the cluster's configuration and makes it available to clients in JSON format. This public config includes valuable information like several service's URLs, timeout settings, etc. and it is available at @/arvados/v1/config@, for example @https://{{ site.arvados_api_host }}/arvados/v1/config@. The new Workbench is one example of a client using this information, as it's a client-side application and doesn't have access to the cluster's config file.
+h2. Exported vocabulary definition
+
+When configured, the Controller also exports the "metadata vocabulary definition":{{site.baseurl}}/admin/metadata-vocabulary.html in JSON format. This functionality is useful for clients like Workbench2 and the Python SDK to provide "identifier to human-readable labels" translations facilities for reading and writing objects on the system. This is available at @/arvados/v1/vocabulary@, for example @https://{{ site.arvados_api_host }}/arvados/v1/vocabulary@.
+
h2. Workbench examples
Many Arvados Workbench pages, under the *Advanced* tab, provide examples of API and SDK use for accessing the current resource .
for c in arvados.util.keyset_list_all(api.collections().list, filters=[["name", "like", "%sample123%"]]):
print("got collection " + c["uuid"])
{% endcodeblock %}
+
+h2. Querying the vocabulary definition
+
+The Python SDK provides facilities to interact with the "active metadata vocabulary":{{ site.baseurl }}/admin/metadata-vocabulary.html in the system. The developer can do key and value lookups in a case-insensitive manner:
+
+{% codeblock as python %}
+from arvados import api, vocabulary
+voc = vocabulary.load_vocabulary(api('v1'))
+
+[k.identifier for k in set(voc.key_aliases.values())]
+# Example output: ['IDTAGCOLORS', 'IDTAGFRUITS', 'IDTAGCOMMENT', 'IDTAGIMPORTANCES', 'IDTAGCATEGORIES', 'IDTAGSIZES', 'IDTAGANIMALS']
+voc['IDTAGSIZES'].preferred_label
+# Example output: 'Size'
+[v.preferred_label for v in set(voc['size'].value_aliases.values())]
+# Example output: ['S', 'M', 'L', 'XL', 'XS']
+voc['size']['s'].aliases
+# Example output: ['S', 'small']
+voc['size']['Small'].identifier
+# Example output: 'IDVALSIZES2'
+{% endcodeblock %}
+
+h2. Translating between vocabulary identifiers and labels
+
+Client software might need to present properties to the user in a human-readable form or take input from the user without requiring them to remember identifiers. For these cases, there're a couple of conversion methods that take a dictionary as input like this:
+
+{% codeblock as python %}
+from arvados import api, vocabulary
+voc = vocabulary.load_vocabulary(api('v1'))
+
+voc.convert_to_labels({'IDTAGIMPORTANCES': 'IDVALIMPORTANCES1'})
+# Example output: {'Importance': 'Critical'}
+voc.convert_to_identifiers({'creature': 'elephant'})
+# Example output: {'IDTAGANIMALS': 'IDVALANIMALS3'}
+{% endcodeblock %}
\ No newline at end of file
See https://docs.nvidia.com/deploy/cuda-compatibility/ for
details.
- cudaComputeCapabilityMin:
- type: string
- doc: Minimum CUDA hardware capability required to run the software, in X.Y format.
- deviceCountMin:
- type: int?
+ cudaComputeCapability:
+ type:
+ - 'string'
+ - 'string[]'
+ doc: |
+ CUDA hardware capability required to run the software, in X.Y
+ format.
+
+ * If this is a single value, it defines only the minimum
+ compute capability. GPUs with higher capability are also
+ accepted.
+
+ * If it is an array value, then only select GPUs with compute
+ capabilities that explicitly appear in the array.
+ cudaDeviceCountMin:
+ type: ['null', int, cwl:Expression]
default: 1
- doc: Minimum number of GPU devices to request, default 1.
- deviceCountMax:
- type: int?
- doc: Maximum number of GPU devices to request. If not specified, same as `deviceCountMin`.
+ doc: |
+ Minimum number of GPU devices to request. If not specified,
+ same as `cudaDeviceCountMax`. If neither are specified,
+ default 1.
+ cudaDeviceCountMax:
+ type: ['null', int, cwl:Expression]
+ doc: |
+ Maximum number of GPU devices to request. If not specified,
+ same as `cudaDeviceCountMin`.
See https://docs.nvidia.com/deploy/cuda-compatibility/ for
details.
- cudaComputeCapabilityMin:
- type: string
- doc: Minimum CUDA hardware capability required to run the software, in X.Y format.
- deviceCountMin:
- type: int?
+ cudaComputeCapability:
+ type:
+ - 'string'
+ - 'string[]'
+ doc: |
+ CUDA hardware capability required to run the software, in X.Y
+ format.
+
+ * If this is a single value, it defines only the minimum
+ compute capability. GPUs with higher capability are also
+ accepted.
+
+ * If it is an array value, then only select GPUs with compute
+ capabilities that explicitly appear in the array.
+ cudaDeviceCountMin:
+ type: ['null', int, cwl:Expression]
default: 1
- doc: Minimum number of GPU devices to request, default 1.
- deviceCountMax:
- type: int?
- doc: Maximum number of GPU devices to request. If not specified, same as `deviceCountMin`.
+ doc: |
+ Minimum number of GPU devices to request. If not specified,
+ same as `cudaDeviceCountMax`. If neither are specified,
+ default 1.
+ cudaDeviceCountMax:
+ type: ['null', int, cwl:Expression]
+ doc: |
+ Maximum number of GPU devices to request. If not specified,
+ same as `cudaDeviceCountMin`.
See https://docs.nvidia.com/deploy/cuda-compatibility/ for
details.
- cudaComputeCapabilityMin:
- type: string
- doc: Minimum CUDA hardware capability required to run the software, in X.Y format.
- deviceCountMin:
- type: int?
+ cudaComputeCapability:
+ type:
+ - 'string'
+ - 'string[]'
+ doc: |
+ CUDA hardware capability required to run the software, in X.Y
+ format.
+
+ * If this is a single value, it defines only the minimum
+ compute capability. GPUs with higher capability are also
+ accepted.
+
+ * If it is an array value, then only select GPUs with compute
+ capabilities that explicitly appear in the array.
+ cudaDeviceCountMin:
+ type: ['null', int, cwl:Expression]
default: 1
- doc: Minimum number of GPU devices to request, default 1.
- deviceCountMax:
- type: int?
- doc: Maximum number of GPU devices to request. If not specified, same as `deviceCountMin`.
+ doc: |
+ Minimum number of GPU devices to request. If not specified,
+ same as `cudaDeviceCountMax`. If neither are specified,
+ default 1.
+ cudaDeviceCountMax:
+ type: ['null', int, cwl:Expression]
+ doc: |
+ Maximum number of GPU devices to request. If not specified,
+ same as `cudaDeviceCountMin`.
cuda_req, _ = self.get_requirement("http://commonwl.org/cwltool#CUDARequirement")
if cuda_req:
runtime_constraints["cuda"] = {
- "device_count": cuda_req.get("deviceCountMin", 1),
+ "device_count": resources.get("cudaDeviceCount", 1),
"driver_version": cuda_req["cudaVersionMin"],
- "hardware_capability": cuda_req["cudaComputeCapabilityMin"]
+ "hardware_capability": aslist(cuda_req["cudaComputeCapability"])[0]
}
if self.timelimit is not None and self.timelimit > 0:
# file to determine what version of cwltool and schema-salad to
# build.
install_requires=[
- 'cwltool==3.1.20220217222804',
+ 'cwltool==3.1.20220224085855',
'schema-salad==8.2.20211116214159',
'arvados-python-client{}'.format(pysdk_dep),
'setuptools',
--- /dev/null
+{
+ "referenceGenomeSequence": {
+ "class": "File",
+ "location": "data/Genomes/Homo_sapiens/GRCh38.p2/WholeGenome/genome.fa",
+ "metadata": {
+ "reference_genome": {
+ "organism": "Homo sapiens",
+ "version": "hg38"
+ },
+ "annotation": {
+ "source": "gencode",
+ "version": "v24"
+ }
+ }
+ },
+ "referenceGenomeSequenceDrosophila": {
+ "class": "File",
+ "location": "data/Genomes/Drosophila_melanogaster/dmel_r6.16/WholeGenome/genome.fa",
+ "metadata": {
+ "reference_genome": {
+ "organism": "Drosophila melanogaster",
+ "version": "rmel_r6.16"
+ }
+ }
+ },
+ "blacklistBed": {
+ "class": "File",
+ "location": "data/Genomes/Blacklist/lists2/hg38-blacklist.v2.bed",
+ "metadata": {
+ "reference_genome": {
+ "organism": "Homo sapiens",
+ "version": "hg38"
+ },
+ "annotation": {
+ "source": "gencode",
+ "version": "v24"
+ }
+ }
+ },
+ "BowtieHumanReference": {
+ "class": "Directory",
+ "location": "data/Genomes/Homo_sapiens/GRCh38.p2/Bowtie2Index/",
+ "metadata": {
+ "reference_genome": {
+ "organism": "Homo sapiens",
+ "version": "hg38"
+ },
+ "annotation": {
+ "source": "gencode",
+ "version": "v24"
+ }
+ }
+ },
+ "BowtieDrosophilaReference": {
+ "class": "Directory",
+ "location": "data/Genomes/Drosophila_melanogaster/dmel_r6.16/Bowtie2Index/",
+ "metadata": {
+ "reference_genome": {
+ "organism": "Drosophila melanogaster",
+ "version": "rmel_r6.16"
+ }
+ }
+ },
+ "sampleName": "LED054_0p03nMR1.0",
+ "inputFastq1": {
+ "class": "File",
+ "metadata": {
+ "user": "kmavrommatis",
+ "sample_id": [
+ 2
+ ]
+ },
+ "location": "DATEST/ChIP-Seq/Raw/fastq/Input_R1.fastq.gz",
+ "secondaryFiles": []
+ },
+ "inputFastq2": {
+ "class": "File",
+ "metadata": {
+ "user": "kmavrommatis",
+ "sample_id": [
+ 2
+ ]
+ },
+ "location": "DATEST/ChIP-Seq/Raw/fastq/Input_R3.fastq.gz",
+ "secondaryFiles": []
+ },
+ "inputFastqUMI": {
+ "class": "File",
+ "metadata": {
+ "user": "kmavrommatis",
+ "sample_id": [
+ 2
+ ]
+ },
+ "location": "DATEST/ChIP-Seq/Raw/fastq/Input_R2.fastq.gz",
+ "secondaryFiles": []
+ }
+}
+
--- /dev/null
+{
+ "$graph": [
+ {
+ "class": "Workflow",
+ "id": "#main",
+ "doc": "Pipeline that is applied on single ChIP-seq samples.\n\nStarts with QC on the reads and trimming (for adapters and based on quality)\n\nAligns to human genome and adds UMI\n\nAligns to Drosophila genome and counts the number of reads.\n\nAfter the alignment to human genome the files are filtered for duplicates, multimappers and alignments in black listed regions",
+ "label": "ChIP-Seq (single sample)",
+ "inputs": [
+ {
+ "id": "#inputFastq1",
+ "type": "File",
+ "https://www.sevenbridges.com/fileTypes": "fastq",
+ "https://www.sevenbridges.com/x": 0,
+ "https://www.sevenbridges.com/y": 1726.25
+ },
+ {
+ "id": "#blacklistBed",
+ "type": "File",
+ "https://www.sevenbridges.com/x": 746.4744873046875,
+ "https://www.sevenbridges.com/y": 1903.265625
+ },
+ {
+ "id": "#referenceGenomeSequence",
+ "type": "File",
+ "secondaryFiles": [
+ ".fai",
+ "^.dict"
+ ],
+ "https://www.sevenbridges.com/fileTypes": "fasta, fa",
+ "https://www.sevenbridges.com/x": 0,
+ "https://www.sevenbridges.com/y": 1405.203125
+ },
+ {
+ "id": "#sampleName",
+ "type": "string",
+ "https://www.sevenbridges.com/x": 0,
+ "https://www.sevenbridges.com/y": 1191.171875
+ },
+ {
+ "id": "#inputFastq2",
+ "type": [
+ "null",
+ "File"
+ ],
+ "https://www.sevenbridges.com/fileTypes": "fastq",
+ "https://www.sevenbridges.com/x": 0,
+ "https://www.sevenbridges.com/y": 1619.234375
+ },
+ {
+ "id": "#inputFastqUMI",
+ "type": "File",
+ "https://www.sevenbridges.com/x": 0,
+ "https://www.sevenbridges.com/y": 1512.21875
+ },
+ {
+ "id": "#BowtieHumanReference",
+ "type": "Directory",
+ "https://www.sevenbridges.com/x": 363.875,
+ "https://www.sevenbridges.com/y": 1519.21875
+ },
+ {
+ "id": "#BowtieDrosophilaReference",
+ "type": "Directory",
+ "https://www.sevenbridges.com/x": 363.875,
+ "https://www.sevenbridges.com/y": 1626.234375
+ },
+ {
+ "id": "#referenceGenomeSequenceDrosophila",
+ "type": "File",
+ "secondaryFiles": [
+ ".fai"
+ ],
+ "https://www.sevenbridges.com/x": 0,
+ "https://www.sevenbridges.com/y": 1298.1875
+ }
+ ],
+ "outputs": [
+ ],
+ "steps": [
+ {
+ "id": "#step1",
+ "in": {
+ "inp": "#inputFastq1"
+ },
+ "out": [],
+ "run": "../cat.cwl"
+ }
+ ],
+ "requirements": [
+ ]
+ },
+ ],
+ "cwlVersion": "v1.0"
+}
runner.api.collections().get().execute.return_value = {
"portable_data_hash": "99999999999999999999999999999993+99"}
- tool = cmap({
- "inputs": [],
- "outputs": [],
- "baseCommand": "nvidia-smi",
- "arguments": [],
- "id": "",
- "cwlVersion": "v1.2",
- "class": "CommandLineTool",
- "requirements": [
- {
+ test_cwl_req = [{
"class": "http://commonwl.org/cwltool#CUDARequirement",
"cudaVersionMin": "11.0",
- "cudaComputeCapabilityMin": "9.0",
- }
- ]
- })
+ "cudaComputeCapability": "9.0",
+ }, {
+ "class": "http://commonwl.org/cwltool#CUDARequirement",
+ "cudaVersionMin": "11.0",
+ "cudaComputeCapability": "9.0",
+ "cudaDeviceCountMin": 2
+ }, {
+ "class": "http://commonwl.org/cwltool#CUDARequirement",
+ "cudaVersionMin": "11.0",
+ "cudaComputeCapability": ["4.0", "5.0"],
+ "cudaDeviceCountMin": 2
+ }]
+
+ test_arv_req = [{
+ 'device_count': 1,
+ 'driver_version': "11.0",
+ 'hardware_capability': "9.0"
+ }, {
+ 'device_count': 2,
+ 'driver_version': "11.0",
+ 'hardware_capability': "9.0"
+ }, {
+ 'device_count': 2,
+ 'driver_version': "11.0",
+ 'hardware_capability': "4.0"
+ }]
+
+ for test_case in range(0, len(test_cwl_req)):
- loadingContext, runtimeContext = self.helper(runner, True)
+ tool = cmap({
+ "inputs": [],
+ "outputs": [],
+ "baseCommand": "nvidia-smi",
+ "arguments": [],
+ "id": "",
+ "cwlVersion": "v1.2",
+ "class": "CommandLineTool",
+ "requirements": [test_cwl_req[test_case]]
+ })
- arvtool = cwltool.load_tool.load_tool(tool, loadingContext)
- arvtool.formatgraph = None
+ loadingContext, runtimeContext = self.helper(runner, True)
- for j in arvtool.job({}, mock.MagicMock(), runtimeContext):
- j.run(runtimeContext)
- runner.api.container_requests().create.assert_called_with(
- body=JsonDiffMatcher({
- 'environment': {
- 'HOME': '/var/spool/cwl',
- 'TMPDIR': '/tmp'
- },
- 'name': 'test_run_True',
- 'runtime_constraints': {
- 'vcpus': 1,
- 'ram': 268435456,
- 'cuda': {
- 'device_count': 1,
- 'driver_version': "11.0",
- 'hardware_capability': "9.0"
- }
- },
- 'use_existing': True,
- 'priority': 500,
- 'mounts': {
- '/tmp': {'kind': 'tmp',
- "capacity": 1073741824
- },
- '/var/spool/cwl': {'kind': 'tmp',
- "capacity": 1073741824 }
- },
- 'state': 'Committed',
- 'output_name': 'Output for step test_run_True',
- 'owner_uuid': 'zzzzz-8i9sb-zzzzzzzzzzzzzzz',
- 'output_path': '/var/spool/cwl',
- 'output_ttl': 0,
- 'container_image': '99999999999999999999999999999993+99',
- 'command': ['nvidia-smi'],
- 'cwd': '/var/spool/cwl',
- 'scheduling_parameters': {},
- 'properties': {},
- 'secret_mounts': {},
- 'output_storage_classes': ["default"]
- }))
+ arvtool = cwltool.load_tool.load_tool(tool, loadingContext)
+ arvtool.formatgraph = None
+
+ for j in arvtool.job({}, mock.MagicMock(), runtimeContext):
+ j.run(runtimeContext)
+ runner.api.container_requests().create.assert_called_with(
+ body=JsonDiffMatcher({
+ 'environment': {
+ 'HOME': '/var/spool/cwl',
+ 'TMPDIR': '/tmp'
+ },
+ 'name': 'test_run_True' + ("" if test_case == 0 else "_"+str(test_case+1)),
+ 'runtime_constraints': {
+ 'vcpus': 1,
+ 'ram': 268435456,
+ 'cuda': test_arv_req[test_case]
+ },
+ 'use_existing': True,
+ 'priority': 500,
+ 'mounts': {
+ '/tmp': {'kind': 'tmp',
+ "capacity": 1073741824
+ },
+ '/var/spool/cwl': {'kind': 'tmp',
+ "capacity": 1073741824 }
+ },
+ 'state': 'Committed',
+ 'output_name': 'Output for step test_run_True' + ("" if test_case == 0 else "_"+str(test_case+1)),
+ 'owner_uuid': 'zzzzz-8i9sb-zzzzzzzzzzzzzzz',
+ 'output_path': '/var/spool/cwl',
+ 'output_ttl': 0,
+ 'container_image': '99999999999999999999999999999993+99',
+ 'command': ['nvidia-smi'],
+ 'cwd': '/var/spool/cwl',
+ 'scheduling_parameters': {},
+ 'properties': {},
+ 'secret_mounts': {},
+ 'output_storage_classes': ["default"]
+ }))
# The test passes no builder.resources
svc.insecure = insecure
svc.request_id = request_id
svc.config = lambda: util.get_config_once(svc)
+ svc.vocabulary = lambda: util.get_vocabulary_once(svc)
kwargs['http'].max_request_size = svc._rootDesc.get('maxRequestSize', 0)
kwargs['http'].cache = None
kwargs['http']._request_id = lambda: svc.request_id or util.new_request_id()
return (None, None)
fn = os.path.join(root, ".arvados#collection")
if os.path.exists(fn):
- with file(fn, 'r') as f:
+ with open(fn, 'r') as f:
c = json.load(f)
return (c["portable_data_hash"], branch)
else:
if not hasattr(svc, '_cached_config'):
svc._cached_config = svc.configs().get().execute()
return svc._cached_config
+
+def get_vocabulary_once(svc):
+ if not svc._rootDesc.get('resources').get('vocabularies', False):
+ # Old API server version, no vocabulary export endpoint
+ return {}
+ if not hasattr(svc, '_cached_vocabulary'):
+ svc._cached_vocabulary = svc.vocabularies().get().execute()
+ return svc._cached_vocabulary
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import logging
+
+from . import api
+
+_logger = logging.getLogger('arvados.vocabulary')
+
+def load_vocabulary(api_client=None):
+ """Load the Arvados vocabulary from the API.
+ """
+ if api_client is None:
+ api_client = api('v1')
+ return Vocabulary(api_client.vocabulary())
+
+class VocabularyError(Exception):
+ """Base class for all vocabulary errors.
+ """
+ pass
+
+class VocabularyKeyError(VocabularyError):
+ pass
+
+class VocabularyValueError(VocabularyError):
+ pass
+
+class Vocabulary(object):
+ def __init__(self, voc_definition={}):
+ self.strict_keys = voc_definition.get('strict_tags', False)
+ self.key_aliases = {}
+
+ for key_id, val in (voc_definition.get('tags') or {}).items():
+ strict = val.get('strict', False)
+ key_labels = [l['label'] for l in val.get('labels', [])]
+ values = {}
+ for v_id, v_val in (val.get('values') or {}).items():
+ labels = [l['label'] for l in v_val.get('labels', [])]
+ values[v_id] = VocabularyValue(v_id, labels)
+ vk = VocabularyKey(key_id, key_labels, values, strict)
+ self.key_aliases[key_id.lower()] = vk
+ for alias in vk.aliases:
+ self.key_aliases[alias.lower()] = vk
+
+ def __getitem__(self, key):
+ return self.key_aliases[key.lower()]
+
+ def convert_to_identifiers(self, obj={}):
+ """Translate key/value pairs to machine readable identifiers.
+ """
+ return self._convert_to_what(obj, 'identifier')
+
+ def convert_to_labels(self, obj={}):
+ """Translate key/value pairs to human readable labels.
+ """
+ return self._convert_to_what(obj, 'preferred_label')
+
+ def _convert_to_what(self, obj={}, what=None):
+ if not isinstance(obj, dict):
+ raise ValueError("obj must be a dict")
+ if what not in ['preferred_label', 'identifier']:
+ raise ValueError("what attr must be 'preferred_label' or 'identifier'")
+ r = {}
+ for k, v in obj.items():
+ # Key validation & lookup
+ key_found = False
+ if not isinstance(k, str):
+ raise VocabularyKeyError("key '{}' must be a string".format(k))
+ k_what, v_what = k, v
+ try:
+ k_what = getattr(self[k], what)
+ key_found = True
+ except KeyError:
+ if self.strict_keys:
+ raise VocabularyKeyError("key '{}' not found in vocabulary".format(k))
+
+ # Value validation & lookup
+ if isinstance(v, list):
+ v_what = []
+ for x in v:
+ if not isinstance(x, str):
+ raise VocabularyValueError("value '{}' for key '{}' must be a string".format(x, k))
+ try:
+ v_what.append(getattr(self[k][x], what))
+ except KeyError:
+ if self[k].strict:
+ raise VocabularyValueError("value '{}' not found for key '{}'".format(x, k))
+ v_what.append(x)
+ else:
+ if not isinstance(v, str):
+ raise VocabularyValueError("{} value '{}' for key '{}' must be a string".format(type(v).__name__, v, k))
+ try:
+ v_what = getattr(self[k][v], what)
+ except KeyError:
+ if key_found and self[k].strict:
+ raise VocabularyValueError("value '{}' not found for key '{}'".format(v, k))
+
+ r[k_what] = v_what
+ return r
+
+class VocabularyData(object):
+ def __init__(self, identifier, aliases=[]):
+ self.identifier = identifier
+ self.aliases = aliases
+
+ def __getattribute__(self, name):
+ if name == 'preferred_label':
+ return self.aliases[0]
+ return super(VocabularyData, self).__getattribute__(name)
+
+class VocabularyValue(VocabularyData):
+ def __init__(self, identifier, aliases=[]):
+ super(VocabularyValue, self).__init__(identifier, aliases)
+
+class VocabularyKey(VocabularyData):
+ def __init__(self, identifier, aliases=[], values={}, strict=False):
+ super(VocabularyKey, self).__init__(identifier, aliases)
+ self.strict = strict
+ self.value_aliases = {}
+ for v_id, v_val in values.items():
+ self.value_aliases[v_id.lower()] = v_val
+ for v_alias in v_val.aliases:
+ self.value_aliases[v_alias.lower()] = v_val
+
+ def __getitem__(self, key):
+ return self.value_aliases[key.lower()]
\ No newline at end of file
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import arvados
+import unittest
+import mock
+
+from arvados import api, vocabulary
+
+class VocabularyTest(unittest.TestCase):
+ EXAMPLE_VOC = {
+ 'tags': {
+ 'IDTAGANIMALS': {
+ 'strict': False,
+ 'labels': [
+ {'label': 'Animal'},
+ {'label': 'Creature'},
+ ],
+ 'values': {
+ 'IDVALANIMAL1': {
+ 'labels': [
+ {'label': 'Human'},
+ {'label': 'Homo sapiens'},
+ ],
+ },
+ 'IDVALANIMAL2': {
+ 'labels': [
+ {'label': 'Elephant'},
+ {'label': 'Loxodonta'},
+ ],
+ },
+ },
+ },
+ 'IDTAGIMPORTANCES': {
+ 'strict': True,
+ 'labels': [
+ {'label': 'Importance'},
+ {'label': 'Priority'},
+ ],
+ 'values': {
+ 'IDVALIMPORTANCE1': {
+ 'labels': [
+ {'label': 'High'},
+ {'label': 'High priority'},
+ ],
+ },
+ 'IDVALIMPORTANCE2': {
+ 'labels': [
+ {'label': 'Medium'},
+ {'label': 'Medium priority'},
+ ],
+ },
+ 'IDVALIMPORTANCE3': {
+ 'labels': [
+ {'label': 'Low'},
+ {'label': 'Low priority'},
+ ],
+ },
+ },
+ },
+ 'IDTAGCOMMENTS': {
+ 'strict': False,
+ 'labels': [
+ {'label': 'Comment'},
+ {'label': 'Notes'},
+ ],
+ 'values': None,
+ },
+ },
+ }
+
+ def setUp(self):
+ self.api = arvados.api('v1')
+ self.voc = vocabulary.Vocabulary(self.EXAMPLE_VOC)
+ self.api.vocabulary = mock.MagicMock(return_value=self.EXAMPLE_VOC)
+
+ def test_vocabulary_keys(self):
+ self.assertEqual(self.voc.strict_keys, False)
+ self.assertEqual(
+ self.voc.key_aliases.keys(),
+ set(['idtaganimals', 'creature', 'animal',
+ 'idtagimportances', 'importance', 'priority',
+ 'idtagcomments', 'comment', 'notes'])
+ )
+
+ vk = self.voc.key_aliases['creature']
+ self.assertEqual(vk.strict, False)
+ self.assertEqual(vk.identifier, 'IDTAGANIMALS')
+ self.assertEqual(vk.aliases, ['Animal', 'Creature'])
+ self.assertEqual(vk.preferred_label, 'Animal')
+ self.assertEqual(
+ vk.value_aliases.keys(),
+ set(['idvalanimal1', 'human', 'homo sapiens',
+ 'idvalanimal2', 'elephant', 'loxodonta'])
+ )
+
+ def test_vocabulary_values(self):
+ vk = self.voc.key_aliases['creature']
+ vv = vk.value_aliases['human']
+ self.assertEqual(vv.identifier, 'IDVALANIMAL1')
+ self.assertEqual(vv.aliases, ['Human', 'Homo sapiens'])
+ self.assertEqual(vv.preferred_label, 'Human')
+
+ def test_vocabulary_indexing(self):
+ self.assertEqual(self.voc['creature']['human'].identifier, 'IDVALANIMAL1')
+ self.assertEqual(self.voc['Creature']['Human'].identifier, 'IDVALANIMAL1')
+ self.assertEqual(self.voc['CREATURE']['HUMAN'].identifier, 'IDVALANIMAL1')
+ with self.assertRaises(KeyError):
+ inexistant = self.voc['foo']
+
+ def test_empty_vocabulary(self):
+ voc = vocabulary.Vocabulary({})
+ self.assertEqual(voc.strict_keys, False)
+ self.assertEqual(voc.key_aliases, {})
+
+ def test_load_vocabulary_with_api(self):
+ voc = vocabulary.load_vocabulary(self.api)
+ self.assertEqual(voc['creature']['human'].identifier, 'IDVALANIMAL1')
+ self.assertEqual(voc['Creature']['Human'].identifier, 'IDVALANIMAL1')
+ self.assertEqual(voc['CREATURE']['HUMAN'].identifier, 'IDVALANIMAL1')
+
+ def test_convert_to_identifiers(self):
+ cases = [
+ {'IDTAGIMPORTANCES': 'IDVALIMPORTANCE1'},
+ {'IDTAGIMPORTANCES': 'High'},
+ {'importance': 'IDVALIMPORTANCE1'},
+ {'priority': 'high priority'},
+ ]
+ for case in cases:
+ self.assertEqual(
+ self.voc.convert_to_identifiers(case),
+ {'IDTAGIMPORTANCES': 'IDVALIMPORTANCE1'},
+ "failing test case: {}".format(case)
+ )
+
+ def test_convert_to_identifiers_multiple_pairs(self):
+ cases = [
+ {'IDTAGIMPORTANCES': 'IDVALIMPORTANCE1', 'IDTAGANIMALS': 'IDVALANIMAL1', 'IDTAGCOMMENTS': 'Very important person'},
+ {'IDTAGIMPORTANCES': 'High', 'IDTAGANIMALS': 'IDVALANIMAL1', 'comment': 'Very important person'},
+ {'importance': 'IDVALIMPORTANCE1', 'animal': 'IDVALANIMAL1', 'notes': 'Very important person'},
+ {'priority': 'high priority', 'animal': 'IDVALANIMAL1', 'NOTES': 'Very important person'},
+ ]
+ for case in cases:
+ self.assertEqual(
+ self.voc.convert_to_identifiers(case),
+ {'IDTAGIMPORTANCES': 'IDVALIMPORTANCE1', 'IDTAGANIMALS': 'IDVALANIMAL1', 'IDTAGCOMMENTS': 'Very important person'},
+ "failing test case: {}".format(case)
+ )
+
+ def test_convert_to_identifiers_value_lists(self):
+ cases = [
+ {'IDTAGIMPORTANCES': ['IDVALIMPORTANCE1', 'IDVALIMPORTANCE2']},
+ {'IDTAGIMPORTANCES': ['High', 'Medium']},
+ {'importance': ['IDVALIMPORTANCE1', 'IDVALIMPORTANCE2']},
+ {'priority': ['high', 'medium']},
+ ]
+ for case in cases:
+ self.assertEqual(
+ self.voc.convert_to_identifiers(case),
+ {'IDTAGIMPORTANCES': ['IDVALIMPORTANCE1', 'IDVALIMPORTANCE2']},
+ "failing test case: {}".format(case)
+ )
+
+ def test_convert_to_identifiers_unknown_key(self):
+ # Non-strict vocabulary
+ self.assertEqual(self.voc.strict_keys, False)
+ self.assertEqual(self.voc.convert_to_identifiers({'foo': 'bar'}), {'foo': 'bar'})
+ # Strict vocabulary
+ strict_voc = arvados.vocabulary.Vocabulary(self.EXAMPLE_VOC)
+ strict_voc.strict_keys = True
+ with self.assertRaises(vocabulary.VocabularyKeyError):
+ strict_voc.convert_to_identifiers({'foo': 'bar'})
+
+ def test_convert_to_identifiers_invalid_key(self):
+ with self.assertRaises(vocabulary.VocabularyKeyError):
+ self.voc.convert_to_identifiers({42: 'bar'})
+ with self.assertRaises(vocabulary.VocabularyKeyError):
+ self.voc.convert_to_identifiers({None: 'bar'})
+ with self.assertRaises(vocabulary.VocabularyKeyError):
+ self.voc.convert_to_identifiers({('f', 'o', 'o'): 'bar'})
+
+ def test_convert_to_identifiers_unknown_value(self):
+ # Non-strict key
+ self.assertEqual(self.voc['animal'].strict, False)
+ self.assertEqual(self.voc.convert_to_identifiers({'Animal': 'foo'}), {'IDTAGANIMALS': 'foo'})
+ # Strict key
+ self.assertEqual(self.voc['priority'].strict, True)
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_identifiers({'Priority': 'foo'})
+
+ def test_convert_to_identifiers_invalid_value(self):
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_identifiers({'Animal': 42})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_identifiers({'Animal': None})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_identifiers({'Animal': {'hello': 'world'}})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_identifiers({'Animal': [42]})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_identifiers({'Animal': [None]})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_identifiers({'Animal': [{'hello': 'world'}]})
+
+ def test_convert_to_identifiers_unknown_value_list(self):
+ # Non-strict key
+ self.assertEqual(self.voc['animal'].strict, False)
+ self.assertEqual(
+ self.voc.convert_to_identifiers({'Animal': ['foo', 'loxodonta']}),
+ {'IDTAGANIMALS': ['foo', 'IDVALANIMAL2']}
+ )
+ # Strict key
+ self.assertEqual(self.voc['priority'].strict, True)
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_identifiers({'Priority': ['foo', 'bar']})
+
+ def test_convert_to_labels(self):
+ cases = [
+ {'IDTAGIMPORTANCES': 'IDVALIMPORTANCE1'},
+ {'IDTAGIMPORTANCES': 'High'},
+ {'importance': 'IDVALIMPORTANCE1'},
+ {'priority': 'high priority'},
+ ]
+ for case in cases:
+ self.assertEqual(
+ self.voc.convert_to_labels(case),
+ {'Importance': 'High'},
+ "failing test case: {}".format(case)
+ )
+
+ def test_convert_to_labels_multiple_pairs(self):
+ cases = [
+ {'IDTAGIMPORTANCES': 'IDVALIMPORTANCE1', 'IDTAGANIMALS': 'IDVALANIMAL1', 'IDTAGCOMMENTS': 'Very important person'},
+ {'IDTAGIMPORTANCES': 'High', 'IDTAGANIMALS': 'IDVALANIMAL1', 'comment': 'Very important person'},
+ {'importance': 'IDVALIMPORTANCE1', 'animal': 'IDVALANIMAL1', 'notes': 'Very important person'},
+ {'priority': 'high priority', 'animal': 'IDVALANIMAL1', 'NOTES': 'Very important person'},
+ ]
+ for case in cases:
+ self.assertEqual(
+ self.voc.convert_to_labels(case),
+ {'Importance': 'High', 'Animal': 'Human', 'Comment': 'Very important person'},
+ "failing test case: {}".format(case)
+ )
+
+ def test_convert_to_labels_value_lists(self):
+ cases = [
+ {'IDTAGIMPORTANCES': ['IDVALIMPORTANCE1', 'IDVALIMPORTANCE2']},
+ {'IDTAGIMPORTANCES': ['High', 'Medium']},
+ {'importance': ['IDVALIMPORTANCE1', 'IDVALIMPORTANCE2']},
+ {'priority': ['high', 'medium']},
+ ]
+ for case in cases:
+ self.assertEqual(
+ self.voc.convert_to_labels(case),
+ {'Importance': ['High', 'Medium']},
+ "failing test case: {}".format(case)
+ )
+
+ def test_convert_to_labels_unknown_key(self):
+ # Non-strict vocabulary
+ self.assertEqual(self.voc.strict_keys, False)
+ self.assertEqual(self.voc.convert_to_labels({'foo': 'bar'}), {'foo': 'bar'})
+ # Strict vocabulary
+ strict_voc = arvados.vocabulary.Vocabulary(self.EXAMPLE_VOC)
+ strict_voc.strict_keys = True
+ with self.assertRaises(vocabulary.VocabularyKeyError):
+ strict_voc.convert_to_labels({'foo': 'bar'})
+
+ def test_convert_to_labels_invalid_key(self):
+ with self.assertRaises(vocabulary.VocabularyKeyError):
+ self.voc.convert_to_labels({42: 'bar'})
+ with self.assertRaises(vocabulary.VocabularyKeyError):
+ self.voc.convert_to_labels({None: 'bar'})
+ with self.assertRaises(vocabulary.VocabularyKeyError):
+ self.voc.convert_to_labels({('f', 'o', 'o'): 'bar'})
+
+ def test_convert_to_labels_unknown_value(self):
+ # Non-strict key
+ self.assertEqual(self.voc['animal'].strict, False)
+ self.assertEqual(self.voc.convert_to_labels({'IDTAGANIMALS': 'foo'}), {'Animal': 'foo'})
+ # Strict key
+ self.assertEqual(self.voc['priority'].strict, True)
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': 'foo'})
+
+ def test_convert_to_labels_invalid_value(self):
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': {'high': True}})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': None})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': 42})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': False})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': [42]})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': [None]})
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': [{'high': True}]})
+
+ def test_convert_to_labels_unknown_value_list(self):
+ # Non-strict key
+ self.assertEqual(self.voc['animal'].strict, False)
+ self.assertEqual(
+ self.voc.convert_to_labels({'IDTAGANIMALS': ['foo', 'IDVALANIMAL1']}),
+ {'Animal': ['foo', 'Human']}
+ )
+ # Strict key
+ self.assertEqual(self.voc['priority'].strict, True)
+ with self.assertRaises(vocabulary.VocabularyValueError):
+ self.voc.convert_to_labels({'IDTAGIMPORTANCES': ['foo', 'bar']})
+
+ def test_convert_roundtrip(self):
+ initial = {'IDTAGIMPORTANCES': 'IDVALIMPORTANCE1', 'IDTAGANIMALS': 'IDVALANIMAL1', 'IDTAGCOMMENTS': 'Very important person'}
+ converted = self.voc.convert_to_labels(initial)
+ self.assertNotEqual(converted, initial)
+ self.assertEqual(self.voc.convert_to_identifiers(converted), initial)
# format is YYYYMMDD, must be fixed width (needs to be lexically
# sortable), updated manually, may be used by clients to
# determine availability of API server features.
- revision: "20210628",
+ revision: "20220222",
source_version: AppVersion.hash,
sourceVersion: AppVersion.hash, # source_version should be deprecated in the future
packageVersion: AppVersion.package_version,
}
}
+ discovery[:resources]['vocabularies'] = {
+ methods: {
+ get: {
+ id: "arvados.vocabularies.get",
+ path: "vocabulary",
+ httpMethod: "GET",
+ description: "Get vocabulary definition",
+ parameters: {
+ },
+ parameterOrder: [
+ ],
+ response: {
+ },
+ scopes: [
+ "https://api.arvados.org/auth/arvados",
+ "https://api.arvados.org/auth/arvados.readonly"
+ ]
+ },
+ }
+ }
+
discovery[:resources]['sys'] = {
methods: {
get: {
projects / arvados.git / commitdiff