From: Tom Clegg Date: Mon, 21 Mar 2022 14:13:57 +0000 (-0400) Subject: Merge branch '18732-crunch-run-log-uids' X-Git-Tag: 2.4.0~37 X-Git-Url: https://git.arvados.org/arvados.git/commitdiff_plain/af448be33f2c5ddde29f2809093722f4ee67a112?hp=fa8710171bbf2175bce1367bf2a44f8fde7832db Merge branch '18732-crunch-run-log-uids' refs #18732 Arvados-DCO-1.1-Signed-off-by: Tom Clegg --- diff --git a/doc/admin/spot-instances.html.textile.liquid b/doc/admin/spot-instances.html.textile.liquid index 7ca57df0ab..3837f30d6d 100644 --- a/doc/admin/spot-instances.html.textile.liquid +++ b/doc/admin/spot-instances.html.textile.liquid @@ -16,13 +16,11 @@ Currently Arvados supports preemptible instances using AWS and Azure spot instan h2. Configuration -First, ensure automatic selection of preemptible instances is not disabled in your configuration file (this is enabled by default, but can be disabled with @AlwaysUsePreemptibleInstances: false@), and add entries to @InstanceTypes@ that have @Preemptible: true@. Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The @Price@ for preemptible instances is the maximum bid price, the actual price paid is dynamic and will likely be lower. For example: +Add entries to @InstanceTypes@ that have @Preemptible: true@. Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The @Price@ for preemptible instances is the maximum bid price, the actual price paid is dynamic and will likely be lower. For example: 
 Clusters:
-  ClusterID: 
-    Containers:
-      AlwaysUsePreemptibleInstances: true
+  ClusterID:
     InstanceTypes:
       m4.large:
         Preemptible: false
@@ -40,7 +38,18 @@ Clusters:
         Price: 0.1
-When @AlwaysUsePreemptibleInstances@ is enabled, child containers (workflow steps) will automatically be made preemptible. Note that because preempting the workflow runner would cancel the entire workflow, the workflow runner runs in a reserved (non-preemptible) instance. +Next, you can choose to enable automatic use of preemptible instances: + +
+    Containers:
+      AlwaysUsePreemptibleInstances: true
+
+ +If @AlwaysUsePreemptibleInstances@ is "true", child containers (workflow steps) will always select preemptible instances, regardless of user option. + +If @AlwaysUsePreemptibleInstances@ is "false" (the default) or unspecified, preemptible instance are "used when requested by the user.":{{site.baseurl}}/user/cwl/cwl-run-options.html#preemptible + +Note that regardless of the value of @AlwaysUsePreemptibleInstances@, the top level workflow runner container always runs in a reserved (non-preemptible) instance, to avoid situations where the workflow runner is killed requiring the entire to be restarted. No additional configuration is required, "arvados-dispatch-cloud":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html will now start preemptible instances where appropriate. diff --git a/doc/user/cwl/cwl-extensions.html.textile.liquid b/doc/user/cwl/cwl-extensions.html.textile.liquid index dd78e989fd..d6148d7eee 100644 --- a/doc/user/cwl/cwl-extensions.html.textile.liquid +++ b/doc/user/cwl/cwl-extensions.html.textile.liquid @@ -63,6 +63,9 @@ hints: cudaComputeCapabilityMin: "9.0" deviceCountMin: 1 deviceCountMax: 1 + + arv:UsePreemptible: + usePreemptible: true {% endcodeblock %} h2(#RunInSingleContainer). arv:RunInSingleContainer @@ -164,6 +167,14 @@ table(table table-bordered table-condensed). |deviceCountMin|integer|Minimum number of GPU devices to allocate on a single node. Required.| |deviceCountMax|integer|Maximum number of GPU devices to allocate on a single node. Optional. If not specified, same as @minDeviceCount@.| +h2(#UsePreemptible). arv:UsePreemptible + +Specify whether a workflow step should request preemptible (e.g. AWS Spot market) instances. Such instances are generally cheaper, but can be taken back by the cloud provider at any time (preempted) causing the step to fail. When this happens, Arvados will automatically re-try the step, up to the configuration value of @Containers.MaxRetryAttempts@ (default 3) times. + +table(table table-bordered table-condensed). +|_. Field |_. Type |_. Description | +|usePreemptible|boolean|Required, true to opt-in to using preemptible instances, false to opt-out.| + h2. arv:dockerCollectionPDH This is an optional extension field appearing on the standard @DockerRequirement@. It specifies the portable data hash of the Arvados collection containing the Docker image. If present, it takes precedence over @dockerPull@ or @dockerImageId@. diff --git a/doc/user/cwl/cwl-run-options.html.textile.liquid b/doc/user/cwl/cwl-run-options.html.textile.liquid index d331dad871..94e46ae1bc 100644 --- a/doc/user/cwl/cwl-run-options.html.textile.liquid +++ b/doc/user/cwl/cwl-run-options.html.textile.liquid @@ -63,6 +63,9 @@ table(table table-bordered table-condensed). |==--priority== PRIORITY|Workflow priority (range 1..1000, higher has precedence over lower)| |==--thread-count== THREAD_COUNT|Number of threads to use for container submit and output collection.| |==--http-timeout== HTTP_TIMEOUT|API request timeout in seconds. Default is 300 seconds (5 minutes).| +|==--enable-preemptible==|Use preemptible instances. Control individual steps with "arv:UsePreemptible":cwl-extensions.html#UsePreemptible hint.| +|==--disable-preemptible==|Don't use preemptible instances.| +|==--skip-schemas==|Skip loading of extension schemas (the $schemas section).| |==--trash-intermediate==|Immediately trash intermediate outputs on workflow success.| |==--no-trash-intermediate==|Do not trash intermediate outputs (default).| @@ -143,3 +146,19 @@ Using @--intermediate-output-ttl@ without @--trash-intermediate@ means that inte h3(#federation). Run workflow on a remote federated cluster By default, the workflow runner will run on the local (home) cluster. Using @--submit-runner-cluster@ you can specify that the runner should be submitted to a remote federated cluster. When doing this, @--project-uuid@ should specify a project on that cluster. Steps making up the workflow will be submitted to the remote federated cluster by default, but the behavior of @arv:ClusterTarget@ is unchanged. Note: when using this option, any resources that need to be uploaded in order to run the workflow (such as files or Docker images) will be uploaded to the local (home) cluster, and streamed to the federated cluster on demand. + +h3(#preemptible). Using preemptible (spot) instances + +Preemptible instances typically offer lower cost computation with a tradeoff of lower service guarantees. If a compute node is preempted, Arvados will restart the computation on a new instance. + +If the sitewide configuration @Containers.AlwaysUsePreemptibleInstances@ is true, workflow steps will always select preemptible instances, regardless of user option. + +If @Containers.AlwaysUsePreemptibleInstances@ is false, you can request preemptible instances for a specific run with the @arvados-cwl-runner --enable-preemptible@ option. + +Within the workflow, you can control whether individual steps should be preemptible with the "arv:UsePreemptible":cwl-extensions.html#UsePreemptible hint. + +If a workflow requests preemptible instances with "arv:UsePreemptible":cwl-extensions.html#UsePreemptible , but you _do not_ want to use preemptible instances, you can override it for a specific run with the @arvados-cwl-runner --disable-preemptible@ option. + +h3(#gpu). Use CUDA GPU instances + +See "cwltool:CUDARequirement":cwl-extensions.html#CUDARequirement . diff --git a/doc/user/cwl/cwl-style.html.textile.liquid b/doc/user/cwl/cwl-style.html.textile.liquid index 853ed3b3e2..303ae37e9e 100644 --- a/doc/user/cwl/cwl-style.html.textile.liquid +++ b/doc/user/cwl/cwl-style.html.textile.liquid @@ -13,7 +13,15 @@ h2(#performance). Performance To get the best perfomance from your workflows, be aware of the following Arvados features, behaviors, and best practices. -Does your application support NVIDIA GPU acceleration? Use "cwltool:CUDARequirement":cwl-extensions.html#CUDARequirement to request nodes with GPUs. +h3. Does your application support NVIDIA GPU acceleration? + +Use "cwltool:CUDARequirement":cwl-extensions.html#CUDARequirement to request nodes with GPUs. + +h3. Trying to reduce costs? + +Try "using preemptible (spot) instances":cwl-run-options.html#preemptible . + +h3. You have a sequence of short-running steps If you have a sequence of short-running steps (less than 1-2 minutes each), use the Arvados extension "arv:RunInSingleContainer":cwl-extensions.html#RunInSingleContainer to avoid scheduling and data transfer overhead by running all the steps together in the same container on the same node. To use this feature, @cwltool@ must be installed in the container image. Example: @@ -42,10 +50,16 @@ steps: run: subworkflow-with-short-steps.cwl {% endcodeblock %} +h3. Avoid declaring @InlineJavascriptRequirement@ or @ShellCommandRequirement@ + Avoid declaring @InlineJavascriptRequirement@ or @ShellCommandRequirement@ unless you specifically need them. Don't include them "just in case" because they change the default behavior and may add extra overhead. +h3. Prefer text substitution to Javascript + When combining a parameter value with a string, such as adding a filename extension, write @$(inputs.file.basename).ext@ instead of @$(inputs.file.basename + 'ext')@. The first form is evaluated as a simple text substitution, the second form (using the @+@ operator) is evaluated as an arbitrary Javascript expression and requires that you declare @InlineJavascriptRequirement@. +h3. Use @ExpressionTool@ to efficiently rearrange input files + Use @ExpressionTool@ to efficiently rearrange input files between steps of a Workflow. For example, the following expression accepts a directory containing files paired by @_R1_@ and @_R2_@ and produces an array of Directories containing each pair. {% codeblock as yaml %} @@ -80,9 +94,13 @@ expression: | } {% endcodeblock %} -Available compute nodes types vary over time and across different cloud providers, so try to limit the RAM requirement to what the program actually needs. However, if you need to target a specific compute node type, see this discussion on "calculating RAM request and choosing instance type for containers.":{{site.baseurl}}/api/execution.html#RAM +h3. Limit RAM requests to what you really need + +Available compute nodes types vary over time and across different cloud providers, so it is important to limit the RAM requirement to what the program actually needs. However, if you need to target a specific compute node type, see this discussion on "calculating RAM request and choosing instance type for containers.":{{site.baseurl}}/api/execution.html#RAM -Instead of scattering separate steps, prefer to scatter over a subworkflow. +h3. Avoid scattering by step by step + +Instead of a scatter step that feeds into another scatter step, prefer to scatter over a subworkflow. With the following pattern, @step1@ has to wait for all samples to complete before @step2@ can start computing on any samples. This means a single long-running sample can prevent the rest of the workflow from moving on: @@ -148,10 +166,16 @@ h2. Portability To write workflows that are easy to modify and portable across CWL runners (in the event you need to share your workflow with others), there are several best practices to follow: +h3. Always provide @DockerRequirement@ + Workflows should always provide @DockerRequirement@ in the @hints@ or @requirements@ section. +h3. Build a reusable library of components + Build a reusable library of components. Share tool wrappers and subworkflows between projects. Make use of and contribute to "community maintained workflows and tools":https://github.com/common-workflow-library and tool registries such as "Dockstore":http://dockstore.org . +h3. Supply scripts as input parameters + CommandLineTools wrapping custom scripts should represent the script as an input parameter with the script file as a default value. Use @secondaryFiles@ for scripts that consist of multiple files. For example: {% codeblock as yaml %} @@ -180,22 +204,21 @@ outputs: glob: "*.fastq" {% endcodeblock %} +h3. Getting the temporary and output directories + You can get the designated temporary directory using @$(runtime.tmpdir)@ in your CWL file, or from the @$TMPDIR@ environment variable in your script. Similarly, you can get the designated output directory using $(runtime.outdir), or from the @HOME@ environment variable in your script. -Avoid specifying resource requirements in CommandLineTool. Prefer to specify them in the workflow. You can provide a default resource requirement in the top level @hints@ section, and individual steps can override it with their own resource requirement. +h3. Specifying @ResourceRequirement@ + +Avoid specifying resources in the @requirements@ section of a @CommandLineTool@, put it in the @hints@ section instead. This enables you to override the tool resource hint with a workflow step level requirement: {% codeblock as yaml %} cwlVersion: v1.0 class: Workflow inputs: inp: File -hints: - ResourceRequirement: - ramMin: 1000 - coresMin: 1 - tmpdirMin: 45000 steps: step1: in: {inp: inp} @@ -205,7 +228,7 @@ steps: in: {inp: step1/inp} out: [out] run: tool2.cwl - hints: + requirements: ResourceRequirement: ramMin: 2000 coresMin: 2 diff --git a/lib/config/config.default.yml b/lib/config/config.default.yml index 9800be7047..0a8f55244b 100644 --- a/lib/config/config.default.yml +++ b/lib/config/config.default.yml @@ -903,11 +903,6 @@ Clusters: # If false, containers are scheduled on preemptible instances # only when requested by the submitter. # - # Note that arvados-cwl-runner does not currently offer a - # feature to request preemptible instances, so this value - # effectively acts as a cluster-wide decision about whether to - # use preemptible instances. - # # This flag is ignored if no preemptible instance types are # configured, and has no effect on top-level containers. AlwaysUsePreemptibleInstances: true diff --git a/sdk/cwl/arvados_cwl/__init__.py b/sdk/cwl/arvados_cwl/__init__.py index 826467cc09..c73b358ecc 100644 --- a/sdk/cwl/arvados_cwl/__init__.py +++ b/sdk/cwl/arvados_cwl/__init__.py @@ -213,6 +213,10 @@ def arg_parser(): # type: () -> argparse.ArgumentParser parser.add_argument("--http-timeout", type=int, default=5*60, dest="http_timeout", help="API request timeout in seconds. Default is 300 seconds (5 minutes).") + exgroup = parser.add_mutually_exclusive_group() + exgroup.add_argument("--enable-preemptible", dest="enable_preemptible", default=None, action="store_true", help="Use preemptible instances. Control individual steps with arv:UsePreemptible hint.") + exgroup.add_argument("--disable-preemptible", dest="enable_preemptible", default=None, action="store_false", help="Don't use preemptible instances.") + parser.add_argument( "--skip-schemas", action="store_true", @@ -255,7 +259,8 @@ def add_arv_hints(): "http://arvados.org/cwl#ClusterTarget", "http://arvados.org/cwl#OutputStorageClass", "http://arvados.org/cwl#ProcessProperties", - "http://commonwl.org/cwltool#CUDARequirement" + "http://commonwl.org/cwltool#CUDARequirement", + "http://arvados.org/cwl#UsePreemptible", ]) def exit_signal_handler(sigcode, frame): diff --git a/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.0.yml b/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.0.yml index 6e2d4f1d92..af75481431 100644 --- a/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.0.yml +++ b/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.0.yml @@ -385,3 +385,18 @@ $graph: doc: | Maximum number of GPU devices to request. If not specified, same as `cudaDeviceCountMin`. + +- name: UsePreemptible + type: record + extends: cwl:ProcessRequirement + inVocab: false + doc: | + Specify a workflow step should opt-in or opt-out of using preemptible (spot) instances. + fields: + class: + type: string + doc: "Always 'arv:UsePreemptible" + jsonldPredicate: + _id: "@type" + _type: "@vocab" + usePreemptible: boolean diff --git a/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.1.yml b/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.1.yml index 0e81347d72..0ae451ccaa 100644 --- a/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.1.yml +++ b/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.1.yml @@ -328,3 +328,18 @@ $graph: doc: | Maximum number of GPU devices to request. If not specified, same as `cudaDeviceCountMin`. + +- name: UsePreemptible + type: record + extends: cwl:ProcessRequirement + inVocab: false + doc: | + Specify a workflow step should opt-in or opt-out of using preemptible (spot) instances. + fields: + class: + type: string + doc: "Always 'arv:UsePreemptible" + jsonldPredicate: + _id: "@type" + _type: "@vocab" + usePreemptible: boolean diff --git a/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.2.yml b/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.2.yml index e9f70bf1cf..de5e55ca01 100644 --- a/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.2.yml +++ b/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.2.yml @@ -330,3 +330,18 @@ $graph: doc: | Maximum number of GPU devices to request. If not specified, same as `cudaDeviceCountMin`. + +- name: UsePreemptible + type: record + extends: cwl:ProcessRequirement + inVocab: false + doc: | + Specify a workflow step should opt-in or opt-out of using preemptible (spot) instances. + fields: + class: + type: string + doc: "Always 'arv:UsePreemptible" + jsonldPredicate: + _id: "@type" + _type: "@vocab" + usePreemptible: boolean diff --git a/sdk/cwl/arvados_cwl/arvcontainer.py b/sdk/cwl/arvados_cwl/arvcontainer.py index 2a5ff3a13a..8c468dd22d 100644 --- a/sdk/cwl/arvados_cwl/arvcontainer.py +++ b/sdk/cwl/arvados_cwl/arvcontainer.py @@ -300,6 +300,17 @@ class ArvadosContainer(JobBase): "hardware_capability": aslist(cuda_req["cudaComputeCapability"])[0] } + if runtimeContext.enable_preemptible is False: + scheduling_parameters["preemptible"] = False + else: + preemptible_req, _ = self.get_requirement("http://arvados.org/cwl#UsePreemptible") + if preemptible_req: + scheduling_parameters["preemptible"] = preemptible_req["usePreemptible"] + elif runtimeContext.enable_preemptible is True: + scheduling_parameters["preemptible"] = True + elif runtimeContext.enable_preemptible is None: + pass + if self.timelimit is not None and self.timelimit > 0: scheduling_parameters["max_run_time"] = self.timelimit @@ -550,6 +561,12 @@ class RunnerContainer(Runner): if self.enable_dev: command.append("--enable-dev") + if runtimeContext.enable_preemptible is True: + command.append("--enable-preemptible") + + if runtimeContext.enable_preemptible is False: + command.append("--disable-preemptible") + command.extend([workflowpath, "/var/lib/cwl/cwl.input.json"]) container_req["command"] = command diff --git a/sdk/cwl/arvados_cwl/context.py b/sdk/cwl/arvados_cwl/context.py index 4239dd3b51..316250106b 100644 --- a/sdk/cwl/arvados_cwl/context.py +++ b/sdk/cwl/arvados_cwl/context.py @@ -37,6 +37,7 @@ class ArvRuntimeContext(RuntimeContext): self.always_submit_runner = False self.collection_cache_size = 256 self.match_local_docker = False + self.enable_preemptible = None super(ArvRuntimeContext, self).__init__(kwargs) diff --git a/sdk/cwl/arvados_cwl/runner.py b/sdk/cwl/arvados_cwl/runner.py index ad17950a2f..38e2c4d806 100644 --- a/sdk/cwl/arvados_cwl/runner.py +++ b/sdk/cwl/arvados_cwl/runner.py @@ -40,7 +40,7 @@ import schema_salad.validate as validate import arvados.collection from .util import collectionUUID -import ruamel.yaml as yaml +from ruamel.yaml import YAML from ruamel.yaml.comments import CommentedMap, CommentedSeq import arvados_cwl.arvdocker @@ -265,7 +265,8 @@ def upload_dependencies(arvrunner, name, document_loader, textIO = StringIO(text.decode('utf-8')) else: textIO = StringIO(text) - return yaml.safe_load(textIO) + yamlloader = YAML(typ='safe', pure=True) + return yamlloader.load(textIO) else: return {} diff --git a/sdk/cwl/tests/test_container.py b/sdk/cwl/tests/test_container.py index 3de90c8d88..798c5af289 100644 --- a/sdk/cwl/tests/test_container.py +++ b/sdk/cwl/tests/test_container.py @@ -1234,6 +1234,103 @@ class TestContainer(unittest.TestCase): body=JsonDiffMatcher(container_request)) + # The test passes no builder.resources + # Hence the default resources will apply: {'cores': 1, 'ram': 1024, 'outdirSize': 1024, 'tmpdirSize': 1024} + @mock.patch("arvados.commands.keepdocker.list_images_in_arv") + def test_run_preemptible_hint(self, keepdocker): + arvados_cwl.add_arv_hints() + for enable_preemptible in (None, True, False): + for preemptible_hint in (None, True, False): + arv_docker_clear_cache() + + runner = mock.MagicMock() + runner.ignore_docker_for_reuse = False + runner.intermediate_output_ttl = 0 + runner.secret_store = cwltool.secrets.SecretStore() + runner.api._rootDesc = {"revision": "20210628"} + + keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")] + runner.api.collections().get().execute.return_value = { + "portable_data_hash": "99999999999999999999999999999993+99"} + + if preemptible_hint is not None: + hints = [{ + "class": "http://arvados.org/cwl#UsePreemptible", + "usePreemptible": preemptible_hint + }] + else: + hints = [] + + tool = cmap({ + "inputs": [], + "outputs": [], + "baseCommand": "ls", + "arguments": [{"valueFrom": "$(runtime.outdir)"}], + "id": "", + "class": "CommandLineTool", + "cwlVersion": "v1.2", + "hints": hints + }) + + loadingContext, runtimeContext = self.helper(runner) + + runtimeContext.name = 'test_run_enable_preemptible_'+str(enable_preemptible)+str(preemptible_hint) + runtimeContext.enable_preemptible = enable_preemptible + + arvtool = cwltool.load_tool.load_tool(tool, loadingContext) + arvtool.formatgraph = None + + # Test the interactions between --enable/disable-preemptible + # and UsePreemptible hint + + if enable_preemptible is None: + if preemptible_hint is None: + sched = {} + else: + sched = {'preemptible': preemptible_hint} + else: + if preemptible_hint is None: + sched = {'preemptible': enable_preemptible} + else: + sched = {'preemptible': enable_preemptible and preemptible_hint} + + for j in arvtool.job({}, mock.MagicMock(), runtimeContext): + j.run(runtimeContext) + runner.api.container_requests().create.assert_called_with( + body=JsonDiffMatcher({ + 'environment': { + 'HOME': '/var/spool/cwl', + 'TMPDIR': '/tmp' + }, + 'name': runtimeContext.name, + 'runtime_constraints': { + 'vcpus': 1, + 'ram': 268435456 + }, + 'use_existing': True, + 'priority': 500, + 'mounts': { + '/tmp': {'kind': 'tmp', + "capacity": 1073741824 + }, + '/var/spool/cwl': {'kind': 'tmp', + "capacity": 1073741824 } + }, + 'state': 'Committed', + 'output_name': 'Output for step '+runtimeContext.name, + 'owner_uuid': 'zzzzz-8i9sb-zzzzzzzzzzzzzzz', + 'output_path': '/var/spool/cwl', + 'output_ttl': 0, + 'container_image': '99999999999999999999999999999993+99', + 'command': ['ls', '/var/spool/cwl'], + 'cwd': '/var/spool/cwl', + 'scheduling_parameters': sched, + 'properties': {}, + 'secret_mounts': {}, + 'output_storage_classes': ["default"] + })) + + class TestWorkflow(unittest.TestCase): def setUp(self): diff --git a/sdk/cwl/tests/test_submit.py b/sdk/cwl/tests/test_submit.py index 10443359b9..61892bf2a4 100644 --- a/sdk/cwl/tests/test_submit.py +++ b/sdk/cwl/tests/test_submit.py @@ -1468,6 +1468,49 @@ class TestSubmit(unittest.TestCase): self.assertEqual(exited, 0) + @stubs + def test_submit_enable_preemptible(self, stubs): + exited = arvados_cwl.main( + ["--submit", "--no-wait", "--api=containers", "--debug", "--enable-preemptible", + "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"], + stubs.capture_stdout, sys.stderr, api_client=stubs.api, keep_client=stubs.keep_client) + + expect_container = copy.deepcopy(stubs.expect_container_spec) + expect_container['command'] = ['arvados-cwl-runner', '--local', '--api=containers', + '--no-log-timestamps', '--disable-validate', '--disable-color', + '--eval-timeout=20', '--thread-count=0', + '--enable-reuse', "--collection-cache-size=256", '--debug', '--on-error=continue', + '--enable-preemptible', + '/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json'] + + stubs.api.container_requests().create.assert_called_with( + body=JsonDiffMatcher(expect_container)) + self.assertEqual(stubs.capture_stdout.getvalue(), + stubs.expect_container_request_uuid + '\n') + self.assertEqual(exited, 0) + + @stubs + def test_submit_disable_preemptible(self, stubs): + exited = arvados_cwl.main( + ["--submit", "--no-wait", "--api=containers", "--debug", "--disable-preemptible", + "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"], + stubs.capture_stdout, sys.stderr, api_client=stubs.api, keep_client=stubs.keep_client) + + expect_container = copy.deepcopy(stubs.expect_container_spec) + expect_container['command'] = ['arvados-cwl-runner', '--local', '--api=containers', + '--no-log-timestamps', '--disable-validate', '--disable-color', + '--eval-timeout=20', '--thread-count=0', + '--enable-reuse', "--collection-cache-size=256", '--debug', '--on-error=continue', + '--disable-preemptible', + '/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json'] + + stubs.api.container_requests().create.assert_called_with( + body=JsonDiffMatcher(expect_container)) + self.assertEqual(stubs.capture_stdout.getvalue(), + stubs.expect_container_request_uuid + '\n') + self.assertEqual(exited, 0) + + class TestCreateWorkflow(unittest.TestCase): existing_workflow_uuid = "zzzzz-7fd4e-validworkfloyml" expect_workflow = StripYAMLComments( diff --git a/tools/arvbash/arvbash.sh b/tools/arvbash/arvbash.sh index ecad0888df..1d4fbade8b 100755 --- a/tools/arvbash/arvbash.sh +++ b/tools/arvbash/arvbash.sh @@ -15,10 +15,10 @@ Syntax: arvswitch Set ARVADOS_API_HOST and ARVADOS_API_TOKEN in the current environment based on $HOME/.config/arvados/.conf - With no arguments, list available Arvados configurations. + With no arguments, print current API host and available Arvados configurations. arvsave - Save values of ARVADOS_API_HOST and ARVADOS_API_TOKEN in the current environment to + Save current values of ARVADOS_API_HOST and ARVADOS_API_TOKEN in the current environment to $HOME/.config/arvados/.conf arvrm @@ -26,12 +26,12 @@ arvrm arvboxswitch Set ARVBOX_CONTAINER to - With no arguments, list available arvboxes. + With no arguments, print current arvbox and available arvboxes. -arvopen: +arvopen Open an Arvados uuid in web browser (http://arvadosapi.com) -arvissue +arvissue Open an Arvados ticket in web browser (http://dev.arvados.org) EOF @@ -61,7 +61,8 @@ arvswitch() { fi else echo "Switch Arvados environment conf" - echo "Usage: arvswitch name" + echo "Current host: ${ARVADOS_API_HOST}" + echo "Usage: arvswitch " echo "Available confs:" $((cd $HOME/.config/arvados && ls --indicator-style=none *.conf) | rev | cut -c6- | rev) fi } @@ -73,7 +74,7 @@ arvsave() { env | grep ARVADOS_ > $HOME/.config/arvados/$1.conf else echo "Save current Arvados environment variables to conf file" - echo "Usage: arvsave name" + echo "Usage: arvsave " fi } @@ -86,25 +87,25 @@ arvrm() { fi else echo "Delete Arvados environment conf" - echo "Usage: arvrm name" + echo "Usage: arvrm " fi } arvboxswitch() { if [[ -n "$1" ]] ; then + export ARVBOX_CONTAINER=$1 if [[ -d $HOME/.arvbox/$1 ]] ; then - export ARVBOX_CONTAINER=$1 echo "Arvbox switched to $1" else - echo "$1 unknown" + echo "Warning: $1 doesn't exist, will be created." fi else if test -z "$ARVBOX_CONTAINER" ; then ARVBOX_CONTAINER=arvbox fi echo "Switch Arvbox environment conf" - echo "Usage: arvboxswitch name" echo "Your current container is: $ARVBOX_CONTAINER" + echo "Usage: arvboxswitch " echo "Available confs:" $(cd $HOME/.arvbox && ls --indicator-style=none) fi } @@ -114,7 +115,7 @@ arvopen() { xdg-open https://arvadosapi.com/$1 else echo "Open Arvados uuid in browser" - echo "Usage: arvopen uuid" + echo "Usage: arvopen " fi } @@ -123,6 +124,6 @@ arvissue() { xdg-open https://dev.arvados.org/issues/$1 else echo "Open Arvados issue in browser" - echo "Usage: arvissue uuid" + echo "Usage: arvissue " fi } diff --git a/tools/arvbox/lib/arvbox/docker/createusers.sh b/tools/arvbox/lib/arvbox/docker/createusers.sh index 9c81a66ced..4cafd8c09c 100755 --- a/tools/arvbox/lib/arvbox/docker/createusers.sh +++ b/tools/arvbox/lib/arvbox/docker/createusers.sh @@ -34,20 +34,12 @@ if ! grep "^arvbox:" /etc/passwd >/dev/null 2>/dev/null ; then chown arvbox:arvbox -R /usr/local $ARVADOS_CONTAINER_PATH \ /var/lib/passenger /var/lib/postgresql \ /var/lib/nginx /var/log/nginx /etc/ssl/private \ - /var/lib/gopath /var/lib/pip /var/lib/npm \ - /var/lib/arvados + /var/lib/gopath /var/lib/pip /var/lib/npm fi mkdir -p /tmp/crunch0 /tmp/crunch1 chown crunch:crunch -R /tmp/crunch0 /tmp/crunch1 - # singularity needs to be owned by root and suid - chown root /var/lib/arvados/bin/singularity \ - /var/lib/arvados/etc/singularity/singularity.conf \ - /var/lib/arvados/etc/singularity/capability.json \ - /var/lib/arvados/etc/singularity/ecl.toml - chmod u+s /var/lib/arvados/bin/singularity - echo "arvbox ALL=(crunch) NOPASSWD: ALL" >> /etc/sudoers cat < /etc/profile.d/paths.sh