X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/36cfafd6e7eae2784c22aefdd9df26783412d42a..544f1dceeb4e5fcc63b683d3edac48e62bca5958:/doc/user/cwl/cwl-extensions.html.textile.liquid

diff --git a/doc/user/cwl/cwl-extensions.html.textile.liquid b/doc/user/cwl/cwl-extensions.html.textile.liquid
index 197816f4a4..e05072ddf6 100644
--- a/doc/user/cwl/cwl-extensions.html.textile.liquid
+++ b/doc/user/cwl/cwl-extensions.html.textile.liquid
@@ -71,6 +71,10 @@ hints:
 
   arv:UsePreemptible:
     usePreemptible: true
+
+  arv:OutOfMemoryRetry:
+    memoryRetryMultipler: 2
+    memoryErrorRegex: "custom memory error"
 {% endcodeblock %}
 
 h2(#RunInSingleContainer). arv:RunInSingleContainer
@@ -188,6 +192,20 @@ table(table table-bordered table-condensed).
 |_. Field |_. Type |_. Description |
 |usePreemptible|boolean|Required, true to opt-in to using preemptible instances, false to opt-out.|
 
+
+h2(#OutOfMemoryRetry). arv:OutOfMemoryRetry
+
+Specify that when a workflow step appears to have failed because it did not request enough RAM, it should be re-submitted with more RAM.  Out of memory conditions are detected either by the container being unexpectedly killed (exit code 137) or by matching a pattern in the container's output (see @memoryErrorRegex@).  Retrying will increase the base RAM request by the value of @memoryRetryMultipler@.  For example, if the original RAM request was 10 GiB and the multiplier is 1.5, then it will re-submit with 15 GiB.
+
+Containers are only re-submitted once.  If it fails a second time after increasing RAM, then the worklow step will still fail.
+
+Also note that expressions that use @$(runtime.ram)@ (such as dynamic command line parameters) are not reevaluated when the container is resubmitted.
+
+table(table table-bordered table-condensed).
+|_. Field |_. Type |_. Description |
+|memoryRetryMultipler|float|Required, the retry will multiply the base memory request by this factor to get the retry memory request.|
+|memoryErrorRegex|string|Optional, a custom regex that, if found in the stdout, stderr or crunch-run logging of a program, will trigger a retry with greater RAM.  If not provided, the default pattern matches "out of memory" (with or without spaces), "memory error" (with or without spaces), "bad_alloc" and "container using over 90% of memory".|
+
 h2. arv:dockerCollectionPDH
 
 This is an optional extension field appearing on the standard @DockerRequirement@.  It specifies the portable data hash of the Arvados collection containing the Docker image.  If present, it takes precedence over @dockerPull@ or @dockerImageId@.