From: Tom Clegg <tom@curii.com>
Date: Thu, 28 Oct 2021 14:14:09 +0000 (-0400)
Subject: Merge branch '16347-container-local-keepstore'
X-Git-Tag: 2.4.0~191
X-Git-Url: https://git.arvados.org/arvados.git/commitdiff_plain/f2788dd5fc3ef725969d6c0fcc4ddee3754829fc

Merge branch '16347-container-local-keepstore'

closes #16347

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
---

f2788dd5fc3ef725969d6c0fcc4ddee3754829fc
diff --cc doc/admin/upgrading.html.textile.liquid
index 304eedb1de,170e2c51b3..aee2b581b2
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@@ -35,9 -35,20 +35,20 @@@ TODO: extract this information based o
  <div class="releasenotes">
  </notextile>
  
+ h2(#main). development main (as of 2021-10-27)
+ 
 -"Upgrading from 2.3.0":#v2_3_0
++"previous: Upgrading from 2.3.0":#v2_3_0
+ 
+ h3. Dedicated keepstore process for each container
+ 
+ When Arvados runs a container via @arvados-dispatch-cloud@, the @crunch-run@ supervisor process now brings up its own keepstore server to handle I/O for mounted collections, outputs, and logs. With the default configuration, the keepstore process allocates one 64 MiB block buffer per VCPU requested by the container. For most workloads this will increase throughput, reduce total network traffic, and make it possible to run more containers at once without provisioning additional keepstore nodes to handle the I/O load.
+ * If you have containers that can effectively handle multiple I/O threads per VCPU, consider increasing the @Containers.LocalKeepBlobBuffersPerVCPU@ value.
+ * If you already have a robust permanent keepstore infrastructure, you can set @Containers.LocalKeepBlobBuffersPerVCPU@ to 0 to disable this feature and preserve the previous behavior of sending container I/O traffic to your separately provisioned keepstore servers.
+ * This feature is enabled only if no volumes use @AccessViaHosts@, and no volumes have underlying @Replication@ less than @Collections.DefaultReplication@. If the feature is configured but cannot be enabled due to an incompatible volume configuration, this will be noted in the @crunch-run.txt@ file in the container log.
+ 
  h2(#v2_3_0). v2.3.0 (2021-10-27)
  
 -"Upgrading from 2.2.0":#v2_2_0
 +"previous: Upgrading from 2.2.0":#v2_2_0
  
  h3. Ubuntu 18.04 packages for arvados-api-server and arvados-workbench now conflict with ruby-bundler