20637: Move the prefetch thread pool to KeepClient
authorPeter Amstutz <peter.amstutz@curii.com>
Fri, 16 Jun 2023 18:35:20 +0000 (14:35 -0400)
committerPeter Amstutz <peter.amstutz@curii.com>
Fri, 16 Jun 2023 19:03:06 +0000 (15:03 -0400)
commitcdf63c1c50a044fe66f224b5cf36b632ecff12a5
tree10da42f0650b6bc5a6dcbb749e63dcaae4e57a57
parent9075ef17087f7b3bdce308bfa5d60b4fe3863b51
20637: Move the prefetch thread pool to KeepClient

This avoids the problem of every Collection with its own BlockManager
creating its own prefetch thread pool, which becomes a resource leak
when reading files from 1000s of separate Collection objects.

The 'put' thread pool remains with the BlockManager but it now stops
the put threads on 'BlockManager.commit_all'.  This is because this
method always flushes pending blocks anyway, and is called before the
collection record is written to the API server -- so we can assume
we've just finished a batch of writes to that collection, and might
not need the put thread pool any more, and if we do, we can just make
a new one.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
sdk/python/arvados/arvfile.py
sdk/python/arvados/collection.py
sdk/python/arvados/keep.py
services/fuse/arvados_fuse/command.py
services/fuse/arvados_fuse/fusedir.py