- Computation with Crunch:
- api/execution.html.textile.liquid
- architecture/dispatchcloud.html.textile.liquid
+ - architecture/singularity.html.textile.liquid
- Other:
- api/permission-model.html.textile.liquid
- architecture/federation.html.textile.liquid
<notextile>
<pre><code># <span class="userinput">apt-get --no-install-recommends install curl gnupg2</span>
-# <span class="userinput">curl https://apt.arvados.org/pubkey.gpg -o /etc/apt/trusted.gpg.d/arvados.asc</span>
+# <span class="userinput">curl -s https://apt.arvados.org/pubkey.gpg -o /etc/apt/trusted.gpg.d/arvados.asc</span>
</code></pre>
</notextile>
--- /dev/null
+---
+layout: default
+navsection: architecture
+title: Singularity
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+Arvados can be configured to use "Singularity":https://sylabs.io/singularity/ instead of Docker to execute containers on cloud nodes or a SLURM/LSF cluster. Singularity may be preferable due to its simpler installation and lack of long-running daemon process and special system users/groups. See the "Singularity page in the installation guide":{{ site.baseurl }}/install/singularity.html for configuration details.
+
+h2. Design overview
+
+When Arvados is configured to use Singularity as the runtime engine for Crunch, containers are executed by Singularity. The images specified in workflows and tool definitions must be Docker images uploaded via @arv-keepdocker@ or @arvados-cwl-runner@. When Singularity is the runtime engine, these images are converted to Singularity format (@.sif@) at runtime, as needed.
+
+To avoid repeating this conversion work unnecessarily, the @.sif@ files are cached in @Keep@. This is done on a per-user basis. If it does not exist yet, a new Arvados project named @.cache@ is automatically created in the user's home project. Similarly, a subproject named @auto-generated singularity images@ will be created in the @.cache@ project. The automatically generated @.sif@ files are stored in collections in that project, with an expiration date two weeks in the future. If the cached image exists when Crunch runs a new container, the expiration date will be pushed out, so that it is always 2 weeks in the future from the most recent start of a container using the image.
+
+It is safe to empty out or even remove the .cache project or any of its contents; if necessary the cache projects and the @.sif@ files will automatically be regenerated.
+
+h2. Notes
+
+* Programs running in Singularity containers may behave differently than when run in Docker, due to differences between Singularity and Docker. For example, the root (image) filesystem is read-only in a Singularity container. Programs that attempt to write outside a designated output or temporary directory are likely to fail.
+
+* When using Singularity as the runtime engine, the compute node needs to have a compatible Singularity executable installed, as well as the @mksquashfs@ program used to convert Docker images to Singularity's @.sif@ format. The Arvados "compute node image build script":{{ site.baseurl }}/install/crunch2-cloud/install-compute-node.html includes these executables since Arvados 2.3.0.
+
+h2. Limitations
+
+Arvados @Singularity@ support is a work in progress. These are the current limitations of the implementation:
+
+* Even when using the Singularity runtime, users' container images are expected to be saved in Docker format. Specifying a @.sif@ file as an image when submitting a container request is not yet supported.
+* Arvados' Singularity implementation does not yet limit the amount of memory available in a container. Each container will have access to all memory on the host where it runs, unless memory use is restricted by SLURM/LSF.
+* The Docker ENTRYPOINT instruction is ignored.
+* Arvados is tested with Singularity version 3.7.4. Other versions may not work.
This can be done via the "cloud console":https://console.cloud.google.com/kubernetes/ or via the command line:
<pre>
-$ gcloud container clusters create <CLUSTERNAME> --zone us-central1-a --machine-type n1-standard-2 --cluster-version 1.15
+$ gcloud container clusters create <CLUSTERNAME> --zone us-central1-a --machine-type n1-standard-2
</pre>
It takes a few minutes for the cluster to be initialized.
Arvados-dispatch-lsf reads the common configuration file at @/etc/arvados/config.yml@.
+Add a DispatchLSF entry to the Services section, using the hostname where @arvados-dispatch-lsf@ will run, and an available port:
+
+<notextile>
+<pre> Services:
+ DispatchLSF:
+ InternalURLs:
+ "http://<code class="userinput">hostname.zzzzz.arvadosapi.com:9007</code>": {}</pre>
+</notextile>
+
Review the following configuration parameters and adjust as needed.
h3(#BsubSudoUser). Containers.LSF.BsubSudoUser
h2(#overview). Overview
-Arvados can be configured to use "Singularity":https://sylabs.io/singularity/ instead of Docker to execute containers on cloud nodes or a SLURM/LSF cluster. Singularity may be preferable due to its simpler installation and lack of long-running daemon process and special system users/groups.
-
-*Current limitations*:
-* Even when using the singularity runtime, users' container images are expected to be saved in Docker format using @arv keep docker@. Arvados converts the Docker image to Singularity format (@.sif@) at runtime as needed. Specifying a @.sif@ file as an image when submitting a container request is not yet supported.
-* Singularity does not limit the amount of memory available in a container. Each container will have access to all memory on the host where it runs, unless memory use is restricted by SLURM/LSF.
-* Programs running in containers may behave differently due to differences between Singularity and Docker.
-** The root (image) filesystem is read-only in a Singularity container. Programs that attempt to write outside a designated output or temporary directory are likely to fail.
-** The Docker ENTRYPOINT instruction is ignored.
-* Arvados is tested with Singularity version 3.7.4. Other versions may not work.
+Please refer to the "Singularity":{{site.baseurl}}/architecture/singularity.html documentation in the Architecture section.
h2(#configuration). Configuration
</notextile>
Restart your dispatcher (@crunch-dispatch-slurm@, @arvados-dispatch-cloud@, or @arvados-dispatch-lsf@) after updating your configuration file.
+
+h2(#singularity_configuration). Singularity configuration
+
+Docker images are converted on the fly by @mksquashfs@, which can consume a considerable amount of RAM. The RAM usage of mksquashfs can be restricted in @/etc/singularity/singularity.conf@ with a line like @mksquashfs mem = 512M@. The amount of memory made available for mksquashfs should be configured lower than the smallest amount of memory requested by a container on the cluster to avoid the conversion being killed for using too much memory.
To use the Python SDK elsewhere, you can install from PyPI or a distribution package.
-As of Arvados 2.1, the Python SDK requires Python 3.5+. The last version to support Python 2.7 is Arvados 2.0.4.
+As of Arvados 2.2, the Python SDK requires Python 3.6+. The last version to support Python 2.7 is Arvados 2.0.4.
h2. Option 1: Install from a distribution package
# Maximum create/destroy-instance operations per second (0 =
# unlimited).
- MaxCloudOpsPerSecond: 0
+ MaxCloudOpsPerSecond: 10
- # Maximum concurrent node creation operations (0 = unlimited). This is
- # recommended by Azure in certain scenarios (see
- # https://docs.microsoft.com/en-us/azure/virtual-machines/linux/capture-image)
- # and can be used with other cloud providers too, if desired.
- MaxConcurrentInstanceCreateOps: 0
+ # Maximum concurrent instance creation operations (0 = unlimited).
+ #
+ # MaxConcurrentInstanceCreateOps limits the number of instance creation
+ # requests that can be in flight at any one time, whereas
+ # MaxCloudOpsPerSecond limits the number of create/destroy operations
+ # that can be started per second.
+ #
+ # Because the API for instance creation on Azure is synchronous, it is
+ # recommended to increase MaxConcurrentInstanceCreateOps when running
+ # on Azure. When using managed images, a value of 20 would be
+ # appropriate. When using Azure Shared Image Galeries, it could be set
+ # higher. For more information, see
+ # https://docs.microsoft.com/en-us/azure/virtual-machines/linux/capture-image
+ #
+ # MaxConcurrentInstanceCreateOps can be increased for other cloud
+ # providers too, if desired.
+ MaxConcurrentInstanceCreateOps: 1
# Interval between cloud provider syncs/updates ("list all
# instances").
# Maximum create/destroy-instance operations per second (0 =
# unlimited).
- MaxCloudOpsPerSecond: 0
+ MaxCloudOpsPerSecond: 10
- # Maximum concurrent node creation operations (0 = unlimited). This is
- # recommended by Azure in certain scenarios (see
- # https://docs.microsoft.com/en-us/azure/virtual-machines/linux/capture-image)
- # and can be used with other cloud providers too, if desired.
- MaxConcurrentInstanceCreateOps: 0
+ # Maximum concurrent instance creation operations (0 = unlimited).
+ #
+ # MaxConcurrentInstanceCreateOps limits the number of instance creation
+ # requests that can be in flight at any one time, whereas
+ # MaxCloudOpsPerSecond limits the number of create/destroy operations
+ # that can be started per second.
+ #
+ # Because the API for instance creation on Azure is synchronous, it is
+ # recommended to increase MaxConcurrentInstanceCreateOps when running
+ # on Azure. When using managed images, a value of 20 would be
+ # appropriate. When using Azure Shared Image Galeries, it could be set
+ # higher. For more information, see
+ # https://docs.microsoft.com/en-us/azure/virtual-machines/linux/capture-image
+ #
+ # MaxConcurrentInstanceCreateOps can be increased for other cloud
+ # providers too, if desired.
+ MaxConcurrentInstanceCreateOps: 1
# Interval between cloud provider syncs/updates ("list all
# instances").
return err
}
+ // Set up a cache and tmp dir for singularity build
+ err = os.Mkdir(e.tmpdir+"/cache", 0700)
+ if err != nil {
+ return err
+ }
+ defer os.RemoveAll(e.tmpdir + "/cache")
+ err = os.Mkdir(e.tmpdir+"/tmp", 0700)
+ if err != nil {
+ return err
+ }
+ defer os.RemoveAll(e.tmpdir + "/tmp")
+
build := exec.Command("singularity", "build", imageFilename, "docker-archive://"+e.tmpdir+"/image.tar")
+ build.Env = os.Environ()
+ build.Env = append(build.Env, "SINGULARITY_CACHEDIR="+e.tmpdir+"/cache")
+ build.Env = append(build.Env, "SINGULARITY_TMPDIR="+e.tmpdir+"/tmp")
e.logf("%v", build.Args)
out, err := build.CombinedOutput()
// INFO: Starting build...
end
-subcommands = %w(copy create edit get keep pipeline run tag ws)
+subcommands = %w(copy create edit get keep tag ws)
def exec_bin bin, opts
bin_path = `which #{bin.shellescape}`.strip
arv_edit client, arvados, global_opts, remaining_opts
when 'get'
arv_get client, arvados, global_opts, remaining_opts
- when 'copy', 'tag', 'ws', 'run'
+ when 'copy', 'tag', 'ws'
exec_bin "arv-#{subcommand}", remaining_opts
when 'keep'
@sub = remaining_opts.shift
loadingContext = self.loadingContext.copy()
loadingContext.do_validate = False
- loadingContext.do_update = False
if submitting:
+ loadingContext.do_update = False
# Document may have been auto-updated. Reload the original
# document with updating disabled because we want to
# submit the document with its original CWL version, not
elif isinstance(pattern, dict):
specs.append(pattern)
elif isinstance(pattern, str):
- specs.append({"pattern": pattern, "required": sf.get("required")})
+ if builder.cwlVersion == "v1.0":
+ specs.append({"pattern": pattern, "required": True})
+ else:
+ specs.append({"pattern": pattern, "required": sf.get("required")})
else:
raise SourceLine(primary["secondaryFiles"], i, validate.ValidationException).makeError(
"Expression must return list, object, string or null")
if isinstance(sf, dict):
if sf.get("class") == "File":
pattern = None
+ if sf.get("location") is None:
+ raise SourceLine(primary["secondaryFiles"], i, validate.ValidationException).makeError(
+ "File object is missing 'location': %s" % sf)
sfpath = sf["location"]
required = True
else:
# file to determine what version of cwltool and schema-salad to
# build.
install_requires=[
- 'cwltool==3.1.20210922203925',
- 'schema-salad==8.2.20210918131710',
+ 'cwltool==3.1.20211020155521',
+ 'schema-salad==8.2.20211020114435',
'arvados-python-client{}'.format(pysdk_dep),
'setuptools',
'ciso8601 >= 2.0.0',
kwargs.setdefault('stdin', subprocess.PIPE)
kwargs.setdefault('stdout', sys.stderr)
try:
- docker_proc = subprocess.Popen(['docker.io'] + cmd, *args, **kwargs)
- except OSError: # No docker.io in $PATH
docker_proc = subprocess.Popen(['docker'] + cmd, *args, **kwargs)
+ except OSError: # No docker in $PATH, try docker.io
+ docker_proc = subprocess.Popen(['docker.io'] + cmd, *args, **kwargs)
if manage_stdin:
docker_proc.stdin.close()
return docker_proc
check_docker(list_proc, "images")
def find_image_hashes(image_search, image_tag=None):
- # Given one argument, search for Docker images with matching hashes,
- # and return their full hashes in a set.
- # Given two arguments, also search for a Docker image with the
- # same repository and tag. If one is found, return its hash in a
- # set; otherwise, fall back to the one-argument hash search.
- # Returns None if no match is found, or a hash search is ambiguous.
- hash_search = image_search.lower()
- hash_matches = set()
- for image in docker_images():
- if (image.repo == image_search) and (image.tag == image_tag):
- return set([image.hash])
- elif image.hash.startswith(hash_search):
- hash_matches.add(image.hash)
- return hash_matches
+ # Query for a Docker images with the repository and tag and return
+ # the image ids in a list. Returns empty list if no match is
+ # found.
+
+ list_proc = popen_docker(['inspect', "%s%s" % (image_search, ":"+image_tag if image_tag else "")], stdout=subprocess.PIPE)
+
+ inspect = list_proc.stdout.read()
+ list_proc.stdout.close()
+
+ imageinfo = json.loads(inspect)
+
+ return [i["Id"] for i in imageinfo]
def find_one_image_hash(image_search, image_tag=None):
hashes = find_image_hashes(image_search, image_tag)
options = {}
OptionParser.new do |parser|
parser.on('--exclusive', 'Manage SSH keys file exclusively.')
- parser.on('--rotate-tokens', 'Always create new user tokens. Usually needed with --token-lifetime.')
+ parser.on('--rotate-tokens', 'Force a rotation of all user tokens.')
parser.on('--skip-missing-users', "Don't try to create any local accounts.")
parser.on('--token-lifetime SECONDS', 'Create user tokens that expire after SECONDS.', Integer)
+ parser.on('--debug', 'Enable debug output')
end.parse!(into: options)
exclusive_banner = "#######################################################################################
keys = ''
begin
+ debug = false
+ if options[:"debug"]
+ debug = true
+ end
arv = Arvados.new({ :suppress_ssl_warnings => false })
logincluster_arv = Arvados.new({ :api_host => (ENV['LOGINCLUSTER_ARVADOS_API_HOST'] || ENV['ARVADOS_API_HOST']),
:api_token => (ENV['LOGINCLUSTER_ARVADOS_API_TOKEN'] || ENV['ARVADOS_API_TOKEN']),
end
else
if pwnam[l[:username]].uid < uid_min
- STDERR.puts "Account #{l[:username]} uid #{pwnam[l[:username]].uid} < uid_min #{uid_min}. Skipping"
+ STDERR.puts "Account #{l[:username]} uid #{pwnam[l[:username]].uid} < uid_min #{uid_min}. Skipping" if debug
true
end
end
# Collect all keys
logins.each do |l|
+ STDERR.puts("Considering #{l[:username]} ...") if debug
keys[l[:username]] = Array.new() if not keys.has_key?(l[:username])
key = l[:public_key]
if !key.nil?
tokenfile = File.join(configarvados, "settings.conf")
begin
- if !File.exist?(tokenfile) || options[:"rotate-tokens"]
+ STDERR.puts "Processing #{tokenfile} ..." if debug
+ newToken = false
+ if File.exist?(tokenfile)
+ # check if the token is still valid
+ myToken = ENV["ARVADOS_API_TOKEN"]
+ userEnv = IO::read(tokenfile)
+ if (m = /^ARVADOS_API_TOKEN=(.*?\n)/m.match(userEnv))
+ begin
+ tmp_arv = Arvados.new({ :api_host => (ENV['LOGINCLUSTER_ARVADOS_API_HOST'] || ENV['ARVADOS_API_HOST']),
+ :api_token => (m[1]),
+ :suppress_ssl_warnings => false })
+ tmp_arv.user.current
+ rescue Arvados::TransactionFailedError => e
+ if e.to_s =~ /401 Unauthorized/
+ STDERR.puts "Account #{l[:username]} token not valid, creating new token."
+ newToken = true
+ else
+ raise
+ end
+ end
+ end
+ elsif !File.exist?(tokenfile) || options[:"rotate-tokens"]
+ STDERR.puts "Account #{l[:username]} token file not found, creating new token."
+ newToken = true
+ end
+ if newToken
aca_params = {owner_uuid: l[:user_uuid], api_client_id: 0}
if options[:"token-lifetime"] && options[:"token-lifetime"] > 0
aca_params.merge!(expires_at: (Time.now + options[:"token-lifetime"]))
. /usr/local/lib/arvbox/common.sh
. /usr/local/lib/arvbox/go-setup.sh
-flock /var/lib/gopath/gopath.lock go install "git.arvados.org/arvados.git/services/keepstore"
-install $GOPATH/bin/keepstore /usr/local/bin
+(cd /usr/local/bin && ln -sf arvados-server keepstore)
if test "$1" = "--only-deps" ; then
exit
projects / arvados.git / commitdiff