* Get links with API list calls, instead of fetching each one
individually.
* Get a list mapping portable data hashes to UUIDs, and add a single
UUID per portable data hash to the fetch list. This helps us avoid
downloading multiple copies the same manifest text, and is probably
the single-biggest win in this entire commit for most use cases.
* Use the Ruby SDK to build the new collection. This lets us avoid
spawning new arv-normalize processes, and piping large manifests to
them. It also lets us build the entire collection and normalize
only when we're done.
Peter Amstutz [Mon, 13 Apr 2015 14:18:16 +0000 (10:18 -0400)]
5692: Add flush flag to manifest_text() which calls commit_all(). Added
portable_manifest_text() which returns the stripped, normalized manifest text
free of side effects. Fixed tests.
Peter Amstutz [Fri, 10 Apr 2015 19:27:05 +0000 (15:27 -0400)]
5692: Collection.manifest_text(strip=False) will flush open files and wait for
all blocks to be committed in order to return a manifest text with valid
authorization tokens. Fix tests affected by the change.
Brett Smith [Wed, 8 Apr 2015 13:46:23 +0000 (09:46 -0400)]
5642: Explicitly make all swap available under Docker in crunch-job.
Without this, Docker 1.2 through 1.5 send subprocesses SIGKILL if they
exceed the memory limit. Refer to #5642 for an example.
--memory-swap is pretty new (newer than 1.3.3), so we don't want to
require it. At the same time, we don't want to impose any memory
limits if we can't use it, because killing subprocesses that exceed a
--memory limit is too strict. This commit arranges to use both
--memory and --memory-swap only if the latter is available.
Brett Smith [Mon, 6 Apr 2015 18:27:21 +0000 (14:27 -0400)]
5653: arv-copy copies multiple commits from the same repository+pipeline.
arv-copy previously used the repository name alone to determine which
job scripts it had already copied to the destination. If a pipeline
used unrelated commits from the same repository, it would skip copying
over all but the first. Track script versions throughout the copy
process and make sure all of them are copied to the destination
repository.
Brett Smith [Sun, 5 Apr 2015 21:10:22 +0000 (17:10 -0400)]
5352: crunch-dispatch treats node allocation failure as temporary.
Imagine a scenario where multiple crunch-dispatch processes are
sitting idle, then suddenly a new job appears in the queue. They will
all race to dispatch the job. When this happens, we frequently see
that salloc fails for most of them, because they all requested the
same node(s) and only the winner will get them. crunch-dispatch has
no way to know the exit code "came from" salloc and not crunch-job,
and so marks the job failed.
This patch sets the SLURM_EXIT_IMMEDIATE environment variable to make
salloc use exit code 75 when the allocation fails. crunch-dispatch
already recognizes this exit code as a temporary failure, and will
leave the Arvados job record unchanged. Refer to salloc(1) and the
long comment in Dispatch#reap_children.
Brett Smith [Mon, 6 Apr 2015 17:49:29 +0000 (13:49 -0400)]
Tighten up DNS check in arvdock.
I got arvdock stuck in an infinite loop because I had the right
nameserver line in /etc/resolv.conf, except it was commented out, so
arvdock kept checking for a name that it would never find. Use awk to
find and check the first functional nameserver line instead.
Brett Smith [Sun, 5 Apr 2015 21:39:33 +0000 (17:39 -0400)]
Fix debug log formatting in PySDK.
When multiple arguments are passed to a logger method, the first
argument is expected to be a printf-style format string, with the
remaining arguments expected to fill in the formatters.
5365: Do not link to job, log, output in pipeline instance if the user, mainly anonymous user, cannot read those objects.
Disable Log tab and show tooltip in such a scenario.
Working in manual testing. Integration test updates are on the way.
Brett Smith [Thu, 2 Apr 2015 21:37:53 +0000 (17:37 -0400)]
4253: Gitolite migration makes a name symlink for arvados repository.
This is necessary for other supporting infrastructure around this
repository in Gitolite installs; e.g., the cron job that keeps them in
sync with upstream. Refs #4253.
Peter Amstutz [Thu, 2 Apr 2015 14:24:56 +0000 (10:24 -0400)]
4752: Websockets works. Fix compute node containers to restart correctly. Fix
git server hostname. Arvdock waits for API/workbench to be ready before telling
the user to go to workbench.
Brett Smith [Wed, 1 Apr 2015 19:50:08 +0000 (15:50 -0400)]
5627: Python file-like objects use SEET_SET as the default whence.
This is a brown paper bag commit. All that time I spent grumbling
that we had the wrong default was completely incorrect. We had it
right earlier, and I blew it. See
<https://docs.python.org/2/library/stdtypes.html#file.seek>.
projects / arvados.git / log