arvados.git
7 years agoMerge branch '11209-unmount-replace'
Tom Clegg [Tue, 28 Mar 2017 20:13:18 +0000 (16:13 -0400)]
Merge branch '11209-unmount-replace'

refs #11209

7 years agoMerge branch '11333-cwl-phone-home' closes #11333
Peter Amstutz [Tue, 28 Mar 2017 20:01:50 +0000 (16:01 -0400)]
Merge branch '11333-cwl-phone-home' closes #11333

7 years ago11333: Add explicit version pin on 'typing'
Peter Amstutz [Tue, 28 Mar 2017 16:57:25 +0000 (12:57 -0400)]
11333: Add explicit version pin on 'typing'

7 years ago11332: Suppress phone-home behavior in check_exists by recognizing
Peter Amstutz [Tue, 28 Mar 2017 16:44:20 +0000 (12:44 -0400)]
11332: Suppress phone-home behavior in check_exists by recognizing
http://arvados.org/cwl and returning True.

7 years agoMerge branch '11332-cwl-filenames' closes #11332
Peter Amstutz [Tue, 28 Mar 2017 19:50:02 +0000 (15:50 -0400)]
Merge branch '11332-cwl-filenames' closes #11332

7 years ago11332: Override relaxed path checks to be even more relaxed, and bump cwltool &
Peter Amstutz [Mon, 27 Mar 2017 21:29:58 +0000 (17:29 -0400)]
11332: Override relaxed path checks to be even more relaxed, and bump cwltool &
schema salad pin for hashes-in-filenames fix.

7 years ago11209: Test using ./bin/arv-mount from source dir.
Tom Clegg [Tue, 28 Mar 2017 05:52:56 +0000 (01:52 -0400)]
11209: Test using ./bin/arv-mount from source dir.

7 years ago11209: Replace "--unmount /path/..." with "--unmount-all /path".
Tom Clegg [Mon, 27 Mar 2017 22:43:56 +0000 (18:43 -0400)]
11209: Replace "--unmount /path/..." with "--unmount-all /path".

7 years ago11209: Fix daemon mode.
Tom Clegg [Mon, 27 Mar 2017 22:36:48 +0000 (18:36 -0400)]
11209: Fix daemon mode.

DaemonContext cannot happen after listen_for_events().

7 years agoUpdate cwl test_submit refs #10401
Peter Amstutz [Mon, 27 Mar 2017 20:22:46 +0000 (16:22 -0400)]
Update cwl test_submit refs #10401

7 years agoMerge branch '10401-run-upload-dir' refs #10401
Peter Amstutz [Mon, 27 Mar 2017 19:54:41 +0000 (15:54 -0400)]
Merge branch '10401-run-upload-dir' refs #10401

7 years ago10401: Support in arv-run for uploading Directories.
Peter Amstutz [Fri, 24 Mar 2017 18:55:02 +0000 (14:55 -0400)]
10401: Support in arv-run for uploading Directories.

7 years agoPin ws4py <0.4 (our websocket tests fail on 0.4.1)
Tom Clegg [Mon, 27 Mar 2017 16:26:32 +0000 (12:26 -0400)]
Pin ws4py <0.4 (our websocket tests fail on 0.4.1)

No issue #

7 years ago11209: Remove unused imports.
Tom Clegg [Fri, 24 Mar 2017 19:59:56 +0000 (15:59 -0400)]
11209: Remove unused imports.

7 years ago11209: Add tests for --unmount and --replace flags.
Tom Clegg [Fri, 24 Mar 2017 19:59:26 +0000 (15:59 -0400)]
11209: Add tests for --unmount and --replace flags.

7 years ago11209: Fix missing import. Use context handler for standalone mode.
Tom Clegg [Fri, 24 Mar 2017 19:55:24 +0000 (15:55 -0400)]
11209: Fix missing import. Use context handler for standalone mode.

7 years agoRemove debug printf.
Tom Clegg [Fri, 24 Mar 2017 18:42:30 +0000 (14:42 -0400)]
Remove debug printf.

No issue #

7 years agoRemove -i flag from Rails3 pg_dump invocation.
Tom Clegg [Fri, 24 Mar 2017 18:37:22 +0000 (14:37 -0400)]
Remove -i flag from Rails3 pg_dump invocation.

No issue #

7 years agoMerge branch '8567-docker-migrator' refs #8567
Peter Amstutz [Fri, 24 Mar 2017 13:31:36 +0000 (09:31 -0400)]
Merge branch '8567-docker-migrator' refs #8567

7 years ago11209: Try unmounting at least once even if timeout=0.
Tom Clegg [Thu, 23 Mar 2017 22:06:01 +0000 (18:06 -0400)]
11209: Try unmounting at least once even if timeout=0.

7 years ago11209: "--unmount /path/..." unmounts /path and all fuse mounts below it.
Tom Clegg [Fri, 17 Mar 2017 20:32:10 +0000 (16:32 -0400)]
11209: "--unmount /path/..." unmounts /path and all fuse mounts below it.

7 years ago11209: Add arv-mount --unmount and --replace flags
Tom Clegg [Thu, 16 Mar 2017 20:49:49 +0000 (16:49 -0400)]
11209: Add arv-mount --unmount and --replace flags

7 years agoMerge branch '11325-no-broken-nodes' refs #11323, refs #11324, refs #11325
Peter Amstutz [Thu, 23 Mar 2017 20:16:33 +0000 (16:16 -0400)]
Merge branch '11325-no-broken-nodes' refs #11323, refs #11324, refs #11325

7 years ago11323: Don't try to offer_arvados_pair on unpaired nodes which are being shut down.
Peter Amstutz [Thu, 23 Mar 2017 17:30:35 +0000 (13:30 -0400)]
11323: Don't try to offer_arvados_pair on unpaired nodes which are being shut down.

7 years ago11324: Fix crash in NodeManagerDaemonActor when receiving a node_can_shutdown
Peter Amstutz [Thu, 23 Mar 2017 18:05:51 +0000 (14:05 -0400)]
11324: Fix crash in NodeManagerDaemonActor when receiving a node_can_shutdown
message for a node that has already been shut down.

7 years agoMerge branch '10766-install-ws'
Tom Clegg [Thu, 23 Mar 2017 20:10:48 +0000 (16:10 -0400)]
Merge branch '10766-install-ws'

closes #10766

7 years agoMerge branch '11065-rotate-logs'
Tom Clegg [Thu, 23 Mar 2017 19:56:44 +0000 (15:56 -0400)]
Merge branch '11065-rotate-logs'

closes #11065

7 years ago11065: Avoid scientific notation in PostgreSQL literal.
Tom Clegg [Thu, 23 Mar 2017 19:56:16 +0000 (15:56 -0400)]
11065: Avoid scientific notation in PostgreSQL literal.

7 years ago11065: Update comments.
Tom Clegg [Thu, 23 Mar 2017 19:18:51 +0000 (15:18 -0400)]
11065: Update comments.

7 years agoMerge branch '11221-always-restart-services'
Tom Clegg [Thu, 23 Mar 2017 18:51:21 +0000 (14:51 -0400)]
Merge branch '11221-always-restart-services'

closes #11221

7 years ago11325: Remove "broken node" check. Assume if the node really isn't
Peter Amstutz [Thu, 23 Mar 2017 17:23:54 +0000 (13:23 -0400)]
11325: Remove "broken node" check.  Assume if the node really isn't
functioning, it should be "down" in SLURM anyway.  Remove test_broken_node_not_counted because broken node check is removed.

7 years ago10766: Fix copy-paste error in godoc link.
Tom Clegg [Thu, 23 Mar 2017 18:05:06 +0000 (14:05 -0400)]
10766: Fix copy-paste error in godoc link.

7 years ago10766: Migrate install guide from puma to arvados-ws.
Tom Clegg [Thu, 23 Mar 2017 17:46:37 +0000 (13:46 -0400)]
10766: Migrate install guide from puma to arvados-ws.

7 years agoMerge branch '10669-safe-http-cache'
Tom Clegg [Thu, 23 Mar 2017 14:56:34 +0000 (10:56 -0400)]
Merge branch '10669-safe-http-cache'

closes #10669

7 years ago10669: Improve error reporting in test case.
Tom Clegg [Thu, 23 Mar 2017 14:43:01 +0000 (10:43 -0400)]
10669: Improve error reporting in test case.

7 years agoMerge branch '11255-docker-host-networking' closes #11255
Peter Amstutz [Thu, 23 Mar 2017 13:52:39 +0000 (09:52 -0400)]
Merge branch '11255-docker-host-networking' closes #11255

7 years ago11255: Fix whitespace
Peter Amstutz [Thu, 23 Mar 2017 13:49:56 +0000 (09:49 -0400)]
11255: Fix whitespace

7 years ago11255: Add documentation about configuration to enable host networking always.
Peter Amstutz [Wed, 22 Mar 2017 16:56:34 +0000 (12:56 -0400)]
11255: Add documentation about configuration to enable host networking always.

7 years ago11255: Add -container-enable-networking and -container-network-mode options.
Peter Amstutz [Thu, 23 Mar 2017 13:49:31 +0000 (09:49 -0400)]
11255: Add -container-enable-networking and -container-network-mode options.

7 years agoMerge branch '11170-stale-squeue-procs'
Lucas Di Pentima [Thu, 23 Mar 2017 13:33:14 +0000 (10:33 -0300)]
Merge branch '11170-stale-squeue-procs'
Closes #11170

7 years ago10669: pep8
Tom Clegg [Thu, 23 Mar 2017 04:19:38 +0000 (00:19 -0400)]
10669: pep8

7 years ago10669: Test that SafeHTTPCache is actually used.
Tom Clegg [Thu, 23 Mar 2017 04:16:26 +0000 (00:16 -0400)]
10669: Test that SafeHTTPCache is actually used.

7 years ago10669: Only delete *.tmp when cleaning old cache files.
Tom Clegg [Wed, 22 Mar 2017 21:57:47 +0000 (17:57 -0400)]
10669: Only delete *.tmp when cleaning old cache files.

7 years ago10699: Delete >2-day old cache entries.
Tom Clegg [Wed, 22 Mar 2017 21:45:32 +0000 (17:45 -0400)]
10699: Delete >2-day old cache entries.

7 years ago10669: Bust old (possibly broken) cache entries by naming our files *.tmp
Tom Clegg [Wed, 22 Mar 2017 21:44:52 +0000 (17:44 -0400)]
10669: Bust old (possibly broken) cache entries by naming our files *.tmp

7 years ago10669: Robust cache for httplib2
Tom Clegg [Wed, 22 Mar 2017 19:16:37 +0000 (15:16 -0400)]
10669: Robust cache for httplib2

7 years agoMerge branch '10218-record-node-info'
Lucas Di Pentima [Wed, 22 Mar 2017 20:18:59 +0000 (17:18 -0300)]
Merge branch '10218-record-node-info'
Closes #10218

7 years ago10218: Avoid running df commands more than necessary. Updated test to check for disk...
Lucas Di Pentima [Wed, 22 Mar 2017 19:56:58 +0000 (16:56 -0300)]
10218: Avoid running df commands more than necessary. Updated test to check for disk inodes report. Added deferred Close() call on CallRaw reader.

7 years ago11065: Merge branch 'master' into 11065-rotate-logs
Tom Clegg [Wed, 22 Mar 2017 19:30:52 +0000 (15:30 -0400)]
11065: Merge branch 'master' into 11065-rotate-logs

7 years ago10218: Added calls to "df -i" to record free i-nodes. Combined stdout and stderr...
Lucas Di Pentima [Wed, 22 Mar 2017 19:07:37 +0000 (16:07 -0300)]
10218: Added calls to "df -i" to record free i-nodes. Combined stdout and stderr when running node info commands.

7 years ago10218: Ask for disk space on 2 specific filesystems (/ & /tmp) to avoid having the...
Lucas Di Pentima [Wed, 22 Mar 2017 18:51:13 +0000 (15:51 -0300)]
10218: Ask for disk space on 2 specific filesystems (/ & /tmp) to avoid having the df command to return an error on special cases.

7 years ago10218: Merge branch 'master' into 10218-record-node-info
Lucas Di Pentima [Wed, 22 Mar 2017 18:45:49 +0000 (15:45 -0300)]
10218: Merge branch 'master' into 10218-record-node-info

7 years ago10218: Get the container record directly from the API server, instead of Marshaling...
Lucas Di Pentima [Wed, 22 Mar 2017 18:01:48 +0000 (15:01 -0300)]
10218: Get the container record directly from the API server, instead of Marshaling runner.Container.

7 years agoMerge branch '7709-sdk-cli-active_support'
Tom Clegg [Wed, 22 Mar 2017 17:33:55 +0000 (13:33 -0400)]
Merge branch '7709-sdk-cli-active_support'

refs #7709

7 years agoFix broken link from crunch2 to crunch1 docs.
Tom Clegg [Wed, 22 Mar 2017 16:07:48 +0000 (12:07 -0400)]
Fix broken link from crunch2 to crunch1 docs.

refs #6520

7 years agoDo not let golang.org/x/tools/cmd/bundle get ahead of Ruby bundler.
Tom Clegg [Wed, 22 Mar 2017 16:04:43 +0000 (12:04 -0400)]
Do not let golang.org/x/tools/cmd/bundle get ahead of Ruby bundler.

refs https://dev.arvados.org/projects/arvados/wiki/Emacs_and_Go

No issue #

7 years ago8567: Add start_docker and kill_docker functions with wait loops.
Peter Amstutz [Wed, 22 Mar 2017 15:57:17 +0000 (11:57 -0400)]
8567: Add start_docker and kill_docker functions with wait loops.

7 years ago7709: arvados-cli allows activesupport>=4
Tom Clegg [Sun, 19 Mar 2017 01:12:45 +0000 (21:12 -0400)]
7709: arvados-cli allows activesupport>=4

7 years ago11170: Calling close method from an ensure block.
Lucas Di Pentima [Wed, 22 Mar 2017 14:18:55 +0000 (11:18 -0300)]
11170: Calling close method from an ensure block.

7 years ago11065: Defer audit log cleanup until after_commit.
Tom Clegg [Wed, 22 Mar 2017 14:18:43 +0000 (10:18 -0400)]
11065: Defer audit log cleanup until after_commit.

7 years ago11170: Updated tests to reflect the use of IO instead of File.
Lucas Di Pentima [Wed, 22 Mar 2017 13:21:17 +0000 (10:21 -0300)]
11170: Updated tests to reflect the use of IO instead of File.

7 years ago8567: Make sure /root/.cache/arvados/docker is tempdir volume mount as well.
Peter Amstutz [Tue, 21 Mar 2017 21:37:30 +0000 (17:37 -0400)]
8567: Make sure /root/.cache/arvados/docker is tempdir volume mount as well.

7 years ago8567: Use vfs Docker graph driver for maximum compatability. --dry-run reports
Peter Amstutz [Tue, 21 Mar 2017 18:02:09 +0000 (14:02 -0400)]
8567: Use vfs Docker graph driver for maximum compatability.  --dry-run reports
maximum image size.  Add --tempdir option.

7 years agoFix test broken by change of default behavior in arv-keepdocker. refs #8567
Peter Amstutz [Tue, 21 Mar 2017 15:41:15 +0000 (11:41 -0400)]
Fix test broken by change of default behavior in arv-keepdocker. refs #8567

7 years ago11065: Delete old audit logs.
Tom Clegg [Tue, 21 Mar 2017 15:39:20 +0000 (11:39 -0400)]
11065: Delete old audit logs.

7 years ago11170: Treat the squeue/scancel calls as files instead of treating them as processes...
Lucas Di Pentima [Mon, 20 Mar 2017 21:39:43 +0000 (18:39 -0300)]
11170: Treat the squeue/scancel calls as files instead of treating them as processes. Calling close when finished reading from them.

7 years ago10218: Merge branch 'master' into 10218-record-node-info
Lucas Di Pentima [Mon, 20 Mar 2017 19:07:40 +0000 (16:07 -0300)]
10218: Merge branch 'master' into 10218-record-node-info

7 years agoIf API server doesn't publish supported Docker image versions, make that an
Peter Amstutz [Mon, 20 Mar 2017 19:04:38 +0000 (15:04 -0400)]
If API server doesn't publish supported Docker image versions, make that an
error instead of a warning (can be overridden with --force-image-format)
refs #8567

7 years ago11170: Set up a thread to reap the status of squeue runs so that they don't become...
Lucas Di Pentima [Mon, 20 Mar 2017 18:56:55 +0000 (15:56 -0300)]
11170: Set up a thread to reap the status of squeue runs so that they don't become zombie processes.

7 years agoMerge branch '8567-docker-migrator' refs #8567
Peter Amstutz [Mon, 20 Mar 2017 18:34:17 +0000 (14:34 -0400)]
Merge branch '8567-docker-migrator' refs #8567

7 years ago8567: Bugfix check for existing repo+tag links.
Peter Amstutz [Mon, 20 Mar 2017 15:58:37 +0000 (11:58 -0400)]
8567: Bugfix check for existing repo+tag links.

7 years agocloses #11290
radhika [Mon, 20 Mar 2017 15:04:44 +0000 (11:04 -0400)]
closes #11290
Merge branch '11290-caret-font'

7 years ago8567: arv-keepdocker catches EPIPE so it doesn't print backtrace when piped to
Peter Amstutz [Mon, 20 Mar 2017 14:02:16 +0000 (10:02 -0400)]
8567: arv-keepdocker catches EPIPE so it doesn't print backtrace when piped to
other commands such as "head".

7 years ago8567: Add migrate19
Peter Amstutz [Mon, 20 Mar 2017 13:36:17 +0000 (09:36 -0400)]
8567: Add migrate19

7 years ago11290: Reverted "fa-caret-down" usage back to 'span class="caret"' with href added...
radhika [Mon, 20 Mar 2017 10:34:32 +0000 (06:34 -0400)]
11290: Reverted "fa-caret-down" usage back to 'span class="caret"' with href added to it. Working with Firefox and Chrome now.

7 years ago8567: Better command line UI
Peter Amstutz [Fri, 17 Mar 2017 21:54:11 +0000 (17:54 -0400)]
8567: Better command line UI

7 years agoMerge branch '11288-slurm-down-reason' closes #11288
Peter Amstutz [Fri, 17 Mar 2017 20:36:41 +0000 (16:36 -0400)]
Merge branch '11288-slurm-down-reason' closes #11288

7 years ago10218: Save the container record as a json file and also write it on the logs.
Lucas Di Pentima [Fri, 17 Mar 2017 20:29:56 +0000 (17:29 -0300)]
10218: Save the container record as a json file and also write it on the logs.

7 years ago11288: Slurm requires reason to put node in DOWN state.
Peter Amstutz [Fri, 17 Mar 2017 17:27:03 +0000 (13:27 -0400)]
11288: Slurm requires reason to put node in DOWN state.

7 years agoMerge branch '11014-hide-node-status'
Lucas Di Pentima [Fri, 17 Mar 2017 14:44:20 +0000 (11:44 -0300)]
Merge branch '11014-hide-node-status'
Closes #11014

7 years ago11014: When PipelineInstance API is off and also show_recent_collections_on_dashboard...
Lucas Di Pentima [Fri, 17 Mar 2017 14:16:05 +0000 (11:16 -0300)]
11014: When PipelineInstance API is off and also show_recent_collections_on_dashboard configuration is off, now the "Recent processes" panel will take full screen width.
Corrected test name.
Avoid calling PipelineInstance.api_exist?(:index) more than once.

7 years agoMerge branch '11254-nodemanager-no-actor' closes #11254
Peter Amstutz [Thu, 16 Mar 2017 20:52:11 +0000 (16:52 -0400)]
Merge branch '11254-nodemanager-no-actor' closes #11254

7 years ago11254: Refactor _node_states
Peter Amstutz [Thu, 16 Mar 2017 20:49:32 +0000 (16:49 -0400)]
11254: Refactor _node_states

7 years ago11254: Cloud nodes where "actor is None" are considered to be in shutdown. The
Peter Amstutz [Thu, 16 Mar 2017 20:11:00 +0000 (16:11 -0400)]
11254: Cloud nodes where "actor is None" are considered to be in shutdown.  The
only time it should be "None" is the period between a successful shutdown and
when the node disappears from the cloud node list.

7 years ago10218: Use a []string for the entire command instead of splitting it.
Lucas Di Pentima [Thu, 16 Mar 2017 19:16:34 +0000 (16:16 -0300)]
10218: Use a []string for the entire command instead of splitting it.

7 years ago11014: Check for PipelineIndex#index API to exist for rendering the compute node...
Lucas Di Pentima [Thu, 16 Mar 2017 18:18:23 +0000 (15:18 -0300)]
11014: Check for PipelineIndex#index API to exist for rendering the compute node status pane on the Dashboard.
Added related test.

7 years ago10218: Wait for container to be started (not just created) before trying to cancel it.
Tom Clegg [Thu, 16 Mar 2017 17:24:58 +0000 (13:24 -0400)]
10218: Wait for container to be started (not just created) before trying to cancel it.

7 years ago10218: Split multi-line command output so that each line is written to the logs indep...
Lucas Di Pentima [Thu, 16 Mar 2017 14:01:19 +0000 (11:01 -0300)]
10218: Split multi-line command output so that each line is written to the logs independently.

7 years ago10218: Logging node information (cpu, mem, disk) by storing command outputs on the...
Lucas Di Pentima [Wed, 15 Mar 2017 22:21:10 +0000 (19:21 -0300)]
10218: Logging node information (cpu, mem, disk) by storing command outputs on the log collection. Added relevant test.

7 years ago8567: Add check that admin token is used and ensure that migration links are
Peter Amstutz [Wed, 15 Mar 2017 15:03:20 +0000 (11:03 -0400)]
8567: Add check that admin token is used and ensure that migration links are
created owned by system user.  Also fix tests now that arv-keepdocker uses
logging instead of printing directly to sys.stderr.

7 years agocloses #11071
radhika [Tue, 14 Mar 2017 20:01:14 +0000 (16:01 -0400)]
closes #11071
Merge branch '11071-fts-perf-test'

7 years ago11071: formatting on the long if statement
radhika [Tue, 14 Mar 2017 20:00:17 +0000 (16:00 -0400)]
11071: formatting on the long if statement

7 years agoAdd missing documentation file. refs #6520
Peter Amstutz [Tue, 14 Mar 2017 19:35:29 +0000 (15:35 -0400)]
Add missing documentation file.  refs #6520

7 years agoMerge branch '8567-api-select-docker-fmt' refs #8567
Peter Amstutz [Tue, 14 Mar 2017 19:17:49 +0000 (15:17 -0400)]
Merge branch '8567-api-select-docker-fmt' refs #8567

7 years agoMerge branch '8567-cwl-docker-img' refs #8567
Peter Amstutz [Tue, 14 Mar 2017 19:17:34 +0000 (15:17 -0400)]
Merge branch '8567-cwl-docker-img' refs #8567

7 years agoMerge branch '6520-nodemanager-docs' closes #11123
Peter Amstutz [Tue, 14 Mar 2017 19:09:00 +0000 (15:09 -0400)]
Merge branch '6520-nodemanager-docs' closes #11123

7 years ago8567: Add note about updating API server configuration.
Peter Amstutz [Tue, 14 Mar 2017 16:44:35 +0000 (12:44 -0400)]
8567: Add note about updating API server configuration.

7 years ago8567: Add docker19 migration instructions to install guide.
Peter Amstutz [Tue, 14 Mar 2017 16:39:24 +0000 (12:39 -0400)]
8567: Add docker19 migration instructions to install guide.

7 years ago8567: Fix migrate links to use PDH instead of UUID.
Peter Amstutz [Thu, 9 Mar 2017 22:41:55 +0000 (17:41 -0500)]
8567: Fix migrate links to use PDH instead of UUID.

Better error reporting.

Migrate script cleans up /var/lib/docker inside container.

7 years ago8567: Add check for ARVADOS_API_HOST_INSECURE
Peter Amstutz [Tue, 7 Mar 2017 15:02:39 +0000 (10:02 -0500)]
8567: Add check for ARVADOS_API_HOST_INSECURE