radhika [Thu, 4 Feb 2016 15:21:00 +0000 (10:21 -0500)]
8183: show only toplevel projects in the Projects dropdown in breadcrumbs.
Joshua Randall [Wed, 3 Feb 2016 12:48:19 +0000 (12:48 +0000)]
Unpins llfuse version and adds fuse 2.9.2 backport
Brett Smith [Thu, 4 Feb 2016 10:33:24 +0000 (05:33 -0500)]
Make install guide slurm.conf more Arvados-compliant.
* SelectType=select/linear allocates entire nodes at a time. The
previous value scheduled individual cores.
* With that change, SelectTypeParameters=CR_CPU_Memory is not valid.
Remove it, as we do in production.
* The setting of FastSchedule seems less pressing, but 0 is what we
use in production, so share that here too.
No issue #.
Peter Amstutz [Wed, 3 Feb 2016 22:51:46 +0000 (17:51 -0500)]
Try to make logging identify the actor consistently
Peter Amstutz [Wed, 3 Feb 2016 20:54:18 +0000 (15:54 -0500)]
Merge branch '6702-gce-node-create-fix' closes #6702
Brett Smith [Wed, 3 Feb 2016 19:24:07 +0000 (14:24 -0500)]
Inhibit udev startup in Debian test Docker images.
The FUSE driver package depends on fuse, which depends on udev.
udev can't start its daemon from postinst in a container.
It has code to try to detect running in a container and skip the
daemon, but it fails intermittently.
e.g., <https://ci.curoverse.com/job/build-packages/1699/consoleFull>
It also skips starting the daemon when /etc/udev/disabled exists, so
create that.
Tom Clegg [Wed, 3 Feb 2016 17:51:54 +0000 (12:51 -0500)]
Merge branch '8288-arv-mount-deadlock' refs #8288
Tom Clegg [Tue, 2 Feb 2016 21:46:35 +0000 (16:46 -0500)]
8288: Do not call operations.destroy() as a last resort, just abandon the llfuse thread.
Tom Clegg [Mon, 1 Feb 2016 08:01:31 +0000 (03:01 -0500)]
8288: Add test case for --exec mode.
Tom Clegg [Mon, 1 Feb 2016 02:43:30 +0000 (21:43 -0500)]
8288: Give fusermount -u a chance to work before resorting to operations.destroy().
Log a warning when resorting to operations.destroy().
De-duplicate setup/teardown code so more of the --exec code path is exercised in tests.
Tom Clegg [Wed, 3 Feb 2016 17:50:31 +0000 (12:50 -0500)]
8123: Install chartjs.js asset file.
...during "setup.py install" too, not just when installing via
package.
refs #8123
Brett Smith [Wed, 3 Feb 2016 13:19:34 +0000 (08:19 -0500)]
.deb dependencies from Python packages ignore iteration.
See comment for rationale.
Inspired by <https://github.com/curoverse/arvados/pull/35>.
No issue #.
Brett Smith [Wed, 3 Feb 2016 11:42:17 +0000 (06:42 -0500)]
Improve install guide Nginx+SCL integration.
No issue #.
Brett Smith [Wed, 3 Feb 2016 11:26:32 +0000 (06:26 -0500)]
login-sync gets user's home from /etc/passwd.
No issue #.
Brett Smith [Wed, 3 Feb 2016 10:37:42 +0000 (05:37 -0500)]
Workbench loads CA certs on Red Hat.
This has the same rationale and logic as #6432 and
9b910084faf3db6fa2071af604620e7d45d12a6c, applied to Workbench.
Changing from `/etc/ssl/certs` to `/etc/ssl/certs/ca-certificates.crt`
is safe, because add_trust_ca accepts either a directory with hashed
certs, or a file with multiple certs. On Debian, the latter path is a
single file built from the hashed certs in the former, so this is
functionally identical there, and more predictable on Red Hat (where I
don't know what it's doing).
No issue #.
Brett Smith [Wed, 3 Feb 2016 09:53:04 +0000 (04:53 -0500)]
Add fuse dependency to FUSE driver package.
When the fuse tools aren't installed, attempting to run arv-mount
fails with "fuse: failed to exec fusermount".
No issue #.
Brett Smith [Wed, 3 Feb 2016 09:39:27 +0000 (04:39 -0500)]
Add curl library dependency to shell install guide.
No isse #.
Brett Smith [Wed, 3 Feb 2016 09:32:39 +0000 (04:32 -0500)]
SLURM install guide notes slurm.conf path on Red Hat.
No issue #.
Brett Smith [Wed, 3 Feb 2016 09:26:49 +0000 (04:26 -0500)]
Add missing ; in keepproxy Nginx config.
No issue #.
radhika [Tue, 2 Feb 2016 23:22:18 +0000 (18:22 -0500)]
8286: add "star" route for projects. Update projects dropdown in breadcrumbs to display only starred projects instead of all my_projects.
Peter Amstutz [Tue, 2 Feb 2016 17:26:57 +0000 (12:26 -0500)]
6702: Refactor create_node to BaseComputeNodeDriver so logic also applies to
Azure. Adds new find_node() method; if returns None or raises an error,
re-raise the original create_node exception.
Peter Amstutz [Tue, 2 Feb 2016 16:31:15 +0000 (11:31 -0500)]
Merge branch '6702-gce-node-create-fix' closes #6702
Peter Amstutz [Tue, 2 Feb 2016 16:05:50 +0000 (11:05 -0500)]
Merge branch 'fix/build-python-llfuse-version' of https://github.com/wtsi-hgi/arvados
no issue #
Peter Amstutz [Tue, 2 Feb 2016 15:56:13 +0000 (10:56 -0500)]
Merge branch 'master' into 6702-gce-node-create-fix
Peter Amstutz [Tue, 2 Feb 2016 15:55:58 +0000 (10:55 -0500)]
Merge branch '8206-gce-retry-init' closes #8206
Peter Amstutz [Tue, 2 Feb 2016 15:55:39 +0000 (10:55 -0500)]
8206: Mock time.sleep() to avoid unnecessary delay in test.
Joshua Randall [Tue, 2 Feb 2016 15:45:46 +0000 (15:45 +0000)]
pins python-llfuse version to 0.41.1 for fpm on all platforms
Peter Amstutz [Tue, 2 Feb 2016 15:03:39 +0000 (10:03 -0500)]
8206: Refactor _retry to RetryMixin. Make retry timing consistent.
Joshua Randall [Tue, 2 Feb 2016 13:29:36 +0000 (13:29 +0000)]
adds numerous crunch-job performance improvements
Brett Smith [Tue, 2 Feb 2016 12:23:10 +0000 (07:23 -0500)]
8005: Install guide suggests slurm-munge on Red Hat SLURM nodes.
This package includes the SLURM plugins that talk to MUNGE.
Refs #8005.
Peter Amstutz [Mon, 1 Feb 2016 19:54:28 +0000 (14:54 -0500)]
6702: Catch GCE create_node() errors and check if the node was actually
created. Added test.
Brett Smith [Mon, 1 Feb 2016 17:43:04 +0000 (12:43 -0500)]
8014: Remove more upgrade script references from install guide.
The steps removed are now handled by Rails package postinst scripts.
This should've been done in
378a988bbf9e29736382339f587582259b641782,
but was overlooked. Refs #8014.
Brett Smith [Mon, 1 Feb 2016 16:53:29 +0000 (11:53 -0500)]
Refresh Gitolite install guide.
* Tested instructions still work with 3.6.4. So noted.
* Prefer cloning Gitolite over HTTPS, since that's less likely to be
firewalled.
No issue #.
Brett Smith [Mon, 1 Feb 2016 16:51:14 +0000 (11:51 -0500)]
Fix install doc rendering of API Nginx config.
<notextile> doesn't actually nest like proper HTML, it's just a
boolean that remembers the last state. Turn it back on after doing an
include that turns it off. No issue #.
Peter Amstutz [Mon, 1 Feb 2016 14:16:52 +0000 (09:16 -0500)]
Pin llfuse to 0.41.1 because 0.42 came out and broke things. no issue #
Peter Amstutz [Mon, 1 Feb 2016 14:14:41 +0000 (09:14 -0500)]
Pin llfuse to 0.41.1 because 0.42 came out and broke things. no issue #
Peter Amstutz [Fri, 29 Jan 2016 21:15:41 +0000 (16:15 -0500)]
Add distinct fpm_exclude so it shows up in the right place on the fpm command
line. refs #8008
Peter Amstutz [Fri, 29 Jan 2016 15:22:16 +0000 (10:22 -0500)]
8319: Fixes to run bcbio-nextgen CWL workflow. Now creates pipeline instance
so that progress is visible on workbench.
Brett Smith [Fri, 29 Jan 2016 00:38:04 +0000 (19:38 -0500)]
Merge branch '8005-centos-3rdparty-installs-wip'
Closes #8005, #8135.
Brett Smith [Fri, 29 Jan 2016 00:27:13 +0000 (19:27 -0500)]
8005: Add tar Ruby build dependency on CentOS 6.
Brett Smith [Thu, 28 Jan 2016 00:02:05 +0000 (19:02 -0500)]
8005: Install guide uses runit packages on Red Hat.
The runit RPMs only provide /etc/service. The .debs provide /etc/sv
and /etc/service. Our understanding is that /etc/sv is for all
service definitions (akin to /etc/init.d), and /etc/service is for
service definitions that runit should start at boot (akin to
/etc/rcN.d). To provide uniformity, our install guide instructs users
to make /etc/sv if needed, and link it to /etc/service.
This commit could go farther. Today it would be best if all the runit
sections in the install guide followed Tom's modern template used for
arv-git-httpd and arvados-docker-cleaner. However, that will probably
require some creation and testing of log/run scripts, and some
adaptation of the run scripts to fit the template. I wish I could
include those improvements now, but unfortunately I'm out of time, so
they'll have to wait for another day.
Brett Smith [Thu, 28 Jan 2016 00:08:33 +0000 (19:08 -0500)]
8005: Install guide gets SLURM and MUNGE from RPMs.
Brett Smith [Wed, 27 Jan 2016 23:54:57 +0000 (18:54 -0500)]
8005: Fix bad Textile markup in compute node install guide.
The switch dashes created strikethrough for much of the notebox.
Brett Smith [Wed, 27 Jan 2016 20:15:23 +0000 (15:15 -0500)]
8005: Document installing Git on CentOS 6 from RepoForge.
Brett Smith [Wed, 27 Jan 2016 20:00:17 +0000 (15:00 -0500)]
8005: DRY up PostgreSQL password auth instructions on CentOS 6.
Peter Amstutz [Thu, 28 Jan 2016 21:41:08 +0000 (16:41 -0500)]
8319: Fix arv-cwl-runner file references when referencing files in Keep.
Peter Amstutz [Thu, 28 Jan 2016 19:55:29 +0000 (14:55 -0500)]
Use volumes for /var/log/nginx and /etc/ssl/private so that changing ownership
sticks. refs #8080
Peter Amstutz [Thu, 28 Jan 2016 19:55:29 +0000 (14:55 -0500)]
Use volumes for /var/log/nginx and /etc/ssl/private so that changing ownership
sticks. refs #8080
Ward Vandewege [Thu, 28 Jan 2016 19:32:00 +0000 (14:32 -0500)]
Make our API server packages for debian-based distributions depend on
libcurl-ssl-dev rather than libcurl4-openssl-dev.
No issue #
Peter Amstutz [Thu, 28 Jan 2016 19:22:23 +0000 (14:22 -0500)]
Arvbox fixes: use 'postgres' database when testing if a user exists. Copy
gitolite.rc to the right place. Print out what is being deleted when using
"reset" or "destroy". refs #8080
Peter Amstutz [Thu, 28 Jan 2016 19:22:23 +0000 (14:22 -0500)]
Arvbox fixes: use 'postgres' database when testing if a user exists. Copy
gitolite.rc to the right place. Print out what is being deleted when using
"reset" or "destroy". refs #8080
Peter Amstutz [Wed, 27 Jan 2016 21:40:53 +0000 (16:40 -0500)]
chown /etc/ssl/private to arvbox refs #8080
Peter Amstutz [Wed, 27 Jan 2016 21:40:53 +0000 (16:40 -0500)]
chown /etc/ssl/private to arvbox refs #8080
Peter Amstutz [Wed, 27 Jan 2016 20:39:38 +0000 (15:39 -0500)]
Bump iteration of rails packages to -4 refs #8008
Peter Amstutz [Wed, 27 Jan 2016 19:34:06 +0000 (14:34 -0500)]
Fix selecting most recent package for package testing, refs #8008.
radhika [Tue, 26 Jan 2016 17:10:04 +0000 (12:10 -0500)]
closes #8198
Merge branch '8198-node-ip-address'
radhika [Tue, 26 Jan 2016 17:09:37 +0000 (12:09 -0500)]
Merge branch 'master' into 8198-node-ip-address
radhika [Tue, 26 Jan 2016 17:08:21 +0000 (12:08 -0500)]
refs #8178
Merge branch '8178-keepstore-trash-interface'
radhika [Tue, 26 Jan 2016 15:41:00 +0000 (10:41 -0500)]
Merge branch '8178-keepstore-trash-interface' of git.curoverse.com:arvados into 8178-keepstore-trash-interface
Conflicts:
services/keepstore/handlers.go
services/keepstore/volume_test.go
radhika [Tue, 26 Jan 2016 15:38:28 +0000 (10:38 -0500)]
8178: untrash should fail when ErrNotImplemented is returned.
radhika [Fri, 22 Jan 2016 22:37:15 +0000 (17:37 -0500)]
8178: (for now) all volumes must return ErrNotImplemented if trash-lifetime != 0
radhika [Thu, 21 Jan 2016 20:25:06 +0000 (15:25 -0500)]
8178: All three currently supported volumes return error when trash-lifetime period is not configured. azure blob and s3 volumes are updated to do so.
Returning an error is causing test failures in unix volume and hence is still a work in progress.
radhika [Thu, 21 Jan 2016 18:59:36 +0000 (13:59 -0500)]
8178: rename Delete api as Trash; add Untrash to volume interface; add UndeleteHandler and test for this endpoint.
Brett Smith [Mon, 25 Jan 2016 23:15:12 +0000 (18:15 -0500)]
8008: Make dependencies for build-dockerfiles.
We've apparently never had code to do this before. I guess all
deployments had the dependencies generated by hand before running.
This makes them automated to fix failures like
<https://ci.curoverse.com/job/build-packages/1668/console>.
Refs #8008.
Peter Amstutz [Mon, 25 Jan 2016 22:02:40 +0000 (17:02 -0500)]
8206: Add test to support retry on create_driver.
Tom Clegg [Mon, 25 Jan 2016 21:08:14 +0000 (16:08 -0500)]
Merge branch '8123-crunchstat-graphs' closes #8123
Tom Clegg [Mon, 25 Jan 2016 21:05:56 +0000 (16:05 -0500)]
8123: Escape HTML chars in page title.
Peter Amstutz [Mon, 25 Jan 2016 20:36:34 +0000 (15:36 -0500)]
8206: Refactor _retry into common function wrapper usable by both dispatch and
compute drivers.
Peter Amstutz [Mon, 25 Jan 2016 17:14:44 +0000 (12:14 -0500)]
Fix markdown for arvbox README.md refs #8080
Peter Amstutz [Mon, 25 Jan 2016 17:14:44 +0000 (12:14 -0500)]
Fix markdown for arvbox README.md refs #8080
Peter Amstutz [Mon, 25 Jan 2016 15:34:02 +0000 (10:34 -0500)]
Add "status" command, refs #8080
Peter Amstutz [Mon, 25 Jan 2016 15:34:02 +0000 (10:34 -0500)]
Add "status" command, refs #8080
Peter Amstutz [Mon, 25 Jan 2016 15:21:22 +0000 (10:21 -0500)]
Just create arvbox superuser instead of creating a whole useless database.
Check directly whether creating the arvbox is required. refs #8080
Peter Amstutz [Mon, 25 Jan 2016 15:21:22 +0000 (10:21 -0500)]
Just create arvbox superuser instead of creating a whole useless database.
Check directly whether creating the arvbox is required. refs #8080
Tom Clegg [Mon, 25 Jan 2016 06:16:44 +0000 (01:16 -0500)]
8123: Explain existing_constraints and use a proper instance variable.
Tom Clegg [Mon, 25 Jan 2016 06:08:27 +0000 (01:08 -0500)]
8123: Fix accidental old-style class.
Tom Clegg [Mon, 25 Jan 2016 06:00:03 +0000 (01:00 -0500)]
8123: Fix type check to accommodate unicode.
Tom Clegg [Mon, 25 Jan 2016 05:59:46 +0000 (00:59 -0500)]
8123: Use -v,-vv instead of --verbose,--debug.
Tom Clegg [Mon, 25 Jan 2016 02:07:42 +0000 (21:07 -0500)]
8123: Change --include-child-jobs to --skip-child-jobs (default False).
Tom Clegg [Mon, 25 Jan 2016 02:06:48 +0000 (21:06 -0500)]
8123: Explain mysterious memory constraint logic.
Tom Clegg [Mon, 25 Jan 2016 02:05:28 +0000 (21:05 -0500)]
8123: Update test dependencies.
Tom Clegg [Mon, 25 Jan 2016 00:48:06 +0000 (19:48 -0500)]
8284: Fix confusion between %proc and %jobstep.
$proc{$pid}->{jobstep} is an index into @jobstep
$proc{$pid}->{jobstepname} is the name we told srun to use
$proc{$pid}->{killtime} is a deadline when we should kill the process
$jobstep[$jobstepid]->{stderr_at} is the time of last stderr received
We were mistakenly using $proc->{$pid}->{stderr_at}, which was always
undef and therefore always less than $last_squeue_check. This resulted
in jobs being killed as "slurm orphans" when the real reason they
hadn't been returned by waitpid() was that we hadn't finished
consuming their stderr yet.
Tom Clegg [Sat, 23 Jan 2016 06:28:38 +0000 (01:28 -0500)]
8123: Include chartjs.js in package.
radhika [Fri, 22 Jan 2016 22:49:08 +0000 (17:49 -0500)]
Merge branch '8178-keepstore-trash-interface' of git.curoverse.com:arvados into 8178-keepstore-trash-interface
Conflicts:
services/keepstore/azure_blob_volume.go
services/keepstore/handler_test.go
services/keepstore/handlers.go
services/keepstore/keepstore.go
services/keepstore/s3_volume.go
services/keepstore/volume_test.go
services/keepstore/volume_unix.go
Ward Vandewege [Fri, 22 Jan 2016 22:44:55 +0000 (17:44 -0500)]
Make sure not to accidentally include config/application.yml and
config/database.yml in our rails packages, take 2: make sure that
config/database.yml *is* present in our workbench packages, where we
want to include it.
No issue #
radhika [Fri, 22 Jan 2016 22:37:15 +0000 (17:37 -0500)]
8178: (for now) all volumes must return ErrNotImplemented if trash-lifetime != 0
Ward Vandewege [Fri, 22 Jan 2016 21:40:26 +0000 (16:40 -0500)]
Merge branch '8066-git-init-from-postinst'
closes #8066
Ward Vandewege [Fri, 22 Jan 2016 21:39:23 +0000 (16:39 -0500)]
Some review fixes.
closes #8066
Ward Vandewege [Fri, 22 Jan 2016 21:21:23 +0000 (16:21 -0500)]
Merge branch 'master' into 8066-git-init-from-postinst
Tom Clegg [Fri, 22 Jan 2016 20:02:21 +0000 (15:02 -0500)]
7263: Avoid getting stuck processing stderr for one task for a long time.
Do not sleep(0.1) unless pipes are idle.
radhika [Thu, 21 Jan 2016 20:25:06 +0000 (15:25 -0500)]
8178: All three currently supported volumes return error when trash-lifetime period is not configured. azure blob and s3 volumes are updated to do so.
Returning an error is causing test failures in unix volume and hence is still a work in progress.
radhika [Thu, 21 Jan 2016 18:59:36 +0000 (13:59 -0500)]
8178: rename Delete api as Trash; add Untrash to volume interface; add UndeleteHandler and test for this endpoint.
radhika [Fri, 22 Jan 2016 18:34:07 +0000 (13:34 -0500)]
8198: consider X-Forwarded-For header by way of request.repote_ip while setting node ip address.
Ward Vandewege [Fri, 22 Jan 2016 15:27:59 +0000 (10:27 -0500)]
Make sure not to accidentally include config/application.yml and
config/database.yml in our rails packages.
No issue #
Peter Amstutz [Fri, 22 Jan 2016 15:26:14 +0000 (10:26 -0500)]
Bump libcloud package version pin to 0.20.2dev1 refs #8235
Peter Amstutz [Fri, 22 Jan 2016 13:23:48 +0000 (08:23 -0500)]
Fix python sdk tests refs #6833
Tom Clegg [Thu, 21 Jan 2016 22:25:48 +0000 (17:25 -0500)]
Merge branch '8281-arv-mount-retry' closes #8281
Peter Amstutz [Thu, 21 Jan 2016 22:25:22 +0000 (17:25 -0500)]
Merge branch '6833-test-token-expiry' closes #6833
Peter Amstutz [Thu, 21 Jan 2016 21:49:45 +0000 (16:49 -0500)]
Merge branch '7846-magic-invalidate-entry' closes #7846
Tom Clegg [Thu, 21 Jan 2016 21:10:11 +0000 (16:10 -0500)]
8281: Limit # write threads to #copies remaining, not #copies total.