arvados.git
3 years ago16997: Test map key order in check/dump. Fix duplicate warnings.
Tom Clegg [Mon, 26 Apr 2021 19:11:58 +0000 (15:11 -0400)]
16997: Test map key order in check/dump. Fix duplicate warnings.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17507-listobjectsv2'
Tom Clegg [Mon, 26 Apr 2021 14:14:15 +0000 (10:14 -0400)]
Merge branch '17507-listobjectsv2'

fixes #17507

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17343-workbench1-optional'
Tom Clegg [Mon, 26 Apr 2021 13:47:01 +0000 (09:47 -0400)]
Merge branch '17343-workbench1-optional'

closes #17343

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17521-dot-slash-glob' refs #17521
Peter Amstutz [Fri, 23 Apr 2021 21:02:41 +0000 (17:02 -0400)]
Merge branch '17521-dot-slash-glob' refs #17521

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17521: Use normpath to clean up the path segment.
Peter Amstutz [Fri, 23 Apr 2021 20:07:02 +0000 (16:07 -0400)]
17521: Use normpath to clean up the path segment.

Removes leading ./, trailing slashes, redundant '.' and blank path
segments, and other stuff that potentially confuses the path matching.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17521: Add tests, refs #10380 and refs #17521
Peter Amstutz [Fri, 23 Apr 2021 19:58:42 +0000 (15:58 -0400)]
17521: Add tests, refs #10380 and refs #17521

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17507: Fix calling Child() without lock.
Tom Clegg [Thu, 22 Apr 2021 20:14:38 +0000 (16:14 -0400)]
17507: Fix calling Child() without lock.

Encountered in services/keep-web tests.

START: handler_test.go:654: IntegrationSuite.TestDirectoryListingWithNoAnonymousToken
START: server_test.go:431: IntegrationSuite.SetUpTest

[...]

fatal error: concurrent map iteration and map write

goroutine 2192 [running]:
runtime.throw(0xe545ab, 0x26)
        /var/lib/arvados/go/src/runtime/panic.go:1112 +0x72 fp=0xc002544c78 sp=0xc002544c48 pc=0x4366b2
runtime.mapiternext(0xc002544d18)
        /var/lib/arvados/go/src/runtime/map.go:853 +0x552 fp=0xc002544cf8 sp=0xc002544c78 pc=0x411442
git.arvados.org/arvados.git/sdk/go/arvados.(*treenode).MemorySize(0xc00202e480, 0x0)
        /home/tom/arvados/sdk/go/arvados/fs_base.go:336 +0x106 fp=0xc002544d98 sp=0xc002544cf8 pc=0x8c7de6
git.arvados.org/arvados.git/sdk/go/arvados.(*treenode).MemorySize(0xc0003a66c0, 0x0)
        /home/tom/arvados/sdk/go/arvados/fs_base.go:337 +0xe3 fp=0xc002544e38 sp=0xc002544d98 pc=0x8c7dc3
git.arvados.org/arvados.git/sdk/go/arvados.(*treenode).MemorySize(0xc0020c3680, 0x0)
        /home/tom/arvados/sdk/go/arvados/fs_base.go:337 +0xe3 fp=0xc002544ed8 sp=0xc002544e38 pc=0x8c7dc3
git.arvados.org/arvados.git/sdk/go/arvados.(*fileSystem).MemorySize(0xc001af0000, 0x0)
        /home/tom/arvados/sdk/go/arvados/fs_base.go:631 +0x33 fp=0xc002544ef8 sp=0xc002544ed8 pc=0x8ca1e3
git.arvados.org/arvados.git/services/keep-web.(*cache).collectionBytes(0xc00261da08, 0x3ff0000000000101)
        /home/tom/arvados/services/keep-web/cache.go:448 +0x21c fp=0xc002544f70 sp=0xc002544ef8 pc=0xc2903c
git.arvados.org/arvados.git/services/keep-web.(*cache).updateGauges(0xc00261da08)
        /home/tom/arvados/services/keep-web/cache.go:170 +0x2f fp=0xc002544f90 sp=0xc002544f70 pc=0xc269bf
git.arvados.org/arvados.git/services/keep-web.(*cache).setup.func1(0xc00261da08)
        /home/tom/arvados/services/keep-web/cache.go:164 +0x6d fp=0xc002544fd8 sp=0xc002544f90 pc=0xc65dad
runtime.goexit()
        /var/lib/arvados/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc002544fe0 sp=0xc002544fd8 pc=0x468dd1
created by git.arvados.org/arvados.git/services/keep-web.(*cache).setup
        /home/tom/arvados/services/keep-web/cache.go:162 +0x1db

goroutine 2144 [runnable]:
git.arvados.org/arvados.git/sdk/go/arvados.(*treenode).Child(0xc00202e480, 0xc002c4e117, 0x9, 0xc0005c2070, 0x10c9680, 0xc0021d0240, 0x0, 0x0)
        /home/tom/arvados/sdk/go/arvados/fs_base.go:289 +0x171
git.arvados.org/arvados.git/sdk/go/arvados.(*lookupnode).Readdir(0xc00202e480, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/tom/arvados/sdk/go/arvados/fs_lookup.go:53 +0x27b
git.arvados.org/arvados.git/sdk/go/arvados.(*filehandle).Readdir(0xc00263c140, 0x0, 0xc002066360, 0xc00263c140, 0x10c4f40, 0xc00263c140, 0x0)
        /home/tom/arvados/sdk/go/arvados/fs_filehandle.go:81 +0x1d2
golang.org/x/net/webdav.walkFS(0x10bbe00, 0xc002b09590, 0x10bd900, 0xc002af1e60, 0xffffffffffffffff, 0xc00161a574, 0xc, 0x10c17c0, 0xc00227c280, 0xc0005c2628, ...)
        /home/tom/arvados/tmp/GOPATH/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/webdav/file.go:772 +0x25e
golang.org/x/net/webdav.walkFS(0x10bbe00, 0xc002b09590, 0x10bd900, 0xc002af1e60, 0xffffffffffffffff, 0xe2f941, 0x5, 0x10c17c0, 0xc002b1a4c0, 0xc0005c2628, ...)
        /home/tom/arvados/tmp/GOPATH/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/webdav/file.go:786 +0x564
golang.org/x/net/webdav.walkFS(0x10bbe00, 0xc002b09590, 0x10bd900, 0xc002af1e60, 0xffffffffffffffff, 0xc00216be76, 0x0, 0x10c17c0, 0xc002b1a0c0, 0xc0005c2628, ...)
        /home/tom/arvados/tmp/GOPATH/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/webdav/file.go:786 +0x564
golang.org/x/net/webdav.(*Handler).handlePropfind(0xc002b1a080, 0x7f1c582d9120, 0xc0029e5400, 0xc002b04900, 0xc0022bc718, 0x415ad3, 0xc002b1a080)
        /home/tom/arvados/tmp/GOPATH/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/webdav/webdav.go:566 +0x3d7
golang.org/x/net/webdav.(*Handler).ServeHTTP(0xc002b1a080, 0x7f1c582d9120, 0xc0029e5400, 0xc002b04900)
        /home/tom/arvados/tmp/GOPATH/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/webdav/webdav.go:67 +0x556
git.arvados.org/arvados.git/services/keep-web.(*handler).serveSiteFS(0xc002221ae0, 0x7f1c582d9120, 0xc0029e5400, 0xc002b04900, 0xc002b22010, 0x1, 0x1, 0x101)
        /home/tom/arvados/services/keep-web/handler.go:593 +0x80b
git.arvados.org/arvados.git/services/keep-web.(*handler).ServeHTTP(0xc002221ae0, 0x10b6ec0, 0xc002b094a0, 0xc002b04900)
        /home/tom/arvados/services/keep-web/handler.go:330 +0x2a0b
git.arvados.org/arvados.git/sdk/go/httpserver.LogRequests.func1(0x7f1c58319c58, 0xc002b09470, 0xc002b04800)
        /home/tom/arvados/sdk/go/httpserver/logger.go:56 +0x8d8
net/http.HandlerFunc.ServeHTTP(0xc002282460, 0x7f1c58319c58, 0xc002b09470, 0xc002b04800)
        /var/lib/arvados/go/src/net/http/server.go:2012 +0x44
git.arvados.org/arvados.git/sdk/go/httpserver.AddRequestIDs.func1(0x7f1c58319c58, 0xc002b09470, 0xc002b04800)
        /home/tom/arvados/sdk/go/httpserver/id_generator.go:57 +0x1a5
net/http.HandlerFunc.ServeHTTP(0xc002282480, 0x7f1c58319c58, 0xc002b09470, 0xc002b04800)
        /var/lib/arvados/go/src/net/http/server.go:2012 +0x44
git.arvados.org/arvados.git/sdk/go/httpserver.HandlerWithContext.func1(0x7f1c58319c58, 0xc002b09470, 0xc002b04700)
        /home/tom/arvados/sdk/go/httpserver/logger.go:30 +0x107
net/http.HandlerFunc.ServeHTTP(0xc002901b30, 0x7f1c58319c58, 0xc002b09470, 0xc002b04700)
        /var/lib/arvados/go/src/net/http/server.go:2012 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1(0x10b8bc0, 0xc002b1a000, 0xc002b04700)
        /home/tom/arvados/tmp/GOPATH/pkg/mod/github.com/prometheus/client_golang@v1.2.1/prometheus/promhttp/instrument_server.go:68 +0x11c
net/http.HandlerFunc.ServeHTTP(0xc002901e00, 0x10b8bc0, 0xc002b1a000, 0xc002b04700)
        /var/lib/arvados/go/src/net/http/server.go:2012 +0x44
git.arvados.org/arvados.git/sdk/go/httpserver.(*metrics).ServeHTTP(0xc002113e00, 0x10b8bc0, 0xc002b1a000, 0xc002b04700)
        /home/tom/arvados/sdk/go/httpserver/metrics.go:70 +0x51
git.arvados.org/arvados.git/services/keep-web.(*IntegrationSuite).testDirectoryListing(0xc000010138, 0xc0007d41e0)
        /home/tom/arvados/services/keep-web/handler_test.go:864 +0x1a07
git.arvados.org/arvados.git/services/keep-web.(*IntegrationSuite).TestDirectoryListingWithNoAnonymousToken(0xc000010138, 0xc0007d41e0)
        /home/tom/arvados/services/keep-web/handler_test.go:656 +0x67
reflect.Value.call(0xe2bbc0, 0xc000010138, 0x4613, 0xe2deb6, 0x4, 0xc000282f08, 0x1, 0x1, 0x171d800, 0xc000282e48, ...)
        /var/lib/arvados/go/src/reflect/value.go:460 +0x8ab
reflect.Value.Call(0xe2bbc0, 0xc000010138, 0x4613, 0xc000282f08, 0x1, 0x1, 0xc0007d42d0, 0xc000128060, 0xc00287a720)
        /var/lib/arvados/go/src/reflect/value.go:321 +0xb4
gopkg.in/check%2ev1.(*suiteRunner).forkTest.func1(0xc0007d41e0)
        /home/tom/arvados/tmp/GOPATH/pkg/mod/gopkg.in/check.v1@v1.0.0-20161208181325-20d25e280405/check.go:772 +0x628
gopkg.in/check%2ev1.(*suiteRunner).forkCall.func1(0xc000174c00, 0xc0007d41e0, 0xc00222b760)
        /home/tom/arvados/tmp/GOPATH/pkg/mod/gopkg.in/check.v1@v1.0.0-20161208181325-20d25e280405/check.go:666 +0x98
created by gopkg.in/check%2ev1.(*suiteRunner).forkCall
        /home/tom/arvados/tmp/GOPATH/pkg/mod/gopkg.in/check.v1@v1.0.0-20161208181325-20d25e280405/check.go:663 +0x1fb

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17507: Support ListObjectsV2 API.
Tom Clegg [Thu, 22 Apr 2021 19:12:17 +0000 (15:12 -0400)]
17507: Support ListObjectsV2 API.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17566-max-request-size'
Tom Clegg [Wed, 21 Apr 2021 20:25:59 +0000 (16:25 -0400)]
Merge branch '17566-max-request-size'

fixes #17566

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17204-not-modified'
Tom Clegg [Wed, 21 Apr 2021 20:25:37 +0000 (16:25 -0400)]
Merge branch '17204-not-modified'

fixes #17204

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17346-keep-balance-empty-block'
Tom Clegg [Wed, 21 Apr 2021 19:50:27 +0000 (15:50 -0400)]
Merge branch '17346-keep-balance-empty-block'

fixes #17346

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17343: Add option to boot without workbench1 to speed up tests.
Tom Clegg [Wed, 21 Apr 2021 19:03:59 +0000 (15:03 -0400)]
17343: Add option to boot without workbench1 to speed up tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17570-remove-extra-example-agrantfile'
Javier Bértoli [Wed, 21 Apr 2021 15:10:24 +0000 (12:10 -0300)]
Merge branch '17570-remove-extra-example-agrantfile'

closes #17570
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(provision): remove unneeded vagantfile
Javier Bértoli [Wed, 21 Apr 2021 15:01:30 +0000 (12:01 -0300)]
fix(provision): remove unneeded vagantfile

refs #17570
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17204: Fix misleading log message on 304 Not Modified response.
Tom Clegg [Tue, 20 Apr 2021 23:43:57 +0000 (19:43 -0400)]
17204: Fix misleading log message on 304 Not Modified response.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17528-install-warnings'
Tom Clegg [Tue, 20 Apr 2021 20:04:22 +0000 (16:04 -0400)]
Merge branch '17528-install-warnings'

fixes #17528

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17437-drop-content-length-header'
Tom Clegg [Tue, 20 Apr 2021 19:04:14 +0000 (15:04 -0400)]
Merge branch '17437-drop-content-length-header'

refs #17437

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17246-salt-install-improvements'
Javier Bértoli [Tue, 20 Apr 2021 18:35:48 +0000 (15:35 -0300)]
Merge branch '17246-salt-install-improvements'

closes #17246

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofeat(provision): pin arvados version to latest stable
Javier Bértoli [Mon, 19 Apr 2021 18:52:17 +0000 (15:52 -0300)]
feat(provision): pin arvados version to latest stable

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17437: Don't copy incoming Content-Length header when proxying.
Tom Clegg [Tue, 20 Apr 2021 16:17:30 +0000 (12:17 -0400)]
17437: Don't copy incoming Content-Length header when proxying.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17566: Don't bypass controller in SDK integration tests.
Tom Clegg [Mon, 19 Apr 2021 22:04:36 +0000 (18:04 -0400)]
17566: Don't bypass controller in SDK integration tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17566: Allow large request bodies in integration tests.
Tom Clegg [Mon, 19 Apr 2021 22:04:14 +0000 (18:04 -0400)]
17566: Allow large request bodies in integration tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17566: Use configured MaxRequestSize.
Tom Clegg [Mon, 19 Apr 2021 22:01:39 +0000 (18:01 -0400)]
17566: Use configured MaxRequestSize.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agofix(provision): default to using the master branch of arvados-formula
Javier Bértoli [Mon, 19 Apr 2021 17:53:10 +0000 (14:53 -0300)]
fix(provision): default to using the master branch of arvados-formula

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agodocs(provision): improve the documentation in the examples
Javier Bértoli [Mon, 19 Apr 2021 13:00:43 +0000 (10:00 -0300)]
docs(provision): improve the documentation in the examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agorefactor(provision): add snakeoil certs in single host multiple hostnames
Javier Bértoli [Fri, 16 Apr 2021 22:05:34 +0000 (19:05 -0300)]
refactor(provision): add snakeoil certs in single host multiple hostnames

We could use the example states in the formula, but added them here
for the sake of completeness

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agodocs(provision): improve the documentation in the examples
Javier Bértoli [Fri, 16 Apr 2021 22:02:27 +0000 (19:02 -0300)]
docs(provision): improve the documentation in the examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agorefactor(provision): add host entries in single host multiple hostnames
Javier Bértoli [Fri, 16 Apr 2021 22:01:12 +0000 (19:01 -0300)]
refactor(provision): add host entries in single host multiple hostnames

We could use the example states in the formula, but added them here
for the sake of completeness

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agochore(provision): enable both examples in vagrant
Javier Bértoli [Fri, 16 Apr 2021 22:00:07 +0000 (19:00 -0300)]
chore(provision): enable both examples in vagrant

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agodocs(provision): explain options in config examples
Javier Bértoli [Fri, 16 Apr 2021 19:09:13 +0000 (16:09 -0300)]
docs(provision): explain options in config examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofeat(provision): add some checks before run
Javier Bértoli [Fri, 16 Apr 2021 19:07:57 +0000 (16:07 -0300)]
feat(provision): add some checks before run

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agorefactor(provision): delete parameters moved to config files
Javier Bértoli [Fri, 16 Apr 2021 19:06:08 +0000 (16:06 -0300)]
refactor(provision): delete parameters moved to config files

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agochore(provision): ignore some files in git
Javier Bértoli [Fri, 16 Apr 2021 19:04:15 +0000 (16:04 -0300)]
chore(provision): ignore some files in git

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agorefactor(provision): use uniform database parameters across examples
Javier Bértoli [Fri, 16 Apr 2021 19:01:52 +0000 (16:01 -0300)]
refactor(provision): use uniform database parameters across examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoMerge branch '17530-arvados-client-fastfail-take2'
Nico Cesar [Fri, 16 Apr 2021 21:00:42 +0000 (17:00 -0400)]
Merge branch '17530-arvados-client-fastfail-take2'

closes #17530

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years agofix(provision): run hosts/snakeoil states before nginx so it does not fail on missing...
Javier Bértoli [Fri, 16 Apr 2021 18:58:34 +0000 (15:58 -0300)]
fix(provision): run hosts/snakeoil states before nginx so it does not fail on missing deps

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agochore(provision): set postgresql cidr in single host examples
Javier Bértoli [Fri, 16 Apr 2021 18:37:07 +0000 (15:37 -0300)]
chore(provision): set postgresql cidr in single host examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(provision): use uniform database parameters across examples
Javier Bértoli [Fri, 16 Apr 2021 18:36:18 +0000 (15:36 -0300)]
fix(provision): use uniform database parameters across examples

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17530: Added check if API host is missing
Nico Cesar [Thu, 15 Apr 2021 16:26:42 +0000 (12:26 -0400)]
17530: Added check if API host is missing

CallRaw() will have a retry logic so 3s * 20 = 60s to fail,
and it will be explicit why this is happening.

KeepClient.discoverServices() to make it fail faster

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years ago17346: Add test case.
Tom Clegg [Fri, 16 Apr 2021 14:48:01 +0000 (10:48 -0400)]
17346: Add test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17529-listinstances-rate-limit'
Tom Clegg [Fri, 16 Apr 2021 14:44:07 +0000 (10:44 -0400)]
Merge branch '17529-listinstances-rate-limit'

refs #17529

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17529: Heed MaxCloudOpsPerSecond in Instances() and returned insts.
Tom Clegg [Fri, 16 Apr 2021 13:59:01 +0000 (09:59 -0400)]
17529: Heed MaxCloudOpsPerSecond in Instances() and returned insts.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17529-ec2-rate-limit'
Tom Clegg [Fri, 16 Apr 2021 13:40:25 +0000 (09:40 -0400)]
Merge branch '17529-ec2-rate-limit'

fixes #17529

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17528: Set ready=0 if config is incomplete/missing during install.
Tom Clegg [Thu, 15 Apr 2021 20:17:03 +0000 (16:17 -0400)]
17528: Set ready=0 if config is incomplete/missing during install.

Reverts accidental change.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17171-pysdk-timeout-fix'
Lucas Di Pentima [Thu, 15 Apr 2021 20:03:45 +0000 (17:03 -0300)]
Merge branch '17171-pysdk-timeout-fix'
Closes #17171

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17528: Fix assets:precompile @ build time, skip at install time.
Tom Clegg [Thu, 15 Apr 2021 20:02:11 +0000 (16:02 -0400)]
17528: Fix assets:precompile @ build time, skip at install time.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17171: Sets default timeout on arvados.api() to 10 seconds.
Lucas Di Pentima [Thu, 15 Apr 2021 19:53:41 +0000 (16:53 -0300)]
17171: Sets default timeout on arvados.api() to 10 seconds.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17528: Remove bin/yarn from Rails projects.
Tom Clegg [Thu, 15 Apr 2021 19:45:40 +0000 (15:45 -0400)]
17528: Remove bin/yarn from Rails projects.

"rake assets:precompile" uses the existence of the bin/yarn stub to
decide whether yarn is needed.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17528: Don't load config when doing assets:precompile in postinst.
Tom Clegg [Thu, 15 Apr 2021 18:55:44 +0000 (14:55 -0400)]
17528: Don't load config when doing assets:precompile in postinst.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17529: Update comment.
Tom Clegg [Thu, 15 Apr 2021 18:21:55 +0000 (14:21 -0400)]
17529: Update comment.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17529: Catch rate limit errors from DescribeInstances, too.
Tom Clegg [Thu, 15 Apr 2021 18:20:04 +0000 (14:20 -0400)]
17529: Catch rate limit errors from DescribeInstances, too.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17529: Heed MaxCloudOpsPerSecond when calling SetTags.
Tom Clegg [Thu, 15 Apr 2021 18:03:13 +0000 (14:03 -0400)]
17529: Heed MaxCloudOpsPerSecond when calling SetTags.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17171: Merge branch 'master' into 17171-pysdk-timeout-fix
Lucas Di Pentima [Thu, 15 Apr 2021 17:10:01 +0000 (14:10 -0300)]
17171: Merge branch 'master' into 17171-pysdk-timeout-fix

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17171: Fixes the bug by removing the default timeout value.
Lucas Di Pentima [Thu, 15 Apr 2021 17:08:07 +0000 (14:08 -0300)]
17171: Fixes the bug by removing the default timeout value.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17171: Exposes bug with a test.
Lucas Di Pentima [Thu, 15 Apr 2021 17:07:36 +0000 (14:07 -0300)]
17171: Exposes bug with a test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoMerge branch '17531-remove-forcelegacyapi14-config'
Tom Clegg [Thu, 15 Apr 2021 14:45:42 +0000 (10:45 -0400)]
Merge branch '17531-remove-forcelegacyapi14-config'

closes #17531

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17462-secondaryfiles-regression' refs #17462
Peter Amstutz [Thu, 15 Apr 2021 14:23:00 +0000 (10:23 -0400)]
Merge branch '17462-secondaryfiles-regression' refs #17462

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17462: Add integration test to confirm fix
Peter Amstutz [Thu, 15 Apr 2021 14:18:07 +0000 (10:18 -0400)]
17462: Add integration test to confirm fix

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17346: Don't try to rebalance the zero-length data block.
Tom Clegg [Thu, 15 Apr 2021 13:46:37 +0000 (09:46 -0400)]
17346: Don't try to rebalance the zero-length data block.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoFix regression refs #17462
Peter Amstutz [Wed, 14 Apr 2021 20:13:26 +0000 (16:13 -0400)]
Fix regression refs #17462

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17529: Return RequestLimitExceeded as a RateLimitError.
Tom Clegg [Wed, 14 Apr 2021 19:40:34 +0000 (15:40 -0400)]
17529: Return RequestLimitExceeded as a RateLimitError.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17522-arvput-stdin-transcode-fix'
Lucas Di Pentima [Wed, 14 Apr 2021 18:34:44 +0000 (15:34 -0300)]
Merge branch '17522-arvput-stdin-transcode-fix'
Closes #17522

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17522: Fixes bug, corrects updated test's assertion.
Lucas Di Pentima [Wed, 14 Apr 2021 18:14:21 +0000 (15:14 -0300)]
17522: Fixes bug, corrects updated test's assertion.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17531: Remove ForceLegacyAPI14 config flag.
Tom Clegg [Wed, 14 Apr 2021 17:56:59 +0000 (13:56 -0400)]
17531: Remove ForceLegacyAPI14 config flag.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17522: Exposes bug through test. Also fixes help message.
Lucas Di Pentima [Wed, 14 Apr 2021 17:52:49 +0000 (14:52 -0300)]
17522: Exposes bug through test. Also fixes help message.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agofix(provision): temp fix for nginx formula issue with snippets
Javier Bértoli [Tue, 13 Apr 2021 21:20:12 +0000 (18:20 -0300)]
fix(provision): temp fix for nginx formula issue with snippets

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofeat(provision): add local.params example for multi-host
Javier Bértoli [Tue, 13 Apr 2021 21:19:29 +0000 (18:19 -0300)]
feat(provision): add local.params example for multi-host

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoMerge branch '16159-token-expiration-on-logout'
Lucas Di Pentima [Tue, 13 Apr 2021 20:35:00 +0000 (17:35 -0300)]
Merge branch '16159-token-expiration-on-logout'
Refs #16159

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoMerge branch '17462-default-secondaryfiles' refs #17462
Peter Amstutz [Tue, 13 Apr 2021 20:25:09 +0000 (16:25 -0400)]
Merge branch '17462-default-secondaryfiles' refs #17462

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17462: Fix tests
Peter Amstutz [Tue, 13 Apr 2021 19:23:26 +0000 (15:23 -0400)]
17462: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agofix(provision): use nginx-formula PR branch until changes are merged
Javier Bértoli [Tue, 13 Apr 2021 19:23:17 +0000 (16:23 -0300)]
fix(provision): use nginx-formula PR branch until changes are merged

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agodocs(provision): add salt usage with roles in multi-host
Javier Bértoli [Wed, 24 Mar 2021 18:25:33 +0000 (15:25 -0300)]
docs(provision): add salt usage with roles in multi-host

refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoMerge branch '17498-python3-migration-metapackages'
Ward Vandewege [Tue, 13 Apr 2021 15:53:10 +0000 (11:53 -0400)]
Merge branch '17498-python3-migration-metapackages'

closes #17498

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17498: address review comments.
Ward Vandewege [Tue, 13 Apr 2021 15:51:39 +0000 (11:51 -0400)]
17498: address review comments.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17498: Build metapackages to ease the migration from python 2 to
Ward Vandewege [Tue, 13 Apr 2021 14:57:25 +0000 (10:57 -0400)]
17498: Build metapackages to ease the migration from python 2 to
       python3.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17498: add a note about the python 3 package names in the upgrade guide.
Ward Vandewege [Tue, 13 Apr 2021 14:55:24 +0000 (10:55 -0400)]
17498: add a note about the python 3 package names in the upgrade guide.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16159: Merge branch 'master' into 16159-token-expiration-on-logout
Lucas Di Pentima [Tue, 13 Apr 2021 14:39:01 +0000 (11:39 -0300)]
16159: Merge branch 'master' into 16159-token-expiration-on-logout

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago16159: Adds tests checking token expiration at logout.
Lucas Di Pentima [Tue, 13 Apr 2021 14:29:42 +0000 (11:29 -0300)]
16159: Adds tests checking token expiration at logout.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoMerge branch '17392-go-write-to-storage-class'
Tom Clegg [Mon, 12 Apr 2021 18:59:13 +0000 (14:59 -0400)]
Merge branch '17392-go-write-to-storage-class'

closes #17392

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17392: Test satisfying storage classes via multiple servers.
Tom Clegg [Mon, 12 Apr 2021 14:52:22 +0000 (10:52 -0400)]
17392: Test satisfying storage classes via multiple servers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16159: Provide a localdb transaction for federation tests.
Tom Clegg [Fri, 9 Apr 2021 17:18:55 +0000 (13:18 -0400)]
16159: Provide a localdb transaction for federation tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17462: Fix fetch_text() signature
Peter Amstutz [Wed, 7 Apr 2021 20:51:46 +0000 (16:51 -0400)]
17462: Fix fetch_text() signature

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17462: Bump cwltool for scandeps bugfix
Peter Amstutz [Wed, 7 Apr 2021 20:14:10 +0000 (16:14 -0400)]
17462: Bump cwltool for scandeps bugfix

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16159: Expires tokens on logout on different login controllers.
Lucas Di Pentima [Tue, 6 Apr 2021 20:46:00 +0000 (17:46 -0300)]
16159: Expires tokens on logout on different login controllers.

Also, optimizes the DB query a bit.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17392: Ensure requested storage classes are satisfied on write.
Tom Clegg [Tue, 6 Apr 2021 15:28:14 +0000 (11:28 -0400)]
17392: Ensure requested storage classes are satisfied on write.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '13382-write-to-storage-class'
Tom Clegg [Tue, 6 Apr 2021 14:25:26 +0000 (10:25 -0400)]
Merge branch '13382-write-to-storage-class'

closes #13382

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16159: Pass token to expire to controller on logout.
Lucas Di Pentima [Mon, 5 Apr 2021 21:35:08 +0000 (18:35 -0300)]
16159: Pass token to expire to controller on logout.

Also, adds a warning on the "current token" page about token's transciency.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago13382: Rename putResult -> putProgress.
Tom Clegg [Mon, 5 Apr 2021 18:02:40 +0000 (14:02 -0400)]
13382: Rename putResult -> putProgress.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoFix golint warning.
Ward Vandewege [Fri, 2 Apr 2021 23:36:37 +0000 (19:36 -0400)]
Fix golint warning.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16159: Adds error logging on token expiration failure. Fixes SQL query.
Lucas Di Pentima [Fri, 2 Apr 2021 14:44:14 +0000 (11:44 -0300)]
16159: Adds error logging on token expiration failure. Fixes SQL query.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoSimplify code a bit.
Ward Vandewege [Fri, 2 Apr 2021 01:09:41 +0000 (21:09 -0400)]
Simplify code a bit.

refs #17503

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago13382: Test storage class failure cases.
Tom Clegg [Thu, 1 Apr 2021 14:59:30 +0000 (10:59 -0400)]
13382: Test storage class failure cases.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago13382: Use caller-specified storage classes when writing.
Tom Clegg [Wed, 31 Mar 2021 20:56:14 +0000 (16:56 -0400)]
13382: Use caller-specified storage classes when writing.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago13382: Report storage class(es) in headers after successful write.
Tom Clegg [Thu, 25 Mar 2021 14:57:52 +0000 (10:57 -0400)]
13382: Report storage class(es) in headers after successful write.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '17508-arvkeepdocker-fix'
Lucas Di Pentima [Thu, 1 Apr 2021 14:32:54 +0000 (11:32 -0300)]
Merge branch '17508-arvkeepdocker-fix'
Closes #17508

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17508: Fixes unit test.
Lucas Di Pentima [Thu, 1 Apr 2021 14:05:20 +0000 (11:05 -0300)]
17508: Fixes unit test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17508: Fixes arv-keepdocker bug.
Lucas Di Pentima [Wed, 31 Mar 2021 20:32:03 +0000 (17:32 -0300)]
17508: Fixes arv-keepdocker bug.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17508: Exposes bug with a test.
Lucas Di Pentima [Wed, 31 Mar 2021 22:05:51 +0000 (19:05 -0300)]
17508: Exposes bug with a test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoFix deduplication report test.
Ward Vandewege [Wed, 31 Mar 2021 19:34:14 +0000 (15:34 -0400)]
Fix deduplication report test.

refs #17503

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoAdd test cases for special chars in filename.
Tom Clegg [Wed, 31 Mar 2021 15:45:16 +0000 (11:45 -0400)]
Add test cases for special chars in filename.

closes #17422

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>