arvados-dev.git
2 years agoFor running our tests, FUSE must be configured with the
Ward Vandewege [Fri, 20 May 2022 19:40:43 +0000 (15:40 -0400)]
For running our tests, FUSE must be configured with the
'user_allow_other' option enabled for Crunch to set up Keep mounts that
are readable by containers.

refs #15370

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoBump the buster base image for our test image.
Ward Vandewege [Fri, 20 May 2022 19:32:49 +0000 (15:32 -0400)]
Bump the buster base image for our test image.

refs #15370

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoAdd docker to our tests image.
Ward Vandewege [Fri, 20 May 2022 16:38:58 +0000 (12:38 -0400)]
Add docker to our tests image.

refs #15370

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoAdd a convenient way to disable our git hooks via an environment
Ward Vandewege [Thu, 19 May 2022 17:41:29 +0000 (13:41 -0400)]
Add a convenient way to disable our git hooks via an environment
variable.

refs #18872

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoOur base test image installs the correct version of Go (detected in the
Ward Vandewege [Wed, 13 Apr 2022 16:53:25 +0000 (12:53 -0400)]
Our base test image installs the correct version of Go (detected in the
usual way from the Arvados source tree).

Check out the git tree using the git protocol, not http.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoMerge branch '16375-arvados-multinode-testing'
Javier Bértoli [Tue, 12 Apr 2022 14:17:47 +0000 (11:17 -0300)]
Merge branch '16375-arvados-multinode-testing'

refs #16375
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

2 years ago16375: add code to test arvados in multi-node
Javier Bértoli [Wed, 6 Apr 2022 22:55:47 +0000 (19:55 -0300)]
16375: add code to test arvados in multi-node

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

2 years agoUpdate build process a bit
Peter Amstutz [Fri, 8 Apr 2022 14:17:34 +0000 (10:17 -0400)]
Update build process a bit

refs #18701

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoFix skips no issue #
Peter Amstutz [Wed, 6 Apr 2022 21:08:22 +0000 (17:08 -0400)]
Fix skips no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoSkip invalid cwl tests
Peter Amstutz [Wed, 6 Apr 2022 20:26:06 +0000 (16:26 -0400)]
Skip invalid cwl tests

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agocwl test Fetch tags
Peter Amstutz [Tue, 5 Apr 2022 20:16:15 +0000 (16:16 -0400)]
cwl test Fetch tags

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoRun the v1.2.0 tagged test suite instead of 'main'
Peter Amstutz [Tue, 5 Apr 2022 20:05:51 +0000 (16:05 -0400)]
Run the v1.2.0 tagged test suite instead of 'main'

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoAdjust position of "Publish stable arvados/jobs Docker image"
Peter Amstutz [Thu, 24 Mar 2022 15:30:38 +0000 (11:30 -0400)]
Adjust position of "Publish stable arvados/jobs Docker image"

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoAdd the zenodo.org step to the release tasks.
Ward Vandewege [Thu, 24 Mar 2022 21:17:15 +0000 (17:17 -0400)]
Add the zenodo.org step to the release tasks.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoReview reminder script: fix another edge case on the date the sprint
Ward Vandewege [Wed, 16 Mar 2022 14:25:36 +0000 (10:25 -0400)]
Review reminder script: fix another edge case on the date the sprint
turns over.

refs #18813

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoBump the packer version in the "build" image to 1.8.0.
Ward Vandewege [Tue, 15 Mar 2022 16:36:02 +0000 (12:36 -0400)]
Bump the packer version in the "build" image to 1.8.0.

refs #18772

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoReview reminder script: review task titles can start with a capitalized
Ward Vandewege [Fri, 11 Mar 2022 16:03:08 +0000 (11:03 -0500)]
Review reminder script: review task titles can start with a capitalized
word.

refs #18813

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoBump the version of lib/redmine to latest to get pagination bugfix. Add
Ward Vandewege [Thu, 10 Mar 2022 18:21:22 +0000 (13:21 -0500)]
Bump the version of lib/redmine to latest to get pagination bugfix. Add
a few more lines to the debug log.

refs #18813

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoA few bugfixes in the redmine library: fix the version filter for
Ward Vandewege [Thu, 10 Mar 2022 18:11:26 +0000 (13:11 -0500)]
A few bugfixes in the redmine library: fix the version filter for
issues, handle pagination in the FilteredIssues function.

refs #18813

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoMerge branch '18813-review-reminders'
Ward Vandewege [Fri, 4 Mar 2022 20:17:04 +0000 (15:17 -0500)]
Merge branch '18813-review-reminders'

closes #18813

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years ago18813: add script to send review reminders for the current sprint. 18813-review-reminders
Ward Vandewege [Fri, 4 Mar 2022 14:46:27 +0000 (09:46 -0500)]
18813: add script to send review reminders for the current sprint.
       Extend our redmine library as needed.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoNote about building compute image
Peter Amstutz [Thu, 9 Dec 2021 16:51:22 +0000 (11:51 -0500)]
Note about building compute image

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoUpdate TASKS to sync with checklist
Peter Amstutz [Thu, 9 Dec 2021 16:33:14 +0000 (11:33 -0500)]
Update TASKS to sync with checklist

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18093-create-release-ticket'
Ward Vandewege [Fri, 19 Nov 2021 22:16:54 +0000 (17:16 -0500)]
Merge branch '18093-create-release-ticket'

closes #18093

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18093: Merge branch 'main' into 18093-create-release-ticket 18093-create-release-ticket
Ward Vandewege [Fri, 19 Nov 2021 22:15:08 +0000 (17:15 -0500)]
18093: Merge branch 'main' into 18093-create-release-ticket

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18093: implement review feedback
Ward Vandewege [Fri, 19 Nov 2021 19:29:39 +0000 (14:29 -0500)]
18093: implement review feedback

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoAptly does not strip spaces from architecture names, sigh.
Ward Vandewege [Thu, 18 Nov 2021 21:47:17 +0000 (16:47 -0500)]
Aptly does not strip spaces from architecture names, sigh.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18093: add functionality to create a new release checklist ticket in
Ward Vandewege [Thu, 18 Nov 2021 01:18:00 +0000 (20:18 -0500)]
18093: add functionality to create a new release checklist ticket in
       Redmine.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUpdate go dependencies for the compute image cleaner.
Ward Vandewege [Tue, 16 Nov 2021 21:43:29 +0000 (16:43 -0500)]
Update go dependencies for the compute image cleaner.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18300-release-ticket-association'
Ward Vandewege [Tue, 16 Nov 2021 20:36:11 +0000 (15:36 -0500)]
Merge branch '18300-release-ticket-association'

closes #18300

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18300: Merge branch 'main' into 18300-release-ticket-association 18300-release-ticket-association
Ward Vandewege [Tue, 16 Nov 2021 20:33:06 +0000 (15:33 -0500)]
18300: Merge branch 'main' into 18300-release-ticket-association

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18300: add the initial version of `art`, the Arvados Release Tool. It
Ward Vandewege [Fri, 12 Nov 2021 20:59:46 +0000 (15:59 -0500)]
18300: add the initial version of `art`, the Arvados Release Tool. It
       can associate redmine tickets with a redmine release, based on a
       git commit history range.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18264-cwl-testing'
Ward Vandewege [Thu, 28 Oct 2021 17:39:27 +0000 (13:39 -0400)]
Merge branch '18264-cwl-testing'

refs #18264

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18264: simplify after review feedback. 18264-cwl-testing
Ward Vandewege [Thu, 28 Oct 2021 17:02:12 +0000 (13:02 -0400)]
18264: simplify after review feedback.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18264: switch to running the 1.2 version of the CWL conformance tests.
Ward Vandewege [Tue, 12 Oct 2021 19:52:41 +0000 (15:52 -0400)]
18264: switch to running the 1.2 version of the CWL conformance tests.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18264: clean up the way we run the CWL tests. Only run the upstream
Ward Vandewege [Tue, 12 Oct 2021 18:03:09 +0000 (14:03 -0400)]
18264: clean up the way we run the CWL tests. Only run the upstream
suite from a new run-cwl-test-suite.sh script (the arvados CWL tests are
run separately now, directly from CI). Remove old run-cwl-test.sh script.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix output when a token is suppressed.
Ward Vandewege [Wed, 6 Oct 2021 23:17:02 +0000 (19:17 -0400)]
Fix output when a token is suppressed.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoBump the debian-10 base image for all our packer images.
Ward Vandewege [Mon, 4 Oct 2021 16:41:07 +0000 (12:41 -0400)]
Bump the debian-10 base image for all our packer images.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoSkip docker_entrypoint test refs #18223
Peter Amstutz [Wed, 29 Sep 2021 18:51:15 +0000 (14:51 -0400)]
Skip docker_entrypoint test refs #18223

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoBump the Debian base image for our main Jenkins image, and increase the
Ward Vandewege [Mon, 27 Sep 2021 17:08:13 +0000 (13:08 -0400)]
Bump the Debian base image for our main Jenkins image, and increase the
disk space to 20 GB (I ran out of space generating a new image).

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoBump the debian-10 base image for all our packer images.
Ward Vandewege [Tue, 17 Aug 2021 20:25:30 +0000 (16:25 -0400)]
Bump the debian-10 base image for all our packer images.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMake sure the workbench2 test image has a much higher filesystem
Ward Vandewege [Tue, 10 Aug 2021 20:32:04 +0000 (16:32 -0400)]
Make sure the workbench2 test image has a much higher filesystem
watchers limit than the default 8192, since React eats those things for
breakfast.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUse apt-get instead of apt so debian/ubuntu don't complain
Javier Bértoli [Thu, 5 Aug 2021 18:08:51 +0000 (15:08 -0300)]
Use apt-get instead of apt so debian/ubuntu don't complain

refs #17535
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoOnly add Debian's backport in Debian images
Javier Bértoli [Thu, 5 Aug 2021 17:38:11 +0000 (14:38 -0300)]
Only add Debian's backport in Debian images

refs #17535
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoDebian/Ubuntu: os-release has an extra string to remove
Javier Bértoli [Thu, 5 Aug 2021 17:07:12 +0000 (14:07 -0300)]
Debian/Ubuntu: os-release has an extra string to remove

refs #17535
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoMerge branch '17535-test-provision-jenkins'
Javier Bértoli [Thu, 5 Aug 2021 16:08:36 +0000 (13:08 -0300)]
Merge branch '17535-test-provision-jenkins'

refs #17535
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoUpdate run_upload_packages.py: add debian 11 (bullseye), remove some no
Ward Vandewege [Thu, 29 Jul 2021 20:49:02 +0000 (16:49 -0400)]
Update run_upload_packages.py: add debian 11 (bullseye), remove some no
longer supported distributions.

refs #17949

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoAdd script to create new release repositories.
Ward Vandewege [Thu, 29 Jul 2021 19:13:38 +0000 (15:13 -0400)]
Add script to create new release repositories.

refs #17949

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17535: missing command
Javier Bértoli [Thu, 22 Jul 2021 20:06:01 +0000 (17:06 -0300)]
17535: missing command

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17535: typo
Javier Bértoli [Thu, 22 Jul 2021 19:56:13 +0000 (16:56 -0300)]
17535: typo

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17535: Use the backport line only in the Debian family
Javier Bértoli [Thu, 22 Jul 2021 19:46:56 +0000 (16:46 -0300)]
17535: Use the backport line only in the Debian family

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17535: typo
Javier Bértoli [Thu, 22 Jul 2021 19:36:26 +0000 (16:36 -0300)]
17535: typo

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17535: Allow to specify distro as part of the image name
Javier Bértoli [Thu, 22 Jul 2021 19:16:57 +0000 (16:16 -0300)]
17535: Allow to specify distro as part of the image name

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17535: Add a basic instance template for provision testing
Javier Bértoli [Thu, 22 Jul 2021 16:52:41 +0000 (13:52 -0300)]
17535: Add a basic instance template for provision testing

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17535: Re-add netcat to the images
Javier Bértoli [Thu, 22 Jul 2021 16:11:35 +0000 (13:11 -0300)]
17535: Re-add netcat to the images

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17535: Add missing Debian's pre-install command
Javier Bértoli [Thu, 22 Jul 2021 15:43:31 +0000 (12:43 -0300)]
17535: Add missing Debian's pre-install command

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17535: Modify the image-common script so it can work both in centos/debian
Javier Bértoli [Thu, 22 Jul 2021 15:37:19 +0000 (12:37 -0300)]
17535: Modify the image-common script so it can work both in centos/debian

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoUpdate check-copyright-headers.rb so it can handle the first push to an
Ward Vandewege [Fri, 16 Jul 2021 18:05:22 +0000 (14:05 -0400)]
Update check-copyright-headers.rb so it can handle the first push to an
empty repository.

refs #17817

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoAllow building a wb2 jenkins image from a specific wb2 git commit.
Ward Vandewege [Tue, 6 Jul 2021 21:23:15 +0000 (17:23 -0400)]
Allow building a wb2 jenkins image from a specific wb2 git commit.

refs #17782

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17844-create-an-image-for-arvados-formula-testing'
Javier Bértoli [Thu, 1 Jul 2021 20:22:15 +0000 (17:22 -0300)]
Merge branch '17844-create-an-image-for-arvados-formula-testing'

closes #17844
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoThe bleeding edge branch on the arvados repo is now called 'main'.
Ward Vandewege [Thu, 1 Jul 2021 18:48:53 +0000 (14:48 -0400)]
The bleeding edge branch on the arvados repo is now called 'main'.

refs #17817

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17844: use curl to get files
Javier Bértoli [Thu, 1 Jul 2021 17:53:09 +0000 (14:53 -0300)]
17844: use curl to get files

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17844: use curl to get files
Javier Bértoli [Thu, 1 Jul 2021 17:39:05 +0000 (14:39 -0300)]
17844: use curl to get files

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoBump packer version to 1.7.2. Checkout arvados-dev from the new URL on
Ward Vandewege [Thu, 1 Jul 2021 15:25:47 +0000 (11:25 -0400)]
Bump packer version to 1.7.2. Checkout arvados-dev from the new URL on
github.

refs #17817

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17844: add a template to build an image to test the Saltstack's arvados-formula
Javier Bértoli [Wed, 30 Jun 2021 21:25:32 +0000 (18:25 -0300)]
17844: add a template to build an image to test the Saltstack's arvados-formula

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoUpdate our commit hooks to use 'main' as the bleeding edge branch.
Ward Vandewege [Thu, 17 Jun 2021 14:24:50 +0000 (10:24 -0400)]
Update our commit hooks to use 'main' as the bleeding edge branch.

refs #17817

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoIn preparation of the migration to 'main', make a copy of the commit
Ward Vandewege [Thu, 17 Jun 2021 13:44:25 +0000 (09:44 -0400)]
In preparation of the migration to 'main', make a copy of the commit
hooks that reference the old 'master' branch, so that we can migrate
gradually.

refs #17817

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoPreseed the run-tests.sh cache to make test runs faster (bugfix).
Ward Vandewege [Fri, 7 May 2021 15:56:09 +0000 (11:56 -0400)]
Preseed the run-tests.sh cache to make test runs faster (bugfix).

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoPreseed the run-tests.sh cache to make test runs faster.
Ward Vandewege [Thu, 6 May 2021 16:01:47 +0000 (12:01 -0400)]
Preseed the run-tests.sh cache to make test runs faster.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoWe need build-essential because of gihub.com/msteinert/pam which
Ward Vandewege [Wed, 5 May 2021 15:31:45 +0000 (11:31 -0400)]
We need build-essential because of gihub.com/msteinert/pam which
apparently uses cgo...

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoPermissions fix.
Ward Vandewege [Wed, 5 May 2021 15:26:22 +0000 (11:26 -0400)]
Permissions fix.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoBump the version of the debian buster image we base our images on.
Ward Vandewege [Wed, 5 May 2021 15:19:25 +0000 (11:19 -0400)]
Bump the version of the debian buster image we base our images on.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoBump go to 1.16.3.
Ward Vandewege [Wed, 5 May 2021 15:16:16 +0000 (11:16 -0400)]
Bump go to 1.16.3.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoCostanalyzer has been moved to the arvados codebase, accessible via
Ward Vandewege [Thu, 18 Mar 2021 01:23:32 +0000 (21:23 -0400)]
Costanalyzer has been moved to the arvados codebase, accessible via
arvados-client.

refs #16950

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoWhen a package can't be published, print which distribution the error is
Ward Vandewege [Thu, 25 Feb 2021 18:22:42 +0000 (13:22 -0500)]
When a package can't be published, print which distribution the error is
with.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoBugfix for the `testing_to_stable_publish_packages.sh` script to reflect
Ward Vandewege [Thu, 25 Feb 2021 18:13:00 +0000 (13:13 -0500)]
Bugfix for the `testing_to_stable_publish_packages.sh` script to reflect
our switch to aptly.

Some refactoring to remove exitcode nonsense.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agorun_upload_packages.py: ignore errors writing timestamp files, but print
Ward Vandewege [Fri, 29 Jan 2021 15:41:55 +0000 (10:41 -0500)]
run_upload_packages.py: ignore errors writing timestamp files, but print
out a warning. We run this script from emphemeral machines now, so the
timestamp mechanism is not used.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agorun_upload_packages.py: give createrepo a cache directory for its
Ward Vandewege [Fri, 29 Jan 2021 15:26:15 +0000 (10:26 -0500)]
run_upload_packages.py: give createrepo a cache directory for its
package checksums, and tell it to use a simple file size/mtime
comparison instead of recalculating the hashes, when adding a package to
the repo.

This should speed up rpm package publishing.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agorun_upload_packages.py should make sure the directory it tries to put a
Ward Vandewege [Thu, 28 Jan 2021 22:48:12 +0000 (17:48 -0500)]
run_upload_packages.py should make sure the directory it tries to put a
timestamp file in exists (bugfix).

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agorun_upload_packages.py should make sure the directory it tries to put a
Ward Vandewege [Thu, 28 Jan 2021 22:30:55 +0000 (17:30 -0500)]
run_upload_packages.py should make sure the directory it tries to put a
timestamp file in exists.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix version number regex to allow 4 number positions & values over 9
Peter Amstutz [Thu, 28 Jan 2021 16:51:43 +0000 (11:51 -0500)]
Fix version number regex to allow 4 number positions & values over 9

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '16916-move-wb2-tests-to-jenkins-satellites'
Ward Vandewege [Fri, 8 Jan 2021 20:06:49 +0000 (15:06 -0500)]
Merge branch '16916-move-wb2-tests-to-jenkins-satellites'

closes #16916

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16916: add packer files for WB2 test image 16916-move-wb2-tests-to-jenkins-satellites
Ward Vandewege [Thu, 7 Jan 2021 22:47:42 +0000 (17:47 -0500)]
16916: add packer files for WB2 test image

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoBugfixes for packaging-move-dev-to-attic.py
Ward Vandewege [Mon, 4 Jan 2021 21:04:47 +0000 (16:04 -0500)]
Bugfixes for packaging-move-dev-to-attic.py

refs #17219

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17219-refactor-attic-scripts'
Ward Vandewege [Mon, 4 Jan 2021 18:55:52 +0000 (13:55 -0500)]
Merge branch '17219-refactor-attic-scripts'

closes #17219

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoAddress review comments. 17219-refactor-attic-scripts
Ward Vandewege [Mon, 4 Jan 2021 18:55:21 +0000 (13:55 -0500)]
Address review comments.

closes #17219

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agopackaging-move-dev-to-attic.py: remove hardcoded list of supported
Ward Vandewege [Mon, 4 Jan 2021 18:27:21 +0000 (13:27 -0500)]
packaging-move-dev-to-attic.py: remove hardcoded list of supported
distributions, instead validate the distribution name by pattern.

refs #17219

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoAdd packaging-move-dev-to-attic.py, which now supports aptly. This is a
Ward Vandewege [Mon, 4 Jan 2021 18:11:33 +0000 (13:11 -0500)]
Add packaging-move-dev-to-attic.py, which now supports aptly. This is a
modified version of an old internal script that supported freight.

refs #17219

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoIterate over packages one by one in run_upload_packages.py
Ward Vandewege [Wed, 23 Dec 2020 03:05:52 +0000 (22:05 -0500)]
Iterate over packages one by one in run_upload_packages.py

refs #17219

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMake the run_upload_packages.py script more verbose.
Ward Vandewege [Wed, 23 Dec 2020 02:27:43 +0000 (21:27 -0500)]
Make the run_upload_packages.py script more verbose.

refs #17219

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoDo not try to add packages to the repository that are already in it.
Ward Vandewege [Wed, 23 Dec 2020 01:45:43 +0000 (20:45 -0500)]
Do not try to add packages to the repository that are already in it.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17218-repository-update'
Ward Vandewege [Tue, 22 Dec 2020 22:09:18 +0000 (17:09 -0500)]
Merge branch '17218-repository-update'

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoWe've made a change in the structure of our apt repositories. 17218-repository-update
Ward Vandewege [Tue, 22 Dec 2020 19:56:34 +0000 (14:56 -0500)]
We've made a change in the structure of our apt repositories.

refs #17218

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMake it possible to deploy a single node.
Ward Vandewege [Wed, 16 Dec 2020 19:19:50 +0000 (14:19 -0500)]
Make it possible to deploy a single node.

refs #16925

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoDo not run arv-keepdocker on every shell node, just on the main shell
Ward Vandewege [Wed, 16 Dec 2020 19:04:40 +0000 (14:04 -0500)]
Do not run arv-keepdocker on every shell node, just on the main shell
node.

refs #16925

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agorun-deploy-salt.sh: make sure to have DEBIAN_FRONTEND=noninteractive set
Ward Vandewege [Wed, 16 Dec 2020 18:50:09 +0000 (13:50 -0500)]
run-deploy-salt.sh: make sure to have DEBIAN_FRONTEND=noninteractive set
when running apt-get -y upgrade. This caused a hang on install/upgrade
of the iptables-persistent package.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoNo need to sudo to run salt.
Ward Vandewege [Wed, 16 Dec 2020 14:54:33 +0000 (09:54 -0500)]
No need to sudo to run salt.

refs #16925

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMake sure to have DEBIAN_FRONTEND=noninteractive set when running
Ward Vandewege [Tue, 15 Dec 2020 15:33:24 +0000 (10:33 -0500)]
Make sure to have DEBIAN_FRONTEND=noninteractive set when running
apt-get -y upgrade. This caused a hang on install/upgrade of the
iptables-persistent package.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17013-upload-wheels-to-pypi' into master
Ward Vandewege [Mon, 23 Nov 2020 16:20:52 +0000 (11:20 -0500)]
Merge branch '17013-upload-wheels-to-pypi' into master

closes #17013

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17013: also upload wheels to PyPI. To make the wheels universal, we 17013-upload-wheels-to-pypi
Ward Vandewege [Mon, 23 Nov 2020 15:27:42 +0000 (10:27 -0500)]
17013: also upload wheels to PyPI. To make the wheels universal, we
       still need a change to setup.cfg in each of our Python
       component directories.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>