Peter Amstutz [Fri, 28 Aug 2015 14:40:06 +0000 (10:40 -0400)]
Preinstall forked version of libcloud, because nodemanager "pip install" won't
pick it up by default. refs #6507
Ward Vandewege [Fri, 28 Aug 2015 13:51:43 +0000 (09:51 -0400)]
run-build-packages-all-targets.sh should abort of WORKSPACE is not set.
No issue #
Ward Vandewege [Thu, 27 Aug 2015 18:51:29 +0000 (14:51 -0400)]
Update/clarify package assumptions for api server and workbench.
No issue #
Tom Clegg [Wed, 26 Aug 2015 15:29:32 +0000 (11:29 -0400)]
Merge branch '6781-arvados-login-sync' refs #6781
Ward Vandewege [Wed, 19 Aug 2015 18:49:00 +0000 (14:49 -0400)]
Stop building broken debs for our gems. We don't use them and they have
never worked. We can reintroduce a working version later.
refs #6781
Tom Clegg [Thu, 13 Aug 2015 15:02:51 +0000 (11:02 -0400)]
6781: Add arvados-login-sync.
Nico Cesar [Mon, 24 Aug 2015 21:35:20 +0000 (17:35 -0400)]
7041: pyvcf was missing from backports in trusty
refs #7041
Brett Smith [Thu, 20 Aug 2015 15:20:54 +0000 (11:20 -0400)]
6940: Use nginx as WWW_OWNER on Red Hat distributions.
We recommend deploying with Nginx, and this script interacts with
Nginx. The default Nginx configuration that Phusion ships runs as the
nginx user on these distros. Use that as the WWW_OWNER.
Ideally we want to move to a more sophisticated place where we
introspect the Web server configuration and use the same server it
does. But for now, this is an improvement.
No issue #, but discovered when working on #6940. I cleared this with
Ward.
Tom Clegg [Fri, 14 Aug 2015 21:32:58 +0000 (17:32 -0400)]
Merge branch '6638-backport-deps' refs #6638
Ward Vandewege [Fri, 14 Aug 2015 19:22:07 +0000 (15:22 -0400)]
Make sure to also deploy switchyard.
No issue #
Tom Clegg [Fri, 14 Aug 2015 18:37:12 +0000 (14:37 -0400)]
6638: Use (git clone; git checkout) -- git clone --branch doesn't work with commit hashes.
Ward Vandewege [Fri, 14 Aug 2015 00:38:53 +0000 (20:38 -0400)]
Performance optimization for the run_upload_packages.py script: only
rebuild the freight cache for the distribution that is being updated.
No issue #
Tom Clegg [Thu, 13 Aug 2015 21:06:29 +0000 (17:06 -0400)]
6934: Add sdk/pam to run_upload_packages.py refs #6934
Tom Clegg [Wed, 12 Aug 2015 20:10:17 +0000 (16:10 -0400)]
6638: Fix rvm usage error. Straighten out "need new gem" and "need to build new package from gem" logic.
Tom Clegg [Wed, 12 Aug 2015 20:09:08 +0000 (16:09 -0400)]
6638: Add postgres libraries to build containers, so bundler can build the pg gem.
Tom Clegg [Wed, 12 Aug 2015 16:01:29 +0000 (12:01 -0400)]
6638: Build current version of arvados-src, not github
Tom Clegg [Wed, 12 Aug 2015 16:05:59 +0000 (12:05 -0400)]
6638: Read package dependencies from fpm-info.sh in source tree.
Tom Clegg [Wed, 12 Aug 2015 13:31:32 +0000 (09:31 -0400)]
6638: Dry up debug printfs.
Tom Clegg [Thu, 13 Aug 2015 03:06:15 +0000 (23:06 -0400)]
Merge branch '6934-pam' refs #6934
Tom Clegg [Thu, 13 Aug 2015 02:36:13 +0000 (22:36 -0400)]
6934: Build libpam-arvados only for targets that have libpam-python.
Tom Clegg [Wed, 12 Aug 2015 20:19:06 +0000 (16:19 -0400)]
6934: Add libpam-python dependency on all targets.
Tom Clegg [Wed, 12 Aug 2015 17:57:06 +0000 (13:57 -0400)]
6934: Merge branch 'master' into 6934-pam
Tom Clegg [Mon, 10 Aug 2015 14:41:00 +0000 (10:41 -0400)]
6934: Package libpam-arvados using Python recipe. Add sdk/pam to run-tests.
Ward Vandewege [Fri, 7 Aug 2015 14:56:45 +0000 (10:56 -0400)]
Make sure that we build arvados-src package with the correct version of
the git tree, not latest.
No issue #
Ward Vandewege [Wed, 5 Aug 2015 02:00:51 +0000 (22:00 -0400)]
Fix logic bug in run-deploy: deploy api server first before gathering
the git revision from it to make sure the correct arvados/jobs docker
image is installed.
No issue #
Tom Clegg [Thu, 30 Jul 2015 19:38:03 +0000 (15:38 -0400)]
Merge branch '6592-test-perl' refs #6592
Tom Clegg [Thu, 30 Jul 2015 15:44:44 +0000 (11:44 -0400)]
6592: Build and install the Perl SDK so sdk/cli tests can use it.
Brett Smith [Mon, 27 Jul 2015 14:50:32 +0000 (10:50 -0400)]
Rebuild Python packages before building their distro packages.
Without this, built scripts lay around between builds, with shebang
lines for the first distro they were built for. This can lead to
problems like Debian packages getting shebang lines for Red Hat
systems. See, e.g.,
<https://ci.curoverse.com/job/deploy-to-staging/1250/console>.
No issue #.
Peter Amstutz [Mon, 27 Jul 2015 13:19:46 +0000 (09:19 -0400)]
Fix parameter ordering for minor change in cwltool command line semantics. no issue #
Brett Smith [Mon, 27 Jul 2015 12:55:43 +0000 (08:55 -0400)]
Merge branch '6691-trusty-packages-wip'
Refs #6691. Closes #6742.
Brett Smith [Thu, 23 Jul 2015 21:10:54 +0000 (17:10 -0400)]
6691: Build and ship packages for Ubuntu 14.04.
Brett Smith [Fri, 24 Jul 2015 16:08:09 +0000 (12:08 -0400)]
Merge branch '6682-use-docker-build-cmd-wip'
Refs #6682. Closes #6741.
Brett Smith [Thu, 23 Jul 2015 02:21:59 +0000 (22:21 -0400)]
6682: Add script to orchestrate builds for multiple targets.
Now that we can build packages for multiple targets, it's nice to have
a one-shot script to build them all. This does that.
Brett Smith [Thu, 23 Jul 2015 20:59:11 +0000 (16:59 -0400)]
6497: Bugfix distro names for DebianPackageSuites.
freight is expecting a codename like wheezy or jessie, but we've been
using our internal distro+version names there. This fixes that.
Nico gave a thumbs up to this patch. Refs #6497.
Nico Cesar [Thu, 23 Jul 2015 16:55:13 +0000 (12:55 -0400)]
one of the scopes wasnt working
closes #6700
Nico Cesar [Thu, 23 Jul 2015 14:53:41 +0000 (10:53 -0400)]
scopes in usage() are wrong.
refs #6700
Brett Smith [Thu, 23 Jul 2015 14:07:54 +0000 (10:07 -0400)]
6750: arvados-api-server-upgrade only installs omniauth.rb if it exists.
New installs configure the SSO server via application.yml. Refs #6750.
Brett Smith [Wed, 22 Jul 2015 20:38:33 +0000 (16:38 -0400)]
Merge branch '6497-separate-package-upload-wip'
Closes #6497, #6533, #6498.
Brett Smith [Wed, 22 Jul 2015 20:38:23 +0000 (16:38 -0400)]
6497: Separate package build and uploader scripts.
This lets us avoid redundant uploads as we build for multiple targets,
and lets us better customize the upload process per target.
run-build-packages had code to avoid doing an upload when there was no
new package to build. run_upload_packages uses a timestamp file to
achieve a similar result.
Other changes to run-build-packages to support this:
* Better detection of when Python and Ruby packages have already been
built.
* Refactor Gem building into a function to fix various bugs (wrong
version checked for existing arvados-cli; distro packages not
installed in the proper directory).
* Refactor git log querying to reduce code duplication.
Nico Cesar [Tue, 21 Jul 2015 14:10:24 +0000 (10:10 -0400)]
added exit if git pull fails
no issue #
Brett Smith [Mon, 20 Jul 2015 18:36:46 +0000 (14:36 -0400)]
6656: python-arvados-python-client Recommends: git.
Because arv-copy uses it. Refs #6656.
Ward Vandewege [Thu, 16 Jul 2015 22:02:30 +0000 (18:02 -0400)]
run-deploy.sh improvements: remove the need for a .ssh/config entry for
the shortname of each API server. Thanks for the suggestion Nico.
No issue #
Ward Vandewege [Thu, 16 Jul 2015 03:14:23 +0000 (23:14 -0400)]
Make sure jessie packages end up in the jessie repository.
No issue #
Ward Vandewege [Thu, 16 Jul 2015 02:59:28 +0000 (22:59 -0400)]
We need to force-tag the arvados/jobs docker image.
No issue #
Ward Vandewege [Thu, 16 Jul 2015 02:53:50 +0000 (22:53 -0400)]
freight is already smart enough to do atomic updates.
No issue #
Ward Vandewege [Thu, 16 Jul 2015 01:55:21 +0000 (21:55 -0400)]
Add dockerfiles/build-all-build-containers.sh script
refs #6504
Peter Amstutz [Wed, 15 Jul 2015 17:29:00 +0000 (13:29 -0400)]
Add datamanager tests. refs #3408
Ward Vandewege [Wed, 15 Jul 2015 17:14:34 +0000 (13:14 -0400)]
Build python3-websocket-client packages, which is now a dependency for
python3-docker-py, which is a dependency for arvados-docker-cleaner.
No issue #
Ward Vandewege [Tue, 14 Jul 2015 01:02:07 +0000 (21:02 -0400)]
Make sure we have a conference/environments/production.rb file so that
we generate the workbench assets correctly before packaging.
refs #6362
Ward Vandewege [Mon, 13 Jul 2015 15:39:25 +0000 (11:39 -0400)]
run_deploy.sh improvements:
* Make sure that group_class is included when creating the public docker
images project.
* Document the required permissions for the arvados token needed to run
run_deploy.sh
closes #6586
Ward Vandewege [Fri, 10 Jul 2015 18:07:39 +0000 (14:07 -0400)]
Add debian8 as a build target.
No issue #
Ward Vandewege [Fri, 10 Jul 2015 02:01:36 +0000 (22:01 -0400)]
Build arvados/jobs docker image separately from the others.
refs #6528
Ward Vandewege [Wed, 8 Jul 2015 20:31:35 +0000 (16:31 -0400)]
Add dependency on libpam-python to the libarvados-pam package.
refs #2256
Ward Vandewege [Wed, 8 Jul 2015 18:56:07 +0000 (14:56 -0400)]
Add libpam-arvados package build to run-build-packages.
refs #6384
Brett Smith [Wed, 8 Jul 2015 14:09:25 +0000 (10:09 -0400)]
Merge branch '6389-add-dockerfiles'
Closes #6389, #6456.
Brett Smith [Mon, 29 Jun 2015 20:59:24 +0000 (16:59 -0400)]
6388: Add Dockerfile to build packages on CentOS 6.
Brett Smith [Mon, 29 Jun 2015 20:56:57 +0000 (16:56 -0400)]
6389: Add Makefile to support Docker builds.
Brett Smith [Mon, 29 Jun 2015 20:56:16 +0000 (16:56 -0400)]
6389: Dockerfiles include the CMD to build packages.
Ward Vandewege [Fri, 26 Jun 2015 21:07:32 +0000 (17:07 -0400)]
Add dockerfiles for Debian7 and Ubuntu1204
refs #6389
Ward Vandewege [Wed, 8 Jul 2015 13:59:54 +0000 (09:59 -0400)]
Fix up git log statements:
* remove duplication between -n1 and --max-count=1
* make sure every git log statement has --first-parent
refs #6096
Ward Vandewege [Wed, 8 Jul 2015 02:05:01 +0000 (22:05 -0400)]
Small fixes for run-deploy.sh
refs #6096
Ward Vandewege [Thu, 11 Jun 2015 01:30:45 +0000 (21:30 -0400)]
Make run-deploy.rb install the correct version of the arvados/jobs
docker image on deploy.
refs #6096
Brett Smith [Mon, 6 Jul 2015 17:29:44 +0000 (13:29 -0400)]
Merge branch '6388-package-scripts-redhat-support-wip'
Refs #6388. Closes #6480.
Brett Smith [Mon, 6 Jul 2015 14:26:44 +0000 (10:26 -0400)]
6388: Extend API server upgrade script to do initial database setup.
This will let administrators duplicating a lot of logic that already
exists in the script.
Brett Smith [Sun, 5 Jul 2015 22:11:23 +0000 (18:11 -0400)]
6388: Rails package upgrade scripts support Red Hat systems.
* Red Hat doesn't use the www-data account name.
* When nginx is installed from a Software Collection, it may have a
name like `nginx16`. Detect and use this.
Brett Smith [Sun, 5 Jul 2015 15:55:25 +0000 (11:55 -0400)]
Bugfix config:check error detection in Rails upgrade scripts.
The previous code was capturing the exit status of echo, which isn't
very interesting. No issue #.
Tom Clegg [Tue, 30 Jun 2015 18:53:47 +0000 (14:53 -0400)]
6377: Don't try to uninstall arvados* gems if they are not installed.
Fixes run-tests failure when no Ruby SDK is installed on the test host
and tmpdirs are new/empty. refs #6377
Tom Clegg [Tue, 30 Jun 2015 18:50:50 +0000 (14:50 -0400)]
6377: Create user-specified tmpdirs if necessary. refs #6377
Ward Vandewege [Mon, 29 Jun 2015 18:18:08 +0000 (14:18 -0400)]
Add rsync to the repo update command in run-build-packages.
No issue #
Ward Vandewege [Fri, 26 Jun 2015 21:09:03 +0000 (17:09 -0400)]
Make sure to also build a backported package for python-llfuse. We need
it for writeable fuse.
refs #3198
Ward Vandewege [Fri, 26 Jun 2015 19:41:47 +0000 (15:41 -0400)]
Update comment references from 'debs' to 'packages'.
refs #6389
Ward Vandewege [Fri, 26 Jun 2015 19:40:26 +0000 (15:40 -0400)]
Drop built packages into packages/$TARGET instead of the old 'debs' directory.
Also, fix the 'find' test so that built packages are actually added to
the repository after upload.
refs #6389
Ward Vandewege [Fri, 26 Jun 2015 19:04:35 +0000 (15:04 -0400)]
Add ubuntu1204 target to run-build-packages.sh
refs #6389
Ward Vandewege [Fri, 26 Jun 2015 18:55:22 +0000 (14:55 -0400)]
Regex match fixes in the character class match for $FPM_RESULTS:
a) we want to match a literal dot, not any character
b) bash 4.2 does not support escaping a dash to include it in the
character class, move the dash to the last character listed in the class
refs #6389
Brett Smith [Fri, 26 Jun 2015 13:12:22 +0000 (09:12 -0400)]
6388: CentOS needs backports for python-daemon and its dependencies.
Refs #6388.
Peter Amstutz [Fri, 26 Jun 2015 03:39:43 +0000 (23:39 -0400)]
Additionally build and upload cwl-runner package. no issue #
Nico Cesar [Thu, 25 Jun 2015 19:53:46 +0000 (15:53 -0400)]
login patch because DockerHub is not friendly
no issue #
Brett Smith [Thu, 25 Jun 2015 19:33:01 +0000 (15:33 -0400)]
Merge branch '6388-python-backports-wip'
Refs #6388. Closes #6425.
Brett Smith [Thu, 25 Jun 2015 19:22:14 +0000 (15:22 -0400)]
6388: Improve portability of Python backports in run-build-packages.
* Prefix package names with the same prefix used by the interpreter
package, to avoid conflicts with system packages when we're using
something like Software Collections.
* Always specify a dependency on that package.
* Exclude more test paths from packages to avoid conflicts.
* Do a little name munging to make sure that output package names
match dependencies generated by fpm.
Brett Smith [Thu, 25 Jun 2015 19:19:22 +0000 (15:19 -0400)]
6388: CentOS 6 needs Python daemon and llfuse backported.
Brett Smith [Thu, 25 Jun 2015 19:18:58 +0000 (15:18 -0400)]
6388: Build specific backport versions for Python OAuth dependencies.
Brett Smith [Thu, 25 Jun 2015 19:16:57 +0000 (15:16 -0400)]
6388: Build PyCURL with NSS on CentOS 6.
Brett Smith [Thu, 25 Jun 2015 15:54:35 +0000 (11:54 -0400)]
6388: Remove old --apt-server switch from run-build-packages.
Refs #6388.
Brett Smith [Thu, 25 Jun 2015 15:41:01 +0000 (11:41 -0400)]
Merge branch '6388-centos-packages-wip'
Refs #6388. Closes #6411.
Brett Smith [Wed, 24 Jun 2015 21:47:30 +0000 (17:47 -0400)]
6388: run-build-packages introspects more Python environment.
* Look for easy_install under a few different names.
* On CentOS, query the package that provides each Python, and list
that as a dependency.
Brett Smith [Wed, 24 Jun 2015 21:06:03 +0000 (17:06 -0400)]
6388: Fix Arvados Perl SDK package build.
Brett Smith [Wed, 24 Jun 2015 20:44:52 +0000 (16:44 -0400)]
6388: Backport pycurl package everywhere.
We need the very latest version per #6073.
Brett Smith [Wed, 24 Jun 2015 17:54:02 +0000 (13:54 -0400)]
6388: Remove version munging from Python package builds.
This code was added to accommodate version numbers that did *not*
comply with PEP 440. We've since fixed those, and they only include
alphanumerics and dots, so we don't need to munge them.
Brett Smith [Wed, 24 Jun 2015 17:51:09 +0000 (13:51 -0400)]
6388: run-build-packages records python2.7 dependency where needed.
Brett Smith [Wed, 24 Jun 2015 17:24:20 +0000 (13:24 -0400)]
6388: Add distro targets to run-build-packages.
This will provide us a single mechanism to make sure that packages are
built to run on a particular distribution: not just in terms of FPM's
output format, but also repository location, backports required, etc.
Brett Smith [Tue, 23 Jun 2015 16:10:54 +0000 (12:10 -0400)]
6388: Remove Debianisms from run-build-packages.
Make sure function and variable names don't refer to Debian when they
can also be used for Red Hat.
Ward Vandewege [Sat, 13 Jun 2015 00:45:38 +0000 (20:45 -0400)]
Fix asset generation in the workbench package. Make sure the build step will error out if asset generation fails for whatever reason.
No issue #
Ward Vandewege [Fri, 12 Jun 2015 18:49:30 +0000 (14:49 -0400)]
Do not build and upload gems that already exist on rubygems.
No issue #
Ward Vandewege [Fri, 12 Jun 2015 15:07:44 +0000 (11:07 -0400)]
Add very rudimentary support for pushing the rpm packages into a yum
repository.
No issue #
Ward Vandewege [Fri, 12 Jun 2015 14:10:11 +0000 (10:10 -0400)]
Add rpm generation support to run-build-packages.sh
No issue #
Ward Vandewege [Thu, 11 Jun 2015 01:11:35 +0000 (21:11 -0400)]
Store the full git hash of the checkout in the arvados-src package, in a file called git-commit.version
refs #6096
Peter Amstutz [Wed, 10 Jun 2015 18:43:59 +0000 (14:43 -0400)]
CWL script won't fail when there are no documentation changes to commit. no issue #
Peter Amstutz [Wed, 10 Jun 2015 14:49:18 +0000 (10:49 -0400)]
Set default identity for cwl repo pushes. no issue #
Peter Amstutz [Wed, 10 Jun 2015 14:28:01 +0000 (10:28 -0400)]
Since jenkins reuses working directories, handle updating existing git repo of
common-workflow-language site that will be automatically pushed. no issue #
Peter Amstutz [Tue, 9 Jun 2015 20:40:47 +0000 (16:40 -0400)]
Automatically build and upload documentation site. no issue #
projects / arvados.git / log