arvados.git
8 years agoAdd --output-tags to arvados-cwl-runner and tests
Jiayong Li [Tue, 8 Nov 2016 19:15:42 +0000 (14:15 -0500)]
Add --output-tags to arvados-cwl-runner and tests

8 years agoMerge branch '10461-cwl-keepdocker' refs #10461
Peter Amstutz [Fri, 4 Nov 2016 13:41:55 +0000 (09:41 -0400)]
Merge branch '10461-cwl-keepdocker' refs #10461

8 years agoMerge branch '9849-cwl-efficient-log-scan' refs #9849
Peter Amstutz [Fri, 4 Nov 2016 13:41:22 +0000 (09:41 -0400)]
Merge branch '9849-cwl-efficient-log-scan' refs #9849

8 years agoMerge branch '9849-cwl-docker-lookup' refs #9849
Peter Amstutz [Fri, 4 Nov 2016 13:41:08 +0000 (09:41 -0400)]
Merge branch '9849-cwl-docker-lookup' refs #9849

8 years ago10461: Remove arv-keepdocker pull from test_with_arvbox.sh. Only install
Peter Amstutz [Fri, 4 Nov 2016 13:15:18 +0000 (09:15 -0400)]
10461: Remove arv-keepdocker pull from test_with_arvbox.sh.  Only install
arvados-cwl-runner in 'dev' configuration.

8 years ago10461: Add test for fetching arvados_jobs image.
Peter Amstutz [Fri, 4 Nov 2016 13:12:51 +0000 (09:12 -0400)]
10461: Add test for fetching arvados_jobs image.

8 years agoCheck exit code before rethrowing error from catching SystemExit from
Peter Amstutz [Thu, 3 Nov 2016 22:13:29 +0000 (18:13 -0400)]
Check exit code before rethrowing error from catching SystemExit from
keepdocker.main.  no issue #

8 years agoMerge branch '10224-efficient-event-poll-startup' refs #10224
Tom Clegg [Thu, 3 Nov 2016 19:45:13 +0000 (15:45 -0400)]
Merge branch '10224-efficient-event-poll-startup' refs #10224

8 years ago10224: Workaround Firefox failure to parse localized dates in metric locales.
Tom Clegg [Thu, 3 Nov 2016 18:20:44 +0000 (14:20 -0400)]
10224: Workaround Firefox failure to parse localized dates in metric locales.

8 years ago10224: Update tests to expect event_type="delete".
Tom Clegg [Thu, 3 Nov 2016 14:10:14 +0000 (10:10 -0400)]
10224: Update tests to expect event_type="delete".

8 years ago10224: Choose a recent-event threshold without querying the entire event history.
Tom Clegg [Tue, 1 Nov 2016 15:28:49 +0000 (11:28 -0400)]
10224: Choose a recent-event threshold without querying the entire event history.

8 years ago10224: Change event_type to "delete" to match HTTP verb and Python SDK code.
Tom Clegg [Tue, 1 Nov 2016 15:29:42 +0000 (11:29 -0400)]
10224: Change event_type to "delete" to match HTTP verb and Python SDK code.

8 years ago9849: Provide stub log to be parsed.
Peter Amstutz [Thu, 3 Nov 2016 18:22:04 +0000 (14:22 -0400)]
9849: Provide stub log to be parsed.

8 years agoUse --net=host when running jobs inside arvbox to work around Docker-in-Docker
Peter Amstutz [Tue, 1 Nov 2016 17:09:46 +0000 (13:09 -0400)]
Use --net=host when running jobs inside arvbox to work around Docker-in-Docker
networking issues. no issue #

8 years ago9849: Use one regex instead of 3 when scanning log.
Peter Amstutz [Fri, 28 Oct 2016 02:48:47 +0000 (22:48 -0400)]
9849: Use one regex instead of 3 when scanning log.

8 years ago9849: Cache docker lookup.
Peter Amstutz [Fri, 28 Oct 2016 01:54:14 +0000 (21:54 -0400)]
9849: Cache docker lookup.

8 years agoMerge branch '10448-cwl-runner-disable-reuse' closes #10448
Peter Amstutz [Thu, 3 Nov 2016 17:28:04 +0000 (13:28 -0400)]
Merge branch '10448-cwl-runner-disable-reuse' closes #10448

8 years ago10448: Tests include enable_reuse flag and that --disable-reuse is passed to
Peter Amstutz [Thu, 3 Nov 2016 14:29:07 +0000 (10:29 -0400)]
10448: Tests include enable_reuse flag and that --disable-reuse is passed to
inner cwl-runner job or container.

8 years ago10448: Pass through enable_reuse flag to runner.
Peter Amstutz [Wed, 2 Nov 2016 21:21:56 +0000 (17:21 -0400)]
10448: Pass through enable_reuse flag to runner.

8 years agoFix git log to get last log for sdk/cwl or sdk/python only, not repository head refs...
Peter Amstutz [Thu, 3 Nov 2016 16:37:35 +0000 (12:37 -0400)]
Fix git log to get last log for sdk/cwl or sdk/python only, not repository head refs #10438

8 years agorefs #10231
radhika [Wed, 2 Nov 2016 23:46:08 +0000 (19:46 -0400)]
refs #10231
Merge branch '10231-keep-cache-runtime-constraints'

8 years ago10231: validate_runtime_constraints should occur before set_container filter.
radhika [Wed, 2 Nov 2016 22:50:46 +0000 (18:50 -0400)]
10231: validate_runtime_constraints should occur before set_container filter.

8 years agocloses #10231
radhika [Wed, 2 Nov 2016 21:19:12 +0000 (17:19 -0400)]
closes #10231
Merge branch '10231-keep-cache-runtime-constraints'

8 years agorefs #10028
radhika [Wed, 2 Nov 2016 21:15:02 +0000 (17:15 -0400)]
refs #10028
Merge branch '10028-api-contents-perf'

8 years ago10028: when an "is_a" filter is used while fetching contents, do not search in all...
radhika [Wed, 2 Nov 2016 18:42:00 +0000 (14:42 -0400)]
10028: when an "is_a" filter is used while fetching contents, do not search in all "other" types.

8 years agoMerge branch '10343-dockercleaner-config' refs #10343
Tom Clegg [Mon, 31 Oct 2016 21:38:50 +0000 (17:38 -0400)]
Merge branch '10343-dockercleaner-config' refs #10343

8 years ago10343: Merge branch 'master' into 10343-dockercleaner-config
Tom Clegg [Mon, 31 Oct 2016 20:08:01 +0000 (16:08 -0400)]
10343: Merge branch 'master' into 10343-dockercleaner-config

8 years ago10343: Add comment about unwieldy ExecStart command.
Tom Clegg [Mon, 31 Oct 2016 19:23:53 +0000 (15:23 -0400)]
10343: Add comment about unwieldy ExecStart command.

8 years agoCompute correct arvados-cwl-runner version, refs #10194
Peter Amstutz [Mon, 31 Oct 2016 18:06:12 +0000 (14:06 -0400)]
Compute correct arvados-cwl-runner version, refs #10194

8 years ago10231: add keep_cache_ram to doc
radhika [Mon, 31 Oct 2016 17:47:20 +0000 (13:47 -0400)]
10231: add keep_cache_ram to doc

8 years agoMerge branch 'master' into 10231-keep-cache-runtime-constraints
radhika [Mon, 31 Oct 2016 16:04:51 +0000 (12:04 -0400)]
Merge branch 'master' into 10231-keep-cache-runtime-constraints

8 years agotag arvados/arvbox-base with latest so that FROM arvados/arvbox-base:latest works...
Peter Amstutz [Mon, 31 Oct 2016 16:00:21 +0000 (12:00 -0400)]
tag arvados/arvbox-base with latest so that FROM arvados/arvbox-base:latest works. refs #10025

8 years agoMerge branch '10315-new-arv-put-performance'
Lucas Di Pentima [Mon, 31 Oct 2016 13:58:38 +0000 (10:58 -0300)]
Merge branch '10315-new-arv-put-performance'
Closes #10315

8 years ago10343: Update postinst script to run on Red Hat systems, and accommodate Python33...
Tom Clegg [Fri, 28 Oct 2016 22:22:24 +0000 (18:22 -0400)]
10343: Update postinst script to run on Red Hat systems, and accommodate Python33 Software Collections.

8 years ago10315: Merge branch 'master' into 10315-new-arv-put-performance
Lucas Di Pentima [Fri, 28 Oct 2016 18:35:19 +0000 (15:35 -0300)]
10315: Merge branch 'master' into 10315-new-arv-put-performance

8 years ago10315: Moved conditional on pending write size checking
Lucas Di Pentima [Fri, 28 Oct 2016 18:33:03 +0000 (15:33 -0300)]
10315: Moved conditional on pending write size checking

8 years ago10343: Improve log message.
Tom Clegg [Fri, 28 Oct 2016 18:06:34 +0000 (14:06 -0400)]
10343: Improve log message.

8 years agoExplicitly list all root-level directories in this repository in the
Ward Vandewege [Fri, 28 Oct 2016 17:26:48 +0000 (13:26 -0400)]
Explicitly list all root-level directories in this repository in the
COPYING file.

No issue #

8 years ago10315: Update pending write size count just before packing small blocks to see if...
Lucas Di Pentima [Fri, 28 Oct 2016 14:56:20 +0000 (11:56 -0300)]
10315: Update pending write size count just before packing small blocks to see if there really are enough to fill a full one.

8 years ago10315: Added back the BlockManager's put threads lazy start, but with a specific...
Lucas Di Pentima [Fri, 28 Oct 2016 14:27:22 +0000 (11:27 -0300)]
10315: Added back the BlockManager's put threads lazy start, but with a specific lock, so it can be called from a @synchronized method.

8 years ago10343: Update arvados-docker-cleaner install docs to use packaged systemd unit file.
Tom Clegg [Fri, 28 Oct 2016 06:57:47 +0000 (02:57 -0400)]
10343: Update arvados-docker-cleaner install docs to use packaged systemd unit file.

8 years ago10343: Allow running without config file, if -config arg is not given.
Tom Clegg [Thu, 27 Oct 2016 16:55:35 +0000 (12:55 -0400)]
10343: Allow running without config file, if -config arg is not given.

8 years agoPin the python-daemon backport for centos7, newer versions have
Ward Vandewege [Wed, 26 Oct 2016 18:58:11 +0000 (14:58 -0400)]
Pin the python-daemon backport for centos7, newer versions have
additional dependencies.

No issue #

8 years agoMerge branch '10008-check-token-exp-on-open' refs #10008
Tom Clegg [Wed, 26 Oct 2016 15:40:42 +0000 (11:40 -0400)]
Merge branch '10008-check-token-exp-on-open' refs #10008

8 years ago10008: Comment to explain known deficiencies.
Tom Clegg [Wed, 26 Oct 2016 15:13:10 +0000 (11:13 -0400)]
10008: Comment to explain known deficiencies.

8 years ago10315: Moved the BlockManager's put threads lazy loading to __init__() to avoid locki...
Lucas Di Pentima [Wed, 26 Oct 2016 15:01:55 +0000 (12:01 -0300)]
10315: Moved the BlockManager's put threads lazy loading to __init__() to avoid locking issues when calling commit_bufferblock() fropm within an already @synchronized method.

8 years ago10315: Brought back the new arv-put tests from 9701 branch.
Lucas Di Pentima [Tue, 25 Oct 2016 23:04:23 +0000 (20:04 -0300)]
10315: Brought back the new arv-put tests from 9701 branch.

8 years ago10315: Enhanced performance when writing lots of small files without flushing by...
Lucas Di Pentima [Tue, 25 Oct 2016 22:52:23 +0000 (19:52 -0300)]
10315: Enhanced performance when writing lots of small files without flushing by adding every closed file's size to a counter instead of scanning the pending bufferblock list every time.

8 years agocloses #10079
radhika [Tue, 25 Oct 2016 22:25:58 +0000 (18:25 -0400)]
closes #10079
Merge branch '10079-pipeline-display-perf'

8 years ago10079: show spinner when a child component details are being displayed using ajax.
radhika [Tue, 25 Oct 2016 16:35:55 +0000 (12:35 -0400)]
10079: show spinner when a child component details are being displayed using ajax.

8 years ago10079: add "parent" to work_unit to aid the child display
radhika [Sun, 23 Oct 2016 01:47:51 +0000 (21:47 -0400)]
10079: add "parent" to work_unit to aid the child display

8 years ago10079: fixed route so that @object is available
radhika [Fri, 21 Oct 2016 21:19:53 +0000 (17:19 -0400)]
10079: fixed route so that @object is available

8 years ago10079: checking log_link on children needs to be an integration test.
radhika [Wed, 19 Oct 2016 21:17:56 +0000 (17:17 -0400)]
10079: checking log_link on children needs to be an integration test.

8 years ago10079: show_child_component method needs anonymous access
radhika [Wed, 19 Oct 2016 20:19:23 +0000 (16:19 -0400)]
10079: show_child_component method needs anonymous access

8 years ago10079: ajax to display child component details; not working yet.
radhika [Tue, 18 Oct 2016 04:24:13 +0000 (00:24 -0400)]
10079: ajax to display child component details; not working yet.

8 years ago10079: Move Log and Output links into child details. This will make it possible
radhika [Thu, 13 Oct 2016 21:21:04 +0000 (17:21 -0400)]
10079: Move Log and Output links into child details. This will make it possible
to not need tt preload the jobs and outputs during the pipeline display. Instead,
they can be fetched when the child panel is expanded by the user.

8 years ago10231: update crunchrun to pass Container.RuntimeConstraints.KeepCacheRAM as file...
radhika [Tue, 25 Oct 2016 19:05:57 +0000 (15:05 -0400)]
10231: update crunchrun to pass Container.RuntimeConstraints.KeepCacheRAM as file-cache argument to arv mount.

8 years ago10291: Update --help message. refs #10291
Tom Clegg [Tue, 25 Oct 2016 19:05:10 +0000 (15:05 -0400)]
10291: Update --help message. refs #10291

8 years ago10231: update arvados-cwl-runner for keep_cache_ram runtime_constrant.
radhika [Tue, 25 Oct 2016 17:27:30 +0000 (13:27 -0400)]
10231: update arvados-cwl-runner for keep_cache_ram runtime_constrant.

8 years agoMerge branch '10172-crunch2-container-output' closes #10172
Peter Amstutz [Tue, 25 Oct 2016 17:00:23 +0000 (13:00 -0400)]
Merge branch '10172-crunch2-container-output' closes #10172

8 years ago10008: Comment to explain ancestor update.
Tom Clegg [Tue, 25 Oct 2016 15:39:17 +0000 (11:39 -0400)]
10008: Comment to explain ancestor update.

8 years ago10172: Add container.output check to TestFullRunWithAPI.
Peter Amstutz [Tue, 25 Oct 2016 15:08:43 +0000 (11:08 -0400)]
10172: Add container.output check to TestFullRunWithAPI.

8 years ago10172: Add skip_before_filter :find_object_by_uuid for current. Typo fixes.
Peter Amstutz [Tue, 25 Oct 2016 14:51:14 +0000 (10:51 -0400)]
10172: Add skip_before_filter :find_object_by_uuid for current.  Typo fixes.

8 years agoMerge branch '10194-jobs-image-versioning' refs #10194
Peter Amstutz [Tue, 25 Oct 2016 01:33:40 +0000 (21:33 -0400)]
Merge branch '10194-jobs-image-versioning' refs #10194

8 years ago10194: Improve versioning of arvados/jobs image.
Peter Amstutz [Fri, 21 Oct 2016 20:35:09 +0000 (16:35 -0400)]
10194: Improve versioning of arvados/jobs image.

arvados-cwl-runner sets version to max(cwl-runner, python-sdk).

build/run-build-docker-jobs-image.sh determines package versions corresponding
to current commit and supplies explicit package versions as Docker build arg.

8 years ago10231: add keep_cache_ram in runtime_constraints
radhika [Mon, 24 Oct 2016 21:22:22 +0000 (17:22 -0400)]
10231: add keep_cache_ram in runtime_constraints

8 years ago10315: Reduced the time between updates.
Lucas Di Pentima [Mon, 24 Oct 2016 14:34:37 +0000 (11:34 -0300)]
10315: Reduced the time between updates.

8 years ago10315: Allow small block packing passing flush=False.
Lucas Di Pentima [Mon, 24 Oct 2016 11:47:32 +0000 (08:47 -0300)]
10315: Allow small block packing passing flush=False.

8 years agoMerge branch '10232-non-exclusive-keys' refs #10232
Tom Clegg [Fri, 21 Oct 2016 20:08:45 +0000 (16:08 -0400)]
Merge branch '10232-non-exclusive-keys' refs #10232

8 years ago10008: Move test setup from __init__() to setUp()
Tom Clegg [Fri, 21 Oct 2016 19:27:34 +0000 (15:27 -0400)]
10008: Move test setup from __init__() to setUp()

8 years ago10008: Merge branch 'master' into 10008-flaky-token-test
Tom Clegg [Fri, 21 Oct 2016 19:19:51 +0000 (15:19 -0400)]
10008: Merge branch 'master' into 10008-flaky-token-test

8 years ago10008: Call checkupdate() on ancestors when opening a file.
Tom Clegg [Fri, 21 Oct 2016 19:16:24 +0000 (15:16 -0400)]
10008: Call checkupdate() on ancestors when opening a file.

8 years ago10232: Call getpwnam() and getgrnam() for every name instead of relying on Etc.to_enum().
Tom Clegg [Fri, 21 Oct 2016 15:18:32 +0000 (11:18 -0400)]
10232: Call getpwnam() and getgrnam() for every name instead of relying on Etc.to_enum().

8 years agoAdd requests to ubuntu1204 backport lists, refs #10194
Peter Amstutz [Fri, 21 Oct 2016 15:06:51 +0000 (11:06 -0400)]
Add requests to ubuntu1204 backport lists, refs #10194

8 years agoMerge branch '10194-cwl-version-skew' closes #10194
Peter Amstutz [Fri, 21 Oct 2016 14:15:56 +0000 (10:15 -0400)]
Merge branch '10194-cwl-version-skew' closes #10194

8 years agoMerge branch 'master' into 10194-cwl-version-skew
Peter Amstutz [Fri, 21 Oct 2016 14:15:10 +0000 (10:15 -0400)]
Merge branch 'master' into 10194-cwl-version-skew

Conflicts:
sdk/cwl/tests/test_job.py

8 years ago10194: Use SETUP_DIR for checking git version and writing _version.py. Add
Peter Amstutz [Fri, 21 Oct 2016 13:52:36 +0000 (09:52 -0400)]
10194: Use SETUP_DIR for checking git version and writing _version.py.  Add
comment about calling cwltool.docker.get_image(dockerRequirement, pull_image).

8 years ago10172: Fix container request test. Replace
Peter Amstutz [Fri, 21 Oct 2016 12:47:05 +0000 (08:47 -0400)]
10172: Fix container request test.  Replace
Thread.current[:api_client_authorization] with
current_api_client_authorization.

8 years ago10232: dont create user logic was wrong
Nico Cesar [Fri, 21 Oct 2016 11:38:37 +0000 (07:38 -0400)]
10232: dont create user logic was wrong

refs #10232

8 years ago10232: --dont-create-user flag and sanity check for non-existing users
Nico Cesar [Fri, 21 Oct 2016 11:10:22 +0000 (07:10 -0400)]
10232: --dont-create-user flag and sanity check for non-existing users

refs #10232

8 years agoMerge branch 'crunch-job_finds_newer_docker_hashes' of https://github.com/tmooney...
Tom Clegg [Fri, 21 Oct 2016 05:18:54 +0000 (01:18 -0400)]
Merge branch 'crunch-job_finds_newer_docker_hashes' of https://github.com/tmooney/arvados

No issue #

8 years agoCheck docker version to decide whether to use -f with docker tag, refs #10025
Peter Amstutz [Fri, 21 Oct 2016 02:46:54 +0000 (22:46 -0400)]
Check docker version to decide whether to use -f with docker tag, refs #10025

8 years ago10172: Add test for API: true and container setting output.
Peter Amstutz [Fri, 21 Oct 2016 01:38:28 +0000 (21:38 -0400)]
10172: Add test for API: true and container setting output.

8 years agoMerge branch '10025-arvbox-layers' closes #10025
Peter Amstutz [Fri, 21 Oct 2016 01:25:15 +0000 (21:25 -0400)]
Merge branch '10025-arvbox-layers' closes #10025

8 years ago10315: Bring back arv-put command from 9701's branch.
Lucas Di Pentima [Thu, 20 Oct 2016 22:11:09 +0000 (19:11 -0300)]
10315: Bring back arv-put command from 9701's branch.

8 years ago10172: If RuntimeConstraints.API is set, refresh container record to check if
Peter Amstutz [Thu, 20 Oct 2016 21:06:48 +0000 (17:06 -0400)]
10172: If RuntimeConstraints.API is set, refresh container record to check if
output field has been already set by the container.

8 years ago10172: Bugfix getting container output when container is reused. Tweak output
Peter Amstutz [Thu, 20 Oct 2016 21:05:28 +0000 (17:05 -0400)]
10172: Bugfix getting container output when container is reused.  Tweak output
to report "Job" or "Container" depending on which API is in use.

8 years ago10172: crunch_script only updates task record on error, will normally be
Peter Amstutz [Thu, 20 Oct 2016 20:08:52 +0000 (16:08 -0400)]
10172: crunch_script only updates task record on error, will normally be
updated in set_crunch_output().

8 years agoMerge branch 'master' into 10172-crunch2-container-output
Peter Amstutz [Thu, 20 Oct 2016 19:47:59 +0000 (15:47 -0400)]
Merge branch 'master' into 10172-crunch2-container-output

8 years ago10172: Detect when running as crunch container or job and set output.
Peter Amstutz [Thu, 20 Oct 2016 19:42:28 +0000 (15:42 -0400)]
10172: Detect when running as crunch container or job and set output.

8 years ago10172: Tests and related fixes for auth_uuid setting output on container.
Peter Amstutz [Thu, 20 Oct 2016 19:16:00 +0000 (15:16 -0400)]
10172: Tests and related fixes for auth_uuid setting output on container.

8 years agoMerge branch '10291-really-disable-apis' closes #10291
Tom Clegg [Thu, 20 Oct 2016 15:53:49 +0000 (11:53 -0400)]
Merge branch '10291-really-disable-apis' closes #10291

8 years ago10291: Test multiple disabled APIs.
Tom Clegg [Thu, 20 Oct 2016 13:11:08 +0000 (09:11 -0400)]
10291: Test multiple disabled APIs.

8 years ago10291: Respond 404 to requests for disabled APIs.
Tom Clegg [Thu, 20 Oct 2016 13:06:10 +0000 (09:06 -0400)]
10291: Respond 404 to requests for disabled APIs.

8 years agoMerge branch '10287-sso-username' refs #10287
Tom Clegg [Thu, 20 Oct 2016 10:01:43 +0000 (06:01 -0400)]
Merge branch '10287-sso-username' refs #10287

8 years agoMerge branch '10291-discovery-blacklist' refs #10291
Tom Clegg [Thu, 20 Oct 2016 09:15:49 +0000 (05:15 -0400)]
Merge branch '10291-discovery-blacklist' refs #10291

8 years ago10287: Perform blacklist and duplicate checks on usernames received from SSO.
Tom Clegg [Thu, 20 Oct 2016 08:58:27 +0000 (04:58 -0400)]
10287: Perform blacklist and duplicate checks on usernames received from SSO.

8 years agoSupport Docker v1.10+ hashes that include sha256: prefix.
Thomas Mooney [Wed, 19 Oct 2016 22:59:52 +0000 (17:59 -0500)]
Support Docker v1.10+ hashes that include sha256: prefix.

8 years ago10287: When creating a new account, if provided use 'username' from SSO server.
Peter Amstutz [Wed, 19 Oct 2016 21:14:51 +0000 (17:14 -0400)]
10287: When creating a new account, if provided use 'username' from SSO server.

8 years agoMerge branch '10216-allow-cr-bypass-reuse'
Lucas Di Pentima [Wed, 19 Oct 2016 20:11:04 +0000 (17:11 -0300)]
Merge branch '10216-allow-cr-bypass-reuse'
Closes #10216