arv-web sample

From 17941a42ec6772f67dc7b451039cd706301f1bdd Mon Sep 17 00:00:00 2001 From: Nico Cesar Date: Tue, 25 Aug 2020 09:54:04 -0400 Subject: [PATCH] Remove arv-web.py and documentation. refs #15886 Arvados-DCO-1.1-Signed-off-by: Nico Cesar --- doc/user/topics/arv-web.html.textile.liquid | 106 -------- services/arv-web/README | 6 - services/arv-web/arv-web.py | 256 ------------------ services/arv-web/sample-cgi-app/docker_image | 1 - .../arv-web/sample-cgi-app/public/.htaccess | 3 - .../arv-web/sample-cgi-app/public/index.cgi | 4 - services/arv-web/sample-cgi-app/tmp/.keepkeep | 0 services/arv-web/sample-rack-app/config.ru | 8 - services/arv-web/sample-rack-app/docker_image | 1 - .../arv-web/sample-rack-app/public/.keepkeep | 0 .../arv-web/sample-rack-app/tmp/.keepkeep | 0 .../arv-web/sample-static-page/docker_image | 1 - .../sample-static-page/public/index.html | 10 - .../arv-web/sample-static-page/tmp/.keepkeep | 0 services/arv-web/sample-wsgi-app/docker_image | 1 - .../arv-web/sample-wsgi-app/passenger_wsgi.py | 7 - .../arv-web/sample-wsgi-app/public/.keepkeep | 0 .../arv-web/sample-wsgi-app/tmp/.keepkeep | 0 18 files changed, 404 deletions(-) delete mode 100644 doc/user/topics/arv-web.html.textile.liquid delete mode 100644 services/arv-web/README delete mode 100755 services/arv-web/arv-web.py delete mode 100644 services/arv-web/sample-cgi-app/docker_image delete mode 100644 services/arv-web/sample-cgi-app/public/.htaccess delete mode 100755 services/arv-web/sample-cgi-app/public/index.cgi delete mode 100644 services/arv-web/sample-cgi-app/tmp/.keepkeep delete mode 100644 services/arv-web/sample-rack-app/config.ru delete mode 100644 services/arv-web/sample-rack-app/docker_image delete mode 100644 services/arv-web/sample-rack-app/public/.keepkeep delete mode 100644 services/arv-web/sample-rack-app/tmp/.keepkeep delete mode 100644 services/arv-web/sample-static-page/docker_image delete mode 100644 services/arv-web/sample-static-page/public/index.html delete mode 100644 services/arv-web/sample-static-page/tmp/.keepkeep delete mode 100644 services/arv-web/sample-wsgi-app/docker_image delete mode 100644 services/arv-web/sample-wsgi-app/passenger_wsgi.py delete mode 100644 services/arv-web/sample-wsgi-app/public/.keepkeep delete mode 100644 services/arv-web/sample-wsgi-app/tmp/.keepkeep diff --git a/doc/user/topics/arv-web.html.textile.liquid b/doc/user/topics/arv-web.html.textile.liquid deleted file mode 100644 index 9671e97096..0000000000 --- a/doc/user/topics/arv-web.html.textile.liquid +++ /dev/null @@ -1,106 +0,0 @@ ---- -layout: default -navsection: userguide -title: "Using arv-web" -... -{% comment %} -Copyright (C) The Arvados Authors. All rights reserved. - -SPDX-License-Identifier: CC-BY-SA-3.0 -{% endcomment %} - -@arv-web@ enables you to run a custom web service from the contents of an Arvados collection. - -{% include 'tutorial_expectations_workstation' %} - -h2. Usage - -@arv-web@ enables you to set up a web service based on the most recent collection in a project. An arv-web application is a reproducible, immutable application bundle where the web app is packaged with both the code to run and the data to serve. Because Arvados Collections can be updated with minimum duplication, it is efficient to produce a new application bundle when the code or data needs to be updated; retaining old application bundles makes it easy to go back and run older versions of your web app. - -

-$ cd $HOME/arvados/services/arv-web
-usage: arv-web.py [-h] --project-uuid PROJECT_UUID [--port PORT]
-                  [--image IMAGE]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  --project-uuid PROJECT_UUID
-                        Project uuid to watch
-  --port PORT           Host port to listen on (default 8080)
-  --image IMAGE         Docker image to run
-

- -At startup, @arv-web@ queries an Arvados project and mounts the most recently modified collection into a temporary directory. It then runs a Docker image with the collection bound to @/mnt@ inside the container. When a new collection is added to the project, or an existing project is updated, it will stop the running Docker container, unmount the old collection, mount the new most recently modified collection, and restart the Docker container with the new mount. - -h2. Docker container - -The @Dockerfile@ in @arvados/docker/arv-web@ builds a Docker image that runs Apache with @/mnt@ as the DocumentRoot. It is configured to run web applications which use Python WSGI, Ruby Rack, or CGI; to serve static HTML; or browse the contents of the @public@ subdirectory of the collection using default Apache index pages. - -To build the Docker image: - - -

~$ cd arvados/docker
-~/arvados/docker$ docker build -t arvados/arv-web arv-web
-

- - -h2. Running sample applications - -First, in Arvados Workbench, create a new project. Copy the project UUID from the URL bar (this is the part of the URL after @projects/...@). - -Now upload a collection containing a "Python WSGI web app:":http://wsgi.readthedocs.org/en/latest/ - - -

~$ cd arvados/services/arv-web
-~/arvados/services/arv-web$ arv-put --project [zzzzz-j7d0g-yourprojectuuid] --name sample-wsgi-app sample-wsgi-app
-0M / 0M 100.0%
-Collection saved as 'sample-wsgi-app'
-zzzzz-4zz18-ebohzfbzh82qmqy
-~/arvados/services/arv-web$ ./arv-web.py --project [zzzzz-j7d0g-yourprojectuuid] --port 8888
-2015-01-30 11:21:00 arvados.arv-web[4897] INFO: Mounting zzzzz-4zz18-ebohzfbzh82qmqy
-2015-01-30 11:21:01 arvados.arv-web[4897] INFO: Starting Docker container arvados/arv-web
-2015-01-30 11:21:02 arvados.arv-web[4897] INFO: Container id e79e70558d585a3e038e4bfbc97e5c511f21b6101443b29a8017bdf3d84689a3
-2015-01-30 11:21:03 arvados.arv-web[4897] INFO: Waiting for events
-

- - -The sample application will be available at @http://localhost:8888@. - -h3. Updating the application - -If you upload a new collection to the same project, arv-web will restart the web service and serve the new collection. For example, uploading a collection containing a "Ruby Rack web app:":https://github.com/rack/rack/wiki - - -

~$ cd arvados/services/arv-web
-~/arvados/services/arv-web$ arv-put --project [zzzzz-j7d0g-yourprojectuuid] --name sample-rack-app sample-rack-app
-0M / 0M 100.0%
-Collection saved as 'sample-rack-app'
-zzzzz-4zz18-dhhm0ay8k8cqkvg
-

- - -@arv-web@ will automatically notice the change, load a new container, and send an update signal (SIGHUP) to the service: - -

-2015-01-30 11:21:03 arvados.arv-web[4897] INFO:Waiting for events
-2015-01-30 11:21:04 arvados.arv-web[4897] INFO:create zzzzz-4zz18-dhhm0ay8k8cqkvg
-2015-01-30 11:21:05 arvados.arv-web[4897] INFO:Mounting zzzzz-4zz18-dhhm0ay8k8cqkvg
-2015-01-30 11:21:06 arvados.arv-web[4897] INFO:Sending refresh signal to container
-2015-01-30 11:21:07 arvados.arv-web[4897] INFO:Waiting for events
-

- -h2. Writing your own applications - -The @arvados/arv-web@ image serves Python and Ruby applications using Phusion Passenger and Apache @mod_passenger@. See "Phusion Passenger users guide for Apache":https://www.phusionpassenger.com/documentation/Users%20guide%20Apache.html for details, and look at the sample apps @arvados/services/arv-web/sample-wsgi-app@ and @arvados/services/arv-web/sample-rack-app@. - -You can serve CGI applications using standard Apache CGI support. See "Apache Tutorial: Dynamic Content with CGI":https://httpd.apache.org/docs/current/howto/cgi.html for details, and look at the sample app @arvados/services/arv-web/sample-cgi-app@. - -You can also serve static content from the @public@ directory of the collection. Look at @arvados/services/arv-web/sample-static-page@ for an example. If no @index.html@ is found in @public/@, it will render default Apache index pages, permitting simple browsing of the collection contents. - -h3. Custom images - -You can provide your own Docker image. The Docker image that will be used create the web application container is specified in the @docker_image@ file in the root of the collection. You can also specify @--image@ on the command @arv-web@ line to choose the docker image (this will override the contents of @docker_image@). - -h3. Reloading the web service - -Stopping the Docker container and starting it again can result in a small amount of downtime. When the collection containing a new or updated web application uses the same Docker image as the currently running web application, it is possible to avoid this downtime by keeping the existing container and only reloading the web server. This is accomplished by providing a file called @reload@ in the root of the collection, which should contain the commands necessary to reload the web server inside the container. diff --git a/services/arv-web/README b/services/arv-web/README deleted file mode 100644 index eaf7624dc4..0000000000 --- a/services/arv-web/README +++ /dev/null @@ -1,6 +0,0 @@ -arv-web enables you to run a custom web service using the contents of an -Arvados collection. - -See "Using arv-web" in the Arvados user guide: - -http://doc.arvados.org/user/topics/arv-web.html diff --git a/services/arv-web/arv-web.py b/services/arv-web/arv-web.py deleted file mode 100755 index 55b710a754..0000000000 --- a/services/arv-web/arv-web.py +++ /dev/null @@ -1,256 +0,0 @@ -#!/usr/bin/env python -# Copyright (C) The Arvados Authors. All rights reserved. -# -# SPDX-License-Identifier: AGPL-3.0 - -# arv-web enables you to run a custom web service from the contents of an Arvados collection. -# -# See http://doc.arvados.org/user/topics/arv-web.html - -import arvados -from arvados.safeapi import ThreadSafeApiCache -import subprocess -from arvados_fuse import Operations, CollectionDirectory -import tempfile -import os -import llfuse -import threading -import Queue -import argparse -import logging -import signal -import sys -import functools - -logger = logging.getLogger('arvados.arv-web') -logger.setLevel(logging.INFO) - -class ArvWeb(object): - def __init__(self, project, docker_image, port): - self.project = project - self.loop = True - self.cid = None - self.prev_docker_image = None - self.mountdir = None - self.collection = None - self.override_docker_image = docker_image - self.port = port - self.evqueue = Queue.Queue() - self.api = ThreadSafeApiCache(arvados.config.settings()) - - if arvados.util.group_uuid_pattern.match(project) is None: - raise arvados.errors.ArgumentError("Project uuid is not valid") - - collections = self.api.collections().list(filters=[["owner_uuid", "=", project]], - limit=1, - order='modified_at desc').execute()['items'] - self.newcollection = collections[0]['uuid'] if collections else None - - self.ws = arvados.events.subscribe(self.api, [["object_uuid", "is_a", "arvados#collection"]], self.on_message) - - def check_docker_running(self): - # It would be less hacky to use "docker events" than poll "docker ps" - # but that would require writing a bigger pile of code. - if self.cid: - ps = subprocess.check_output(["docker", "ps", "--no-trunc=true", "--filter=status=running"]) - for l in ps.splitlines(): - if l.startswith(self.cid): - return True - return False - - # Handle messages from Arvados event bus. - def on_message(self, ev): - if 'event_type' in ev: - old_attr = None - if 'old_attributes' in ev['properties'] and ev['properties']['old_attributes']: - old_attr = ev['properties']['old_attributes'] - if self.project not in (ev['properties']['new_attributes']['owner_uuid'], - old_attr['owner_uuid'] if old_attr else None): - return - - et = ev['event_type'] - if ev['event_type'] == 'update': - if ev['properties']['new_attributes']['owner_uuid'] != ev['properties']['old_attributes']['owner_uuid']: - if self.project == ev['properties']['new_attributes']['owner_uuid']: - et = 'add' - else: - et = 'remove' - if ev['properties']['new_attributes']['trash_at'] is not None: - et = 'remove' - - self.evqueue.put((self.project, et, ev['object_uuid'])) - - # Run an arvados_fuse mount under the control of the local process. This lets - # us switch out the contents of the directory without having to unmount and - # remount. - def run_fuse_mount(self): - self.mountdir = tempfile.mkdtemp() - - self.operations = Operations(os.getuid(), os.getgid(), self.api, "utf-8") - self.cdir = CollectionDirectory(llfuse.ROOT_INODE, self.operations.inodes, self.api, 2, self.collection) - self.operations.inodes.add_entry(self.cdir) - - # Initialize the fuse connection - llfuse.init(self.operations, self.mountdir, ['allow_other']) - - t = threading.Thread(None, llfuse.main) - t.start() - - # wait until the driver is finished initializing - self.operations.initlock.wait() - - def mount_collection(self): - if self.newcollection != self.collection: - self.collection = self.newcollection - if not self.mountdir and self.collection: - self.run_fuse_mount() - - if self.mountdir: - with llfuse.lock: - self.cdir.clear() - # Switch the FUSE directory object so that it stores - # the newly selected collection - if self.collection: - logger.info("Mounting %s", self.collection) - else: - logger.info("Mount is empty") - self.cdir.change_collection(self.collection) - - - def stop_docker(self): - if self.cid: - logger.info("Stopping Docker container") - subprocess.call(["docker", "stop", self.cid]) - self.cid = None - - def run_docker(self): - try: - if self.collection is None: - self.stop_docker() - return - - docker_image = None - if self.override_docker_image: - docker_image = self.override_docker_image - else: - try: - with llfuse.lock: - if "docker_image" in self.cdir: - docker_image = self.cdir["docker_image"].readfrom(0, 1024).strip() - except IOError as e: - pass - - has_reload = False - try: - with llfuse.lock: - has_reload = "reload" in self.cdir - except IOError as e: - pass - - if docker_image is None: - logger.error("Collection must contain a file 'docker_image' or must specify --image on the command line.") - self.stop_docker() - return - - if docker_image == self.prev_docker_image and self.cid is not None and has_reload: - logger.info("Running container reload command") - subprocess.check_call(["docker", "exec", self.cid, "/mnt/reload"]) - return - - self.stop_docker() - - logger.info("Starting Docker container %s", docker_image) - self.cid = subprocess.check_output(["docker", "run", - "--detach=true", - "--publish=%i:80" % (self.port), - "--volume=%s:/mnt:ro" % self.mountdir, - docker_image]).strip() - - self.prev_docker_image = docker_image - logger.info("Container id %s", self.cid) - - except subprocess.CalledProcessError: - self.cid = None - - def wait_for_events(self): - if not self.cid: - logger.warning("No service running! Will wait for a new collection to appear in the project.") - else: - logger.info("Waiting for events") - - running = True - self.loop = True - while running: - # Main run loop. Wait on project events, signals, or the - # Docker container stopping. - - try: - # Poll the queue with a 1 second timeout, if we have no - # timeout the Python runtime doesn't have a chance to - # process SIGINT or SIGTERM. - eq = self.evqueue.get(True, 1) - logger.info("%s %s", eq[1], eq[2]) - self.newcollection = self.collection - if eq[1] in ('add', 'update', 'create'): - self.newcollection = eq[2] - elif eq[1] == 'remove': - collections = self.api.collections().list(filters=[["owner_uuid", "=", self.project]], - limit=1, - order='modified_at desc').execute()['items'] - self.newcollection = collections[0]['uuid'] if collections else None - running = False - except Queue.Empty: - pass - - if self.cid and not self.check_docker_running(): - logger.warning("Service has terminated. Will try to restart.") - self.cid = None - running = False - - - def run(self): - try: - while self.loop: - self.loop = False - self.mount_collection() - try: - self.run_docker() - self.wait_for_events() - except (KeyboardInterrupt): - logger.info("Got keyboard interrupt") - self.ws.close() - self.loop = False - except Exception as e: - logger.exception("Caught fatal exception, shutting down") - self.ws.close() - self.loop = False - finally: - self.stop_docker() - - if self.mountdir: - logger.info("Unmounting") - subprocess.call(["fusermount", "-u", self.mountdir]) - os.rmdir(self.mountdir) - - -def main(argv): - parser = argparse.ArgumentParser() - parser.add_argument('--project-uuid', type=str, required=True, help="Project uuid to watch") - parser.add_argument('--port', type=int, default=8080, help="Host port to listen on (default 8080)") - parser.add_argument('--image', type=str, help="Docker image to run") - - args = parser.parse_args(argv) - - signal.signal(signal.SIGTERM, lambda signal, frame: sys.exit(0)) - - try: - arvweb = ArvWeb(args.project_uuid, args.image, args.port) - arvweb.run() - except arvados.errors.ArgumentError as e: - logger.error(e) - return 1 - - return 0 - -if __name__ == '__main__': - sys.exit(main(sys.argv[1:])) diff --git a/services/arv-web/sample-cgi-app/docker_image b/services/arv-web/sample-cgi-app/docker_image deleted file mode 100644 index 57f344fcd7..0000000000 --- a/services/arv-web/sample-cgi-app/docker_image +++ /dev/null @@ -1 +0,0 @@ -arvados/arv-web \ No newline at end of file diff --git a/services/arv-web/sample-cgi-app/public/.htaccess b/services/arv-web/sample-cgi-app/public/.htaccess deleted file mode 100644 index e5145bd37d..0000000000 --- a/services/arv-web/sample-cgi-app/public/.htaccess +++ /dev/null @@ -1,3 +0,0 @@ -Options +ExecCGI -AddHandler cgi-script .cgi -DirectoryIndex index.cgi diff --git a/services/arv-web/sample-cgi-app/public/index.cgi b/services/arv-web/sample-cgi-app/public/index.cgi deleted file mode 100755 index 57bc2a9a01..0000000000 --- a/services/arv-web/sample-cgi-app/public/index.cgi +++ /dev/null @@ -1,4 +0,0 @@ -#!/usr/bin/perl - -print "Content-type: text/html\n\n"; -print "Hello world from perl!"; diff --git a/services/arv-web/sample-cgi-app/tmp/.keepkeep b/services/arv-web/sample-cgi-app/tmp/.keepkeep deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/services/arv-web/sample-rack-app/config.ru b/services/arv-web/sample-rack-app/config.ru deleted file mode 100644 index 65f3c7ca36..0000000000 --- a/services/arv-web/sample-rack-app/config.ru +++ /dev/null @@ -1,8 +0,0 @@ -# Copyright (C) The Arvados Authors. All rights reserved. -# -# SPDX-License-Identifier: AGPL-3.0 - -app = proc do |env| - [200, { "Content-Type" => "text/html" }, ["hello world from ruby"]] -end -run app diff --git a/services/arv-web/sample-rack-app/docker_image b/services/arv-web/sample-rack-app/docker_image deleted file mode 100644 index 57f344fcd7..0000000000 --- a/services/arv-web/sample-rack-app/docker_image +++ /dev/null @@ -1 +0,0 @@ -arvados/arv-web \ No newline at end of file diff --git a/services/arv-web/sample-rack-app/public/.keepkeep b/services/arv-web/sample-rack-app/public/.keepkeep deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/services/arv-web/sample-rack-app/tmp/.keepkeep b/services/arv-web/sample-rack-app/tmp/.keepkeep deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/services/arv-web/sample-static-page/docker_image b/services/arv-web/sample-static-page/docker_image deleted file mode 100644 index 57f344fcd7..0000000000 --- a/services/arv-web/sample-static-page/docker_image +++ /dev/null @@ -1 +0,0 @@ -arvados/arv-web \ No newline at end of file diff --git a/services/arv-web/sample-static-page/public/index.html b/services/arv-web/sample-static-page/public/index.html deleted file mode 100644 index e8608a5ebe..0000000000 --- a/services/arv-web/sample-static-page/public/index.html +++ /dev/null @@ -1,10 +0,0 @@ - - - - arv-web sample - -

Hello world static page

- - diff --git a/services/arv-web/sample-static-page/tmp/.keepkeep b/services/arv-web/sample-static-page/tmp/.keepkeep deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/services/arv-web/sample-wsgi-app/docker_image b/services/arv-web/sample-wsgi-app/docker_image deleted file mode 100644 index 57f344fcd7..0000000000 --- a/services/arv-web/sample-wsgi-app/docker_image +++ /dev/null @@ -1 +0,0 @@ -arvados/arv-web \ No newline at end of file diff --git a/services/arv-web/sample-wsgi-app/passenger_wsgi.py b/services/arv-web/sample-wsgi-app/passenger_wsgi.py deleted file mode 100644 index faec3c23cd..0000000000 --- a/services/arv-web/sample-wsgi-app/passenger_wsgi.py +++ /dev/null @@ -1,7 +0,0 @@ -# Copyright (C) The Arvados Authors. All rights reserved. -# -# SPDX-License-Identifier: AGPL-3.0 - -def application(environ, start_response): - start_response('200 OK', [('Content-Type', 'text/plain')]) - return [b"hello world from python!\n"] diff --git a/services/arv-web/sample-wsgi-app/public/.keepkeep b/services/arv-web/sample-wsgi-app/public/.keepkeep deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/services/arv-web/sample-wsgi-app/tmp/.keepkeep b/services/arv-web/sample-wsgi-app/tmp/.keepkeep deleted file mode 100644 index e69de29bb2..0000000000 -- 2.30.2