h3. Logs table purpose & behavior
-This database table currently serves three purposes:
-* It's an audit log, permitting admins and users to look up the time and details of past changes to Arvados objects via @arvados.v1.logs.*@ endpoints.
-* It's a mechanism for passing cache-invalidation events, used by websocket servers, the Python SDK "events" library, and @arvados-cwl-runner@ to detect when an object has changed.
-* It's a staging area for stdout/stderr text coming from users' containers, permitting users to see what their containers are doing while they are still running (i.e., before those text files are written to Keep).
+This database table is accessed via "the @logs@ endpoint.":../api/methods/logs.html
+
+This table currently serves several purposes:
+
+* Audit logging, permitting admins and users to look up the time and details of past changes to Arvados objects.
+* Logging other system events, specifically "file uploads and downloads from keep-web.":restricting-upload-download.html#audit_logs
+* The source for cache-invalidation events, published through websockets to Workbench to refresh the view. It can also be monitored by the Python SDK "events module.":../sdk/python/events.html
+* Prior to Arvados 2.7, it was used a staging area for stdout/stderr text coming from users' containers, permitting users to see what their containers are doing while they are still running (i.e., before those text files are written to Keep). Starting with Arvados 2.7, this is superseded by a more efficient mechanism, so these logs are disabled by default. See "2.7.0 upgrade notes":upgrading.html#v2_7_0 for details.
As a result, this table grows indefinitely, even on sites where policy does not require an audit log; making backups, migrations, and upgrades unnecessarily slow and painful.
-h3. API Server configuration
+h3. Configuration
-To solve the problem mentioned above, the API server offers the possibility to limit the amount of log information stored on the table:
+To solve the problem mentioned above, the @AuditLogs@ section of @config.yml@ offers several options to limit the amount of log information stored on the table:
<pre>
-# Time to keep audit logs (a row in the log table added each time an
-# Arvados object is created, modified, or deleted) in the PostgreSQL
-# database. Currently, websocket event notifications rely on audit
-# logs, so this should not be set lower than 600 (10 minutes).
-max_audit_log_age: 1209600
-</pre>
+ AuditLogs:
+ # Time to keep audit logs. (An audit log is a row added
+ # to the "logs" table in the PostgreSQL database each time an
+ # Arvados object is created, modified, or deleted.)
+ #
+ # Currently, websocket event notifications rely on audit logs, so
+ # this should not be set lower than 5 minutes.
+ MaxAge: 336h
-...and to prevent surprises and avoid bad database behavior (especially the first time the cleanup job runs on an existing cluster with a huge backlog) a maximum number of rows to delete in a single transaction.
+ # Maximum number of log rows to delete in a single SQL transaction,
+ # to prevent surprises and avoid bad database behavior
+ # (especially the first time the cleanup job runs on an existing
+ # cluster with a huge backlog) a maximum number of rows to
+ # delete in a single transaction.
+ #
+ # If MaxDeleteBatch is 0, log entries will never be
+ # deleted by Arvados. Cleanup can be done by an external process
+ # without affecting any Arvados system processes, as long as very
+ # recent (<5 minutes old) logs are not deleted.
+ #
+ # 100000 is a reasonable batch size for most sites.
+ MaxDeleteBatch: 0
-<pre>
-# Maximum number of log rows to delete in a single SQL transaction.
-#
-# If max_audit_log_delete_batch is 0, log entries will never be
-# deleted by Arvados. Cleanup can be done by an external process
-# without affecting any Arvados system processes, as long as very
-# recent (<5 minutes old) logs are not deleted.
-#
-# 100000 is a reasonable batch size for most sites.
-max_audit_log_delete_batch: 0
+ # Attributes to suppress in events and audit logs. Notably,
+ # specifying {"manifest_text": {}} here typically makes the database
+ # smaller and faster.
+ #
+ # Warning: Using any non-empty value here can have undesirable side
+ # effects for any client or component that relies on event logs.
+ # Use at your own risk.
+ UnloggedAttributes: {}
</pre>
-This feature works when both settings are non-zero, periodically dispatching a background task that deletes all log rows older than @max_audit_log_age@.
-The events being cleaned up by this process don't include job/container stderr logs (they're handled by the existing @delete job/container logs@ rake tasks)
h3. Additional consideration
projects / arvados.git / blobdiff