Merge pull request #620 from fschade/backport-antivirus-max-size-fix

fix(antivirus | backport): introduce a default max scan size for the full example deployment

deployments/examples/opencloud_full/.env

@@ -99,7 +99,7 @@ MINIO_DOMAIN=
 #DECOMPOSED=:decomposed.yml
 
 # Define SMPT settings if you would like to send OpenCloud email notifications.
-# 
+#
 # NOTE: when configuring Inbucket, these settings have no effect, see inbucket.yml for details.
 # SMTP host to connect to.
 SMTP_HOST=
@@ -210,6 +210,10 @@ COLLABORA_SSL_VERIFICATION=false
 # envvar in the OpenCloud Settings above by adding 'antivirus' to the list.
 # Note: the leading colon is required to enable the service.
 #CLAMAV=:clamav.yml
+# The maximum scan size the virus scanner can handle, needs adjustment in the scanner config as well.
+# Usable common abbreviations: [KB, KiB, MB, MiB, GB, GiB, TB, TiB, PB, PiB, EB, EiB], example: 2GB.
+# Defaults to "100MB"
+#ANTIVIRUS_MAX_SCAN_SIZE=
 # Image version of the ClamAV container.
 # Defaults to "latest"
 CLAMAV_DOCKER_TAG=

deployments/examples/opencloud_full/clamav.yml

@@ -4,6 +4,7 @@ services:
     environment:
       ANTIVIRUS_SCANNER_TYPE: "clamav"
       ANTIVIRUS_CLAMAV_SOCKET: "/var/run/clamav/clamd.sock"
+      ANTIVIRUS_MAX_SCAN_SIZE: ${ANTIVIRUS_MAX_SCAN_SIZE:-100MB}
       # the antivirus service needs manual startup, see .env and opencloud.yaml for START_ADDITIONAL_SERVICES
       # configure the antivirus service
       POSTPROCESSING_STEPS: "virusscan"

services/antivirus/README.md

@@ -4,7 +4,10 @@ The `antivirus` service is responsible for scanning files for viruses.
 
 ## Memory Considerations
 
-The antivirus service can consume considerably amounts of memory. This is relevant to provide or define sufficient memory for the deployment selected. To avoid out of memory (OOM) situations, the following equation gives a rough overview based on experiences made. The memory calculation comes without any guarantee, is intended as overview only and subject of change.
+The antivirus service can consume considerable amounts of memory.
+This is relevant to provide or define sufficient memory for the deployment selected.
+To avoid out of memory (OOM) situations, the following equation gives a rough overview based on experiences made.
+The memory calculation comes without any guarantee, is intended as overview only and subject of change.
 
 `memory limit` = `max file size` x `workers` x `factor 8 - 14`
 
@@ -19,17 +22,25 @@ With:
 
 ### Antivirus Scanner Type
 
-The antivirus service currently supports [ICAP](https://tools.ietf.org/html/rfc3507) and [ClamAV](http://www.clamav.net/index.html) as antivirus scanners. The `ANTIVIRUS_SCANNER_TYPE` environment variable is used to select the scanner. The detailed configuration for each scanner heavily depends on the scanner type selected. See the environment variables for more details.
+The antivirus service currently supports [ICAP](https://tools.ietf.org/html/rfc3507) and [ClamAV](http://www.clamav.net/index.html) as antivirus scanners.
+The `ANTIVIRUS_SCANNER_TYPE` environment variable is used to select the scanner.
+The detailed configuration for each scanner heavily depends on the scanner type selected.
+See the environment variables for more details.
 
   -   For `icap`, only scanners using the `X-Infection-Found` header are currently supported.
   -   For `clamav` only local sockets can currently be configured.
 
 ### Maximum Scan Size
 
-Several factors can make it necessary to limit the maximum filesize the antivirus service will use for scanning. Use the `ANTIVIRUS_MAX_SCAN_SIZE` environment variable to scan only a given amount of bytes. Obviously, it is recommended to scan the whole file, but several factors like scanner type and version, bandwidth, performance issues, etc. might make a limit necessary.
+Several factors can make it necessary to limit the maximum filesize the antivirus service uses for scanning.
+Use the `ANTIVIRUS_MAX_SCAN_SIZE` environment variable to specify the maximum file size to be scanned.
+
+Even if it's recommended to scan each file, several factors like scanner type and version,
+bandwidth, performance issues, etc. might make a limit necessary.
 
 **IMPORTANT**
-> Streaming of files to the virus scan service still [needs to be implemented](https://github.com/owncloud/ocis/issues/6803). To prevent OOM errors `ANTIVIRUS_MAX_SCAN_SIZE` needs to be set lower than available ram.
+> Streaming of files to the virus scan service still [needs to be implemented](https://github.com/owncloud/ocis/issues/6803).
+> To prevent OOM errors `ANTIVIRUS_MAX_SCAN_SIZE` needs to be set lower than available ram and or the maximum file size that can be scanned by the virus scanner.
 
 ### Antivirus Workers
 
@@ -41,7 +52,7 @@ The antivirus service allows three different ways of handling infected files. Th
 
   -   `delete`: (default): Infected files will be deleted immediately, further postprocessing is cancelled.
   -   `abort`:  (advanced option): Infected files will be kept, further postprocessing is cancelled. Files can be manually retrieved and inspected by an admin. To identify the file for further investigation, the antivirus service logs the abort/infected state including the file ID. The file is located in the `storage/users/uploads` folder of the OpenCloud data directory and persists until it is manually deleted by the admin via the [Manage Unfinished Uploads](https://github.com/opencloud-eu/opencloud/tree/main/services/storage-users#manage-unfinished-uploads) command.
-  -   `continue`:  (obviously not recommended): Infected files will be marked via metadata as infected but postprocessing continues normally. Note: Infected Files are moved to their final destination and therefore not prevented from download which includes the risk of spreading viruses.
+  -   `continue`:  (not recommended): Infected files will be marked via metadata as infected, but postprocessing continues normally. Note: Infected Files are moved to their final destination and therefore not prevented from download, which includes the risk of spreading viruses.
 
 In all cases, a log entry is added declaring the infection and handling method and a notification via the `userlog` service sent.
 

services/antivirus/pkg/config/config.go

@@ -21,7 +21,7 @@ type Config struct {
 	Workers              int `yaml:"workers" env:"ANTIVIRUS_WORKERS" desc:"The number of concurrent go routines that fetch events from the event queue." introductionVersion:"1.0.0"`
 
 	Scanner     Scanner
-	MaxScanSize string `yaml:"max-scan-size" env:"ANTIVIRUS_MAX_SCAN_SIZE" desc:"The maximum scan size the virus scanner can handle. Only this many bytes of a file will be scanned. 0 means unlimited and is the default. Usable common abbreviations: [KB, KiB, MB, MiB, GB, GiB, TB, TiB, PB, PiB, EB, EiB], example: 2GB." introductionVersion:"1.0.0"`
+	MaxScanSize string `yaml:"max-scan-size" env:"ANTIVIRUS_MAX_SCAN_SIZE" desc:"The maximum scan size the virus scanner can handle. 0 means unlimited. Usable common abbreviations: [KB, KiB, MB, MiB, GB, GiB, TB, TiB, PB, PiB, EB, EiB], example: 2GB." introductionVersion:"1.0.0"`
 
 	Context context.Context `json:"-" yaml:"-"`