configurable antivirus workers

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

changelog/unreleased/antivirus-workers.md

@@ -0,0 +1,5 @@
+Bugfix: make antivirus workers configurable
+
+We made the number of go routines that pull events from the queue configurable. 
+
+https://github.com/owncloud/ocis/pull/10383

services/antivirus/README.md

@@ -15,6 +15,9 @@ The antivirus service currently supports [ICAP](https://tools.ietf.org/html/rfc3
 
 Several factors can make it necessary to limit the maximum filesize the antivirus service will use for scanning. Use the `ANTIVIRUS_MAX_SCAN_SIZE` environment variable to scan only a given amount of bytes. Obviously, it is recommended to scan the whole file, but several factors like scanner type and version, bandwidth, performance issues, etc. might make a limit necessary.
 
+> [!CAUTION]
+> Streaming of files to the virus scan service still [needs to be implemented](https://github.com/owncloud/ocis/issues/6803). To prevent OOM errors `ANTIVIRUS_MAX_SCAN_SIZE` needs to be set lower than available ram.
+
 ### Infected File Handling
 
 The antivirus service allows three different ways of handling infected files. Those can be set via the `ANTIVIRUS_INFECTED_FILE_HANDLING` environment variable:
@@ -36,3 +39,9 @@ The antivirus service can scan files during `postprocessing`. `on demand` scanni
 ### Postprocessing
 
 The antivirus service will scan files during postprocessing. It listens for a postprocessing step called `virusscan`. This step can be added in the environment variable `POSTPROCESSING_STEPS`. Read the documentation of the [postprocessing service](https://github.com/owncloud/ocis/tree/master/services/postprocessing) for more details.
+
+The number of concurrent scans can be increased by setting `ANTIVIRUS_WORKERS`, but be aware that this will also increase the memory usage.
+
+### Scaling in Kubernetes
+
+In kubernetes `ANTIVIRUS_WORKERS` and `ANTIVIRUS_MAX_SCAN_SIZE` can be used to trigger the horizontal pod autoscaler by requesting a memory size that is below `ANTIVIRUS_MAX_SCAN_SIZE`. Keep in mind that `ANTIVIRUS_MAX_SCAN_SIZE` amount of memory might be held by `ANTIVIRUS_WORKERS` number of go routines.

services/antivirus/pkg/config/defaults/defaultconfig.go

@@ -28,6 +28,7 @@ func DefaultConfig() *config.Config {
 			Endpoint: "127.0.0.1:9233",
 			Cluster:  "ocis-cluster",
 		},
+		Workers:              10,
 		InfectedFileHandling: "delete",
 		Scanner: config.Scanner{
 			Type: "clamav",

services/antivirus/pkg/service/service.go

@@ -9,6 +9,7 @@ import (
 	"io"
 	"net/http"
 	"os"
+	"sync"
 	"time"
 
 	"github.com/cs3org/reva/v2/pkg/bytesize"
@@ -116,9 +117,11 @@ func (av Antivirus) Run() error {
 		return err
 	}
 
-	// Spawn workers that'll concurrently work the queue
+	wg := sync.WaitGroup{}
 	for i := 0; i < av.c.Workers; i++ {
-		go (func() {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
 			for e := range ch {
 				err := av.processEvent(e, natsStream)
 				if err != nil {
@@ -132,8 +135,9 @@ func (av Antivirus) Run() error {
 					}
 				}
 			}
-		})()
+		}()
 	}
+	wg.Wait()
 
 	return nil
 }