sevenhill/QSfera
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Allow for retrying postprocessing steps with an exponential backoff

Browse Source
This commit is contained in:
André Duffeck
2023-12-12 14:08:21 +01:00
parent 5482b0cacc
commit c43b9d9bc6
6 changed files with 92 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/postprocessing/README.md
+9 -1
View File
@@ -57,7 +57,15 @@ For using custom postprocessing steps you need a custom service listening to the
#### Workflow
When setting a custom postprocessing step (eg. `"customstep"`) the postprocessing service will eventually sent an event during postprocessing. The event will be of type `StartPostprocessingStep` with its field `StepToStart` set to `"customstep"`. When the custom service receives this event it can safely execute its actions, postprocessing service will wait until it has finished its work. The event contains further information (filename, executing user, size, ...) and also required tokens and urls to download the file in case byte inspection is necessary.
Once the custom service has finished its work, it should sent an event of type `PostprocessingFinished` via the configured events system. This event needs to contain a `FinishedStep` field set to `"customstep"`. It also must contain the outcome of the step, which can be one of "delete" (abort postprocessing, delete the file), "abort" (abort postprocessing, keep the file) and "continue" (continue postprocessing, this is the success case).
Once the custom service has finished its work, it should sent an event of type `PostprocessingFinished` via the configured events system. This event needs to contain a `FinishedStep` field set to `"customstep"`. It also must contain the outcome of the step, which can be one of the following:
- "delete": abort postprocessing, delete the file
- "abort": abort postprocessing, keep the file
- "retry": there was a temporary issue that will likely be resolved soon. retry the current step after some backoff duration
- "continue": continue postprocessing, this is the success case
The backoff behavior can be configured using the `POSTPROCESSING_RETRY_BACKOFF_DURATION` and `POSTPROCESSING_MAX_RETRIES` environment variables. The backoff duration is calculated using the following formula after each failure: `backoff_duration = POSTPROCESSING_RETRY_BACKOFF_DURATION * 2^(number of failures - 1)`.
Steps that still don't succeed after the maximum number of retries will be moved into the `abort` state.
See the [cs3 org](https://github.com/cs3org/reva/blob/edge/pkg/events/postprocessing.go) for up-to-date information of reserved step names and event definitions.