Initial QSfera import
This commit is contained in:
Курнат Андрей
@@ -0,0 +1,11 @@
|
||||
SHELL := bash
|
||||
NAME := storage-users
|
||||
|
||||
ifneq (, $(shell command -v go 2> /dev/null)) # suppress `command not found warnings` for non go targets in CI
|
||||
include ../../.bingo/Variables.mk
|
||||
endif
|
||||
|
||||
include ../../.make/default.mk
|
||||
include ../../.make/go.mk
|
||||
include ../../.make/release.mk
|
||||
include ../../.make/docs.mk
|
||||
@@ -0,0 +1,229 @@
|
||||
# Storage-Users
|
||||
|
||||
Purpose and description to be added
|
||||
|
||||
## Graceful Shutdown
|
||||
|
||||
You can define a graceful shutdown period for the `storage-users` service.
|
||||
|
||||
IMPORTANT: The graceful shutdown period is only applicable if the `storage-users` service runs as standalone service. It does not apply if the `storage-users` service runs as part of the single binary or as single Docker environment. To build an environment where the `storage-users` service runs as a standalone service, you must start two instances, one _without_ the `storage-users` service and one _only with_ the the `storage-users` service. Note that both instances must be able to communicate on the same network.
|
||||
|
||||
When hard-stopping КуСфера, for example with the `kill <pid>` command (SIGKILL), it is possible and likely that not all data from the decomposedfs (metadata) has been written to the storage which may result in an inconsistent decomposedfs. When gracefully shutting down КуСфера, using a command like SIGTERM, the process will no longer accept any write requests from _other_ services and will try to write the internal open requests which can take an undefined duration based on many factors. To mitigate that situation, the following things have been implemented:
|
||||
|
||||
* With the value of the environment variable `STORAGE_USERS_GRACEFUL_SHUTDOWN_TIMEOUT`, the `storage-users` service will delay its shutdown giving it time to finalize writing necessary data. This delay can be necessary if there is a lot of data to be saved and/or if storage access/thruput is slow. In such a case you would receive an error log entry informing you that not all data could be saved in time. To prevent such occurrences, you must increase the default value.
|
||||
|
||||
* If a shutdown error has been logged, the command-line maintenance tool to inspect and manipulate node metadata can help to fix the issue. Please contact support for details.
|
||||
|
||||
## CLI Commands
|
||||
|
||||
For any command listed, use `--help` to get more details and possible options and arguments.
|
||||
|
||||
To authenticate CLI commands use:
|
||||
|
||||
* `OC_SERVICE_ACCOUNT_SECRET=<acc-secret>` and
|
||||
* `OC_SERVICE_ACCOUNT_ID=<acc-id>`.
|
||||
|
||||
The `storage-users` CLI tool uses the default address to establish the connection to the `gateway` service. If the connection fails, check your custom `gateway` service `GATEWAY_GRPC_ADDR` configuration and set the same address in `storage-users` `OC_GATEWAY_GRPC_ADDR` or `STORAGE_USERS_GATEWAY_GRPC_ADDR`.
|
||||
|
||||
### Manage Unfinished Uploads
|
||||
|
||||
<!-- referencing: [oCIS FS] clean up aborted uploads https://github.com/owncloud/ocis/issues/2622 -->
|
||||
|
||||
When using КуСфера as user storage, a directory named `storage/users/uploads` can be found in the КуСфера data folder. This is an intermediate directory based on [TUS](https://tus.io) which is an open protocol for resumable uploads. Each upload consists of a _blob_ and a _blob.info_ file. Note that the term _blob_ is just a placeholder.
|
||||
|
||||
* **If an upload succeeds**, the _blob_ file will be moved to the target and the _blob.info_ file will be deleted.
|
||||
|
||||
* **In case of incomplete uploads**, the _blob_ and _blob.info_ files will continue to receive data until either the upload succeeds in time or the upload expires based on the `STORAGE_USERS_UPLOAD_EXPIRATION` variable, see the table below for details.
|
||||
|
||||
* **In case of expired uploads**, the _blob_ and _blob.info_ files will _not_ be removed automatically. Thus a lot of data can pile up over time wasting storage space.
|
||||
|
||||
* **In the rare case of a failure**, after the upload succeeded but the file was not moved to its target location, which can happen when postprocessing fails, the situation is the same as with expired uploads.
|
||||
|
||||
Example cases for expired uploads:
|
||||
|
||||
* When a user uploads a big file but the file exceeds the user-quota, the upload can't be moved to the target after it has finished. The file stays at the upload location until it is manually cleared.
|
||||
|
||||
* If the bandwidth is limited and the file to transfer can't be transferred completely before the upload expiration time is reached, the file expires and can't be processed.
|
||||
|
||||
* If the upload was technically successful, but the postprocessing step failed due to an internal error, it will not get further processed. See the procedure **Resume Post-Processing** in the `postprocessing` service documentation for details how to solve this.
|
||||
|
||||
The following commands are available to manage unfinished uploads:
|
||||
|
||||
```bash
|
||||
qsfera storage-users uploads <command>
|
||||
```
|
||||
|
||||
```plaintext
|
||||
COMMANDS:
|
||||
sessions Print a list of upload sessions
|
||||
```
|
||||
|
||||
#### Sessions command
|
||||
|
||||
The `sessions` command is the entry point for listing, restarting and cleaning unfinished uploads.
|
||||
|
||||
**IMPORTANT**
|
||||
> If not noted otherwise, commands with the `restart` option can also use the `resume` option. This changes behaviour slightly.
|
||||
>
|
||||
> * `restart`\
|
||||
> When restarting an upload, all steps for open items will be restarted, except if otherwise defined.
|
||||
> * `resume`\
|
||||
> When resuming an upload, processing will continue unfinished items from their last completed step.
|
||||
|
||||
```bash
|
||||
qsfera storage-users uploads sessions <commandoptions>
|
||||
```
|
||||
|
||||
```
|
||||
NAME:
|
||||
qsfera storage-users uploads sessions - Print a list of upload sessions
|
||||
|
||||
USAGE:
|
||||
qsfera storage-users uploads sessions [command options]
|
||||
|
||||
OPTIONS:
|
||||
--id value filter sessions by upload session id (default: unset)
|
||||
--processing filter sessions by processing status (default: unset)
|
||||
--expired filter sessions by expired status (default: unset)
|
||||
--has-virus filter sessions by virus scan result (default: unset)
|
||||
--json output as json (default: false)
|
||||
--restart send restart event for all listed sessions (default: false)
|
||||
--resume send resume event for all listed sessions (default: false)
|
||||
--clean remove uploads (default: false)
|
||||
--help, -h show help
|
||||
```
|
||||
|
||||
This will always output a list of uploads that match the criteria. See Command Examples section.
|
||||
|
||||
Some additional information on returned information:
|
||||
- `Offset` is the amount of bytes the server has already received. If `Offset` == `Size` the server has reveived all bytes of the upload.
|
||||
- `Processing` indicates if the uploaded file is currently going through postprocessing.
|
||||
- `Scan Date` and `Scan Result` indicate the scanning status. If `Scan Date` is set and `Scan Result` is empty the file is not virus infected.
|
||||
|
||||
#### Command Examples
|
||||
|
||||
Command to list ongoing upload sessions
|
||||
|
||||
```bash
|
||||
qsfera storage-users uploads sessions --expired=false --processing=false
|
||||
```
|
||||
|
||||
```plaintext
|
||||
Not expired sessions:
|
||||
+--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+---------------------------+-----------------------+
|
||||
| Space | Upload Id | Name | Offset | Size | Executant | Owner | Expires | Processing | Scan Date | Scan Result |
|
||||
+--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+---------------------------+-----------------------+
|
||||
| f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 5e387954-7313-4223-a904-bf996da6ec0b | foo.txt | 0 | 1234 | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2024-01-26T13:04:31+01:00 | false | 2024-04-24T11:24:14+02:00 | infected: virus A |
|
||||
| f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f066244d-97b2-48e7-a30d-b40fcb60cec6 | bar.txt | 0 | 4321 | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2024-01-26T13:18:47+01:00 | false | 2024-04-24T14:38:29+02:00 | |
|
||||
+--------------------------------------+--------------------------------------+---------+--------+------+--------------------------------------+--------------------------------------+---------------------------+------------+---------------------------+-----------------------+
|
||||
```
|
||||
|
||||
The sessions command can also output json
|
||||
|
||||
```bash
|
||||
qsfera storage-users uploads sessions --expired=false --processing=false --json
|
||||
```
|
||||
|
||||
```json
|
||||
{"id":"5e387954-7313-4223-a904-bf996da6ec0b","space":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","filename":"foo.txt","offset":0,"size":1234,"executant":{"idp":"https://cloud.qsfera.test","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"spaceowner":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"expires":"2024-01-26T13:04:31+01:00","processing":false, "scanDate": "2024-04-24T11:24:14+02:00", "scanResult": "infected: virus A"}
|
||||
{"id":"f066244d-97b2-48e7-a30d-b40fcb60cec6","space":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","filename":"bar.txt","offset":0,"size":4321,"executant":{"idp":"https://cloud.qsfera.test","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"spaceowner":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"expires":"2024-01-26T13:18:47+01:00","processing":false, "scanDate": "2024-04-24T14:38:29+02:00", "scanResult": ""}
|
||||
```
|
||||
|
||||
The sessions command can also clear and restart/resume uploads. The output is the same as if run without `--clean` or `--restart` flag.
|
||||
Note: It is recommended to run to command first without the `--clean` (`--processing`) flag to double check which uploads get cleaned (restarted/resumed).
|
||||
|
||||
```bash
|
||||
# cleans all expired uploads regardless of processing and virus state.
|
||||
qsfera storage-users uploads sessions --expired=true --clean
|
||||
|
||||
# resumes all uploads that are processing and are not virus infected
|
||||
qsfera storage-users uploads sessions --processing=false --has-virus=false --resume
|
||||
```
|
||||
|
||||
### Manage Trash-Bin Items
|
||||
|
||||
This command set provides commands to get an overview of trash-bin items, restore items and purge old items of `personal` spaces and `project` spaces (spaces that have been created manually). `trash-bin` commands require a `spaceID` as parameter.
|
||||
|
||||
```bash
|
||||
qsfera storage-users trash-bin <command>
|
||||
```
|
||||
|
||||
```plaintext
|
||||
COMMANDS:
|
||||
purge-expired Purge expired trash-bin items
|
||||
list Print a list of all trash-bin items of a space.
|
||||
restore-all Restore all trash-bin items for a space.
|
||||
restore Restore a trash-bin item by ID.
|
||||
```
|
||||
|
||||
#### Purge Expired
|
||||
|
||||
* Purge all expired items from the trash-bin.
|
||||
```bash
|
||||
qsfera storage-users trash-bin purge-expired
|
||||
```
|
||||
|
||||
The behaviour of the `purge-expired` command can be configured by using the following environment variables.
|
||||
|
||||
* `STORAGE_USERS_PURGE_TRASH_BIN_USER_ID`\
|
||||
Used to obtain space trash-bin information and takes the system admin user as the default which is the `OC_ADMIN_USER_ID` but can be set individually. It should be noted, that the `OC_ADMIN_USER_ID` is only assigned automatically when using the single binary deployment and must be manually assigned in all other deployments. The command only considers spaces to which the assigned user has access and delete permission.
|
||||
|
||||
* `STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE`\
|
||||
Has a default value of `720h` which equals `30 days`. This means, the command will delete all files older than `30 days`. The value is human-readable. A value of `0` is equivalent to disable and prevents the deletion of `personal space` trash-bin files.
|
||||
|
||||
* `STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE`\
|
||||
Has a default value of `720h` which equals `30 days`. This means, the command will delete all files older than `30 days`. The value is human-readable. A value of `0` is equivalent to disable and prevents the deletion of `project space` trash-bin files.
|
||||
|
||||
#### List and Restore Trash-Bins Items
|
||||
|
||||
Restoring is possible only to the original location. The personal or project `spaceID` is required for the items to be restored. To authenticate the CLI tool use:
|
||||
|
||||
```bash
|
||||
OC_SERVICE_ACCOUNT_SECRET=<acc-secret>
|
||||
OC_SERVICE_ACCOUNT_ID=<acc-id>
|
||||
```
|
||||
|
||||
The `storage-users` CLI tool uses the default address to establish the connection to the `gateway` service. If the connection fails, check the `GATEWAY_GRPC_ADDR` configuration from your `gateway` service and set the same address to the `storage-users` variable `STORAGE_USERS_GATEWAY_GRPC_ADDR` or globally with `OC_GATEWAY_GRPC_ADDR`.
|
||||
|
||||
* Export the gateway address if your configuration differs from the default
|
||||
```bash
|
||||
export STORAGE_USERS_GATEWAY_GRPC_ADDR=127.0.0.1:9142
|
||||
```
|
||||
|
||||
* Print a list of all trash-bin items of a space
|
||||
```bash
|
||||
qsfera storage-users trash-bin list [command options] ['spaceID' required]
|
||||
```
|
||||
|
||||
The restore option defines the behavior for an item to be restored, when the item name already exists in the target space. Supported options are: `skip`, `replace` and `keep-both`. The default value is `skip`.
|
||||
|
||||
When the CLI tool restores the item with the `replace` option, the existing item will be moved to a trash-bin. When the CLI tool restores the item with the `keep-both` option and the designated item already exists, the name of the restored item will be changed by adding a numeric suffix in parentheses. The variable `STORAGE_USERS_CLI_MAX_ATTEMPTS_RENAME_FILE` defines a maximum number of attempts to rename an item.
|
||||
|
||||
* Restore all trash-bin items for a space
|
||||
```bash
|
||||
qsfera storage-users trash-bin restore-all [command options] ['spaceID' required]
|
||||
```
|
||||
|
||||
* Restore a trash-bin item by ID
|
||||
```bash
|
||||
qsfera storage-users trash-bin restore [command options] ['spaceID' required] ['itemID' required]
|
||||
```
|
||||
|
||||
## Caching
|
||||
|
||||
The `storage-users` service caches stat, metadata and uuids of files and folders via the configured store in `STORAGE_USERS_FILEMETADATA_CACHE_STORE` and `STORAGE_USERS_ID_CACHE_STORE`. Possible stores are:
|
||||
- `memory`: Basic in-memory store and the default.
|
||||
- `redis-sentinel`: Stores data in a configured Redis Sentinel cluster.
|
||||
- `nats-js-kv`: Stores data using key-value-store feature of [nats jetstream](https://docs.nats.io/nats-concepts/jetstream/key-value-store)
|
||||
- `noop`: Stores nothing. Useful for testing. Not recommended in production environments.
|
||||
|
||||
Other store types may work but are not supported currently.
|
||||
|
||||
Note: The service can only be scaled if not using `memory` store and the stores are configured identically over all instances!
|
||||
|
||||
Note that if you have used one of the deprecated stores, you should reconfigure to one of the supported ones as the deprecated stores will be removed in a later version.
|
||||
|
||||
Store specific notes:
|
||||
- When using `redis-sentinel`, the Redis master to use is configured via e.g. `OC_CACHE_STORE_NODES` in the form of `<sentinel-host>:<sentinel-port>/<redis-master>` like `10.10.0.200:26379/mymaster`.
|
||||
- When using `nats-js-kv` it is recommended to set `OC_CACHE_STORE_NODES` to the same value as `OC_EVENTS_ENDPOINT`. That way the cache uses the same nats instance as the event bus.
|
||||
- When using the `nats-js-kv` store, it is possible to set `OC_CACHE_DISABLE_PERSISTENCE` to instruct nats to not persist cache data on disc.
|
||||
@@ -0,0 +1,54 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
"github.com/qsfera/server/pkg/config/configlog"
|
||||
"github.com/qsfera/server/pkg/log"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config/parser"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// Health is the entrypoint for the health command.
|
||||
func Health(cfg *config.Config) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "health",
|
||||
Short: "check health status",
|
||||
PreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
return configlog.ReturnError(parser.ParseConfig(cfg))
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
logger := log.Configure(cfg.Service.Name, cfg.Commons, cfg.LogLevel)
|
||||
|
||||
resp, err := http.Get(
|
||||
fmt.Sprintf(
|
||||
"http://%s/healthz",
|
||||
cfg.Debug.Addr,
|
||||
),
|
||||
)
|
||||
|
||||
if err != nil {
|
||||
logger.Fatal().
|
||||
Err(err).
|
||||
Msg("Failed to request health check")
|
||||
}
|
||||
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
logger.Fatal().
|
||||
Int("code", resp.StatusCode).
|
||||
Msg("Health seems to be in bad state")
|
||||
}
|
||||
|
||||
logger.Debug().
|
||||
Int("code", resp.StatusCode).
|
||||
Msg("Health got a good state")
|
||||
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/qsfera/server/pkg/clihelper"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// GetCommands provides all commands for this service
|
||||
func GetCommands(cfg *config.Config) []*cobra.Command {
|
||||
return []*cobra.Command{
|
||||
// start this service
|
||||
Server(cfg),
|
||||
|
||||
// interaction with this service
|
||||
Uploads(cfg),
|
||||
TrashBin(cfg),
|
||||
|
||||
// infos about this service
|
||||
Health(cfg),
|
||||
Version(cfg),
|
||||
}
|
||||
}
|
||||
|
||||
// Execute is the entry point for the qsfera-storage-users command.
|
||||
func Execute(cfg *config.Config) error {
|
||||
app := clihelper.DefaultApp(&cobra.Command{
|
||||
Use: "storage-users",
|
||||
Short: "Provide storage for users and projects in КуСфера",
|
||||
})
|
||||
app.AddCommand(GetCommands(cfg)...)
|
||||
app.SetArgs(os.Args[1:])
|
||||
|
||||
return app.ExecuteContext(cfg.Context)
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/signal"
|
||||
|
||||
"github.com/qsfera/server/pkg/config/configlog"
|
||||
"github.com/qsfera/server/pkg/log"
|
||||
"github.com/qsfera/server/pkg/registry"
|
||||
"github.com/qsfera/server/pkg/runner"
|
||||
"github.com/qsfera/server/pkg/tracing"
|
||||
"github.com/qsfera/server/pkg/version"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config/parser"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/event"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/revaconfig"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/server/debug"
|
||||
"github.com/opencloud-eu/reva/v2/cmd/revad/runtime"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// Server is the entry point for the server command.
|
||||
func Server(cfg *config.Config) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "server",
|
||||
Short: fmt.Sprintf("start the %s service without runtime (unsupervised mode)", cfg.Service.Name),
|
||||
PreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
return configlog.ReturnFatal(parser.ParseConfig(cfg))
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
logger := log.Configure(cfg.Service.Name, cfg.Commons, cfg.LogLevel)
|
||||
traceProvider, err := tracing.GetTraceProvider(cmd.Context(), cfg.Commons.TracesExporter, cfg.Service.Name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var cancel context.CancelFunc
|
||||
if cfg.Context == nil {
|
||||
cfg.Context, cancel = signal.NotifyContext(context.Background(), runner.StopSignals...)
|
||||
defer cancel()
|
||||
}
|
||||
ctx := cfg.Context
|
||||
|
||||
gr := runner.NewGroup()
|
||||
|
||||
{
|
||||
// run the appropriate reva servers based on the config
|
||||
rCfg := revaconfig.StorageUsersConfigFromStruct(cfg)
|
||||
if rServer := runtime.NewDrivenHTTPServerWithOptions(rCfg,
|
||||
runtime.WithLogger(&logger.Logger),
|
||||
runtime.WithRegistry(registry.GetRegistry()),
|
||||
runtime.WithTraceProvider(traceProvider),
|
||||
); rServer != nil {
|
||||
gr.Add(runner.NewRevaServiceRunner(cfg.Service.Name+".rhttp", rServer))
|
||||
}
|
||||
if rServer := runtime.NewDrivenGRPCServerWithOptions(rCfg,
|
||||
runtime.WithLogger(&logger.Logger),
|
||||
runtime.WithRegistry(registry.GetRegistry()),
|
||||
runtime.WithTraceProvider(traceProvider),
|
||||
); rServer != nil {
|
||||
gr.Add(runner.NewRevaServiceRunner(cfg.Service.Name+".rgrpc", rServer))
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
debugServer, err := debug.Server(
|
||||
debug.Logger(logger),
|
||||
debug.Context(ctx),
|
||||
debug.Config(cfg),
|
||||
)
|
||||
if err != nil {
|
||||
logger.Info().Err(err).Str("server", "debug").Msg("Failed to initialize server")
|
||||
return err
|
||||
}
|
||||
|
||||
gr.Add(runner.NewGolangHttpServerRunner("storage-users_debug", debugServer))
|
||||
}
|
||||
|
||||
grpcSvc := registry.BuildGRPCService(cfg.GRPC.Namespace+"."+cfg.Service.Name, cfg.GRPC.Protocol, cfg.GRPC.Addr, version.GetString())
|
||||
if err := registry.RegisterService(ctx, logger, grpcSvc, cfg.Debug.Addr); err != nil {
|
||||
logger.Fatal().Err(err).Msg("failed to register the grpc service")
|
||||
}
|
||||
|
||||
{
|
||||
stream, err := event.NewStream(cfg)
|
||||
if err != nil {
|
||||
logger.Fatal().Err(err).Msg("can't connect to nats")
|
||||
}
|
||||
|
||||
selector, err := pool.GatewaySelector(cfg.Reva.Address, pool.WithRegistry(registry.GetRegistry()), pool.WithTracerProvider(traceProvider))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
eventSVC, err := event.NewService(ctx, selector, stream, logger, *cfg)
|
||||
if err != nil {
|
||||
logger.Fatal().Err(err).Msg("can't create event handler")
|
||||
}
|
||||
// The event service Run() function handles the stop signal itself
|
||||
go func() {
|
||||
err := eventSVC.Run()
|
||||
if err != nil {
|
||||
logger.Fatal().Err(err).Msg("can't run event server")
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
grResults := gr.Run(ctx)
|
||||
|
||||
// return the first non-nil error found in the results
|
||||
for _, grResult := range grResults {
|
||||
if grResult.RunnerError != nil {
|
||||
return grResult.RunnerError
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,486 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/qsfera/server/pkg/config/configlog"
|
||||
zlog "github.com/qsfera/server/pkg/log"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config/parser"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/event"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/events"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/utils"
|
||||
|
||||
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
|
||||
rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1"
|
||||
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
|
||||
"github.com/mohae/deepcopy"
|
||||
"github.com/olekukonko/tablewriter"
|
||||
"github.com/olekukonko/tablewriter/tw"
|
||||
"github.com/rs/zerolog"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
const (
|
||||
SKIP = iota
|
||||
REPLACE
|
||||
KEEP_BOTH
|
||||
)
|
||||
|
||||
// TrashBin wraps trash-bin related sub-commands.
|
||||
func TrashBin(cfg *config.Config) *cobra.Command {
|
||||
trashBinCmd := &cobra.Command{
|
||||
Use: "trash-bin",
|
||||
Short: "manage trash-bin's",
|
||||
}
|
||||
|
||||
trashBinCmd.AddCommand([]*cobra.Command{
|
||||
PurgeExpiredResources(cfg),
|
||||
listTrashBinItems(cfg),
|
||||
restoreAllTrashBinItems(cfg),
|
||||
restoreTrashBinItem(cfg),
|
||||
}...)
|
||||
return trashBinCmd
|
||||
}
|
||||
|
||||
// PurgeExpiredResources cli command removes old trash-bin items.
|
||||
func PurgeExpiredResources(cfg *config.Config) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "purge-expired",
|
||||
Short: "Purge expired trash-bin items",
|
||||
PreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
return configlog.ReturnFatal(parser.ParseConfig(cfg))
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
stream, err := event.NewStream(cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := events.Publish(cmd.Context(), stream, event.PurgeTrashBin{ExecutionTime: time.Now()}); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// go-micro nats implementation uses async publishing,
|
||||
// therefore we need to manually wait.
|
||||
//
|
||||
// FIXME: upstream pr
|
||||
//
|
||||
// https://github.com/go-micro/plugins/blob/3e77393890683be4bacfb613bc5751867d584692/v4/events/natsjs/nats.go#L115
|
||||
time.Sleep(5 * time.Second)
|
||||
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func listTrashBinItems(cfg *config.Config) *cobra.Command {
|
||||
listTrashBinItemsCmd := &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Print a list of all trash-bin items of a space.",
|
||||
// TODO: n might need to equal 2 not sure.
|
||||
Args: cobra.ExactArgs(1),
|
||||
PreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
return configlog.ReturnFatal(parser.ParseConfig(cfg))
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
verbose, _ := cmd.Flags().GetBool("verbose")
|
||||
log := cliLogger(verbose)
|
||||
var spaceID string
|
||||
if len(args) > 0 {
|
||||
spaceID = args[0]
|
||||
}
|
||||
if spaceID == "" {
|
||||
_ = cmd.Help()
|
||||
return fmt.Errorf("spaceID is requiered")
|
||||
}
|
||||
log.Info().Msgf("Getting trash-bin items for spaceID: '%s' ...", spaceID)
|
||||
|
||||
ref, err := storagespace.ParseReference(spaceID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
client, err := pool.GetGatewayServiceClient(cfg.RevaGatewayGRPCAddr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error selecting gateway client %w", err)
|
||||
}
|
||||
ctx, err := utils.GetServiceUserContext(cfg.ServiceAccount.ServiceAccountID, client, cfg.ServiceAccount.ServiceAccountSecret)
|
||||
if err != nil {
|
||||
return fmt.Errorf("could not get service user context %w", err)
|
||||
}
|
||||
res, err := listRecycle(ctx, client, ref)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
table := itemsTable(len(res.GetRecycleItems()))
|
||||
for _, item := range res.GetRecycleItems() {
|
||||
table.Append([]string{item.GetKey(), item.GetRef().GetPath(), itemType(item.GetType()), utils.TSToTime(item.GetDeletionTime()).UTC().Format(time.RFC3339)})
|
||||
}
|
||||
table.Render()
|
||||
fmt.Println("Use an itemID to restore an item.")
|
||||
return nil
|
||||
},
|
||||
}
|
||||
listTrashBinItemsCmd.Flags().BoolP(
|
||||
"verbose",
|
||||
"v",
|
||||
false,
|
||||
"Get more verbose output",
|
||||
)
|
||||
return listTrashBinItemsCmd
|
||||
}
|
||||
|
||||
func restoreAllTrashBinItems(cfg *config.Config) *cobra.Command {
|
||||
var overwriteOption int
|
||||
restoreAllTrashBinItemsCmd := &cobra.Command{
|
||||
Use: "restore-all",
|
||||
Short: "Restore all trash-bin items for a space.",
|
||||
// TODO: not sure this could also be 2
|
||||
Args: cobra.ExactArgs(1),
|
||||
PreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
return configlog.ReturnFatal(parser.ParseConfig(cfg))
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
verbose, _ := cmd.Flags().GetBool("verbose")
|
||||
log := cliLogger(verbose)
|
||||
var spaceID string
|
||||
if len(args) > 0 {
|
||||
spaceID = args[0]
|
||||
}
|
||||
if spaceID == "" {
|
||||
_ = cmd.Help()
|
||||
return fmt.Errorf("spaceID is requiered")
|
||||
}
|
||||
option, _ := cmd.Flags().GetString("option")
|
||||
switch option {
|
||||
case "skip":
|
||||
overwriteOption = SKIP
|
||||
case "replace":
|
||||
overwriteOption = REPLACE
|
||||
case "keep-both":
|
||||
overwriteOption = KEEP_BOTH
|
||||
default:
|
||||
_ = cmd.Help()
|
||||
return fmt.Errorf("option flag '%s' is invalid", option)
|
||||
}
|
||||
log.Info().Msgf("Restoring trash-bin items for spaceID: '%s' ...", spaceID)
|
||||
|
||||
ref, err := storagespace.ParseReference(spaceID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
client, err := pool.GetGatewayServiceClient(cfg.RevaGatewayGRPCAddr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error selecting gateway client %w", err)
|
||||
}
|
||||
ctx, err := utils.GetServiceUserContext(cfg.ServiceAccount.ServiceAccountID, client, cfg.ServiceAccount.ServiceAccountSecret)
|
||||
if err != nil {
|
||||
return fmt.Errorf("could not get service user context %w", err)
|
||||
}
|
||||
res, err := listRecycle(ctx, client, ref)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
applyYesFlag, _ := cmd.Flags().GetBool("yes")
|
||||
if !applyYesFlag {
|
||||
for {
|
||||
fmt.Printf("Found %d items that could be restored, continue (Y/n), show the items list (s): ", len(res.GetRecycleItems()))
|
||||
var i string
|
||||
_, err := fmt.Scanf("%s", &i)
|
||||
if err != nil {
|
||||
log.Err(err).Send()
|
||||
continue
|
||||
}
|
||||
if strings.ToLower(i) == "y" {
|
||||
break
|
||||
} else if strings.ToLower(i) == "n" {
|
||||
return nil
|
||||
} else if strings.ToLower(i) == "s" {
|
||||
table := itemsTable(len(res.GetRecycleItems()))
|
||||
for _, item := range res.GetRecycleItems() {
|
||||
table.Append([]string{item.GetKey(), item.GetRef().GetPath(), itemType(item.GetType()), utils.TSToTime(item.GetDeletionTime()).UTC().Format(time.RFC3339)})
|
||||
}
|
||||
table.Render()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
log.Info().Msgf("Run restoring-all with option=%s", option)
|
||||
for _, item := range res.GetRecycleItems() {
|
||||
log.Info().Msgf("restoring itemID: '%s', path: '%s', type: '%s'", item.GetKey(), item.GetRef().GetPath(), itemType(item.GetType()))
|
||||
dstRes, err := restore(ctx, client, ref, item, overwriteOption, cfg.CliMaxAttemptsRenameFile, log)
|
||||
if err != nil {
|
||||
log.Err(err).Msg("trash-bin item restoring error")
|
||||
continue
|
||||
}
|
||||
fmt.Printf("itemID: '%s', path: '%s', restored as '%s'\n", item.GetKey(), item.GetRef().GetPath(), dstRes.GetPath())
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
restoreAllTrashBinItemsCmd.Flags().BoolP(
|
||||
"verbose",
|
||||
"v",
|
||||
false,
|
||||
"Get more verbose output",
|
||||
)
|
||||
restoreAllTrashBinItemsCmd.Flags().StringP(
|
||||
"option",
|
||||
"o",
|
||||
"skip",
|
||||
"The restore option defines the behavior for a file to be restored, where the file name already already exists in the target space. Supported values are: 'skip', 'replace' and 'keep-both'. The default value is 'skip' overwriting an existing file.",
|
||||
)
|
||||
restoreAllTrashBinItemsCmd.Flags().BoolP(
|
||||
"yes",
|
||||
"y",
|
||||
false,
|
||||
"Automatic yes to prompts. Assume 'yes' as answer to all prompts and run non-interactively.",
|
||||
)
|
||||
|
||||
return restoreAllTrashBinItemsCmd
|
||||
}
|
||||
|
||||
func restoreTrashBinItem(cfg *config.Config) *cobra.Command {
|
||||
var overwriteOption int
|
||||
restoreTrashBinItemCmd := &cobra.Command{
|
||||
Use: "restore",
|
||||
Short: "Restore a trash-bin item by ID.",
|
||||
Args: cobra.ExactArgs(2),
|
||||
PreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
return configlog.ReturnFatal(parser.ParseConfig(cfg))
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
verbose, _ := cmd.Flags().GetBool("verbose")
|
||||
log := cliLogger(verbose)
|
||||
var spaceID, itemID string
|
||||
spaceID = args[0]
|
||||
itemID = args[1]
|
||||
if spaceID == "" {
|
||||
_ = cmd.Help()
|
||||
return fmt.Errorf("spaceID is requered")
|
||||
}
|
||||
if itemID == "" {
|
||||
_ = cmd.Help()
|
||||
return fmt.Errorf("itemID is requered")
|
||||
}
|
||||
option, _ := cmd.Flags().GetString("option")
|
||||
switch option {
|
||||
case "skip":
|
||||
overwriteOption = SKIP
|
||||
case "replace":
|
||||
overwriteOption = REPLACE
|
||||
case "keep-both":
|
||||
overwriteOption = KEEP_BOTH
|
||||
default:
|
||||
_ = cmd.Help()
|
||||
return fmt.Errorf("option flag '%s' is invalid", option)
|
||||
}
|
||||
log.Info().Msgf("Restoring trash-bin item for spaceID: '%s' itemID: '%s' ...", spaceID, itemID)
|
||||
|
||||
ref, err := storagespace.ParseReference(spaceID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
client, err := pool.GetGatewayServiceClient(cfg.RevaGatewayGRPCAddr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error selecting gateway client %w", err)
|
||||
}
|
||||
ctx, err := utils.GetServiceUserContext(cfg.ServiceAccount.ServiceAccountID, client, cfg.ServiceAccount.ServiceAccountSecret)
|
||||
if err != nil {
|
||||
return fmt.Errorf("could not get service user context %w", err)
|
||||
}
|
||||
res, err := listRecycle(ctx, client, ref)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var found bool
|
||||
var itemRef *provider.RecycleItem
|
||||
for _, item := range res.GetRecycleItems() {
|
||||
if item.GetKey() == itemID {
|
||||
itemRef = item
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("itemID '%s' not found", itemID)
|
||||
}
|
||||
log.Info().Msgf("Run restoring with option=%s", option)
|
||||
log.Info().Msgf("restoring itemID: '%s', path: '%s', type: '%s", itemRef.GetKey(), itemRef.GetRef().GetPath(), itemType(itemRef.GetType()))
|
||||
dstRes, err := restore(ctx, client, ref, itemRef, overwriteOption, cfg.CliMaxAttemptsRenameFile, log)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("itemID: '%s', path: '%s', restored as '%s'\n", itemRef.GetKey(), itemRef.GetRef().GetPath(), dstRes.GetPath())
|
||||
return nil
|
||||
},
|
||||
}
|
||||
restoreTrashBinItemCmd.Flags().BoolP(
|
||||
"verbose",
|
||||
"v",
|
||||
false,
|
||||
"Get more verbose output",
|
||||
)
|
||||
restoreTrashBinItemCmd.Flags().StringP(
|
||||
"option",
|
||||
"o",
|
||||
"skip",
|
||||
"The restore option defines the behavior for a file to be restored, where the file name already already exists in the target space. Supported values are: 'skip', 'replace' and 'keep-both'. The default value is 'skip' overwriting an existing file.",
|
||||
)
|
||||
return restoreTrashBinItemCmd
|
||||
}
|
||||
|
||||
func listRecycle(ctx context.Context, client gateway.GatewayAPIClient, ref provider.Reference) (*provider.ListRecycleResponse, error) {
|
||||
_retrievingErrorMsg := "trash-bin items retrieving error"
|
||||
res, err := client.ListRecycle(ctx, &provider.ListRecycleRequest{Ref: &ref, Key: ""})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s %w", _retrievingErrorMsg, err)
|
||||
}
|
||||
if res.Status.Code != rpc.Code_CODE_OK {
|
||||
return nil, fmt.Errorf("%s %s", _retrievingErrorMsg, res.Status.Code)
|
||||
}
|
||||
if len(res.GetRecycleItems()) == 0 {
|
||||
fmt.Errorf("The trash-bin is empty. Nothing to restore")
|
||||
os.Exit(0)
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
func restore(ctx context.Context, client gateway.GatewayAPIClient, ref provider.Reference, item *provider.RecycleItem, overwriteOption int, maxRenameAttempt int, log zlog.Logger) (*provider.Reference, error) {
|
||||
dst, _ := deepcopy.Copy(ref).(provider.Reference)
|
||||
dst.Path = utils.MakeRelativePath(item.GetRef().GetPath())
|
||||
// Restore request
|
||||
req := &provider.RestoreRecycleItemRequest{
|
||||
Ref: &ref,
|
||||
Key: path.Join(item.GetKey(), "/"),
|
||||
RestoreRef: &dst,
|
||||
}
|
||||
|
||||
exists, dstStatRes, err := isDestinationExists(ctx, client, dst)
|
||||
if err != nil {
|
||||
return &dst, err
|
||||
}
|
||||
|
||||
if exists {
|
||||
log.Info().Msgf("destination '%s' exists.", dstStatRes.GetInfo().GetPath())
|
||||
switch overwriteOption {
|
||||
case SKIP:
|
||||
return &dst, nil
|
||||
case REPLACE:
|
||||
// delete existing tree
|
||||
delReq := &provider.DeleteRequest{Ref: &dst}
|
||||
delRes, err := client.Delete(ctx, delReq)
|
||||
if err != nil {
|
||||
return &dst, fmt.Errorf("error sending grpc delete request %w", err)
|
||||
}
|
||||
if delRes.Status.Code != rpc.Code_CODE_OK && delRes.Status.Code != rpc.Code_CODE_NOT_FOUND {
|
||||
return &dst, fmt.Errorf("deleting error %w", err)
|
||||
}
|
||||
case KEEP_BOTH:
|
||||
// modify the file name
|
||||
req.RestoreRef, err = resolveDestination(ctx, client, dst, maxRenameAttempt)
|
||||
if err != nil {
|
||||
return &dst, err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
res, err := client.RestoreRecycleItem(ctx, req)
|
||||
if err != nil {
|
||||
return req.RestoreRef, fmt.Errorf("restoring error %w", err)
|
||||
}
|
||||
if res.Status.Code != rpc.Code_CODE_OK {
|
||||
return req.RestoreRef, fmt.Errorf("can not restore %s", res.Status.Code)
|
||||
}
|
||||
return req.RestoreRef, nil
|
||||
}
|
||||
|
||||
func resolveDestination(ctx context.Context, client gateway.GatewayAPIClient, dstRef provider.Reference, maxRenameAttempt int) (*provider.Reference, error) {
|
||||
dst := dstRef
|
||||
if maxRenameAttempt < 100 {
|
||||
maxRenameAttempt = 100
|
||||
}
|
||||
for i := 1; i < maxRenameAttempt; i++ {
|
||||
dst.Path = modifyFilename(dstRef.Path, i)
|
||||
exists, _, err := isDestinationExists(ctx, client, dst)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if exists {
|
||||
continue
|
||||
}
|
||||
return &dst, nil
|
||||
}
|
||||
return nil, fmt.Errorf("too many attempts to resolve the destination")
|
||||
}
|
||||
|
||||
func isDestinationExists(ctx context.Context, client gateway.GatewayAPIClient, dst provider.Reference) (bool, *provider.StatResponse, error) {
|
||||
dstStatReq := &provider.StatRequest{Ref: &dst}
|
||||
dstStatRes, err := client.Stat(ctx, dstStatReq)
|
||||
if err != nil {
|
||||
return false, nil, fmt.Errorf("error sending grpc stat request %w", err)
|
||||
}
|
||||
if dstStatRes.GetStatus().GetCode() == rpc.Code_CODE_OK {
|
||||
return true, dstStatRes, nil
|
||||
}
|
||||
if dstStatRes.GetStatus().GetCode() == rpc.Code_CODE_NOT_FOUND {
|
||||
return false, dstStatRes, nil
|
||||
}
|
||||
return false, dstStatRes, fmt.Errorf("stat request failed %s", dstStatRes.GetStatus())
|
||||
}
|
||||
|
||||
// modify the file name like UI do
|
||||
func modifyFilename(filename string, mod int) string {
|
||||
var extension string
|
||||
var found bool
|
||||
expected := []string{".tar.gz", ".tar.bz", ".tar.bz2"}
|
||||
for _, s := range expected {
|
||||
var prefix string
|
||||
prefix, found = strings.CutSuffix(strings.ToLower(filename), s)
|
||||
if found {
|
||||
extension = strings.TrimPrefix(filename, prefix)
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
extension = filepath.Ext(filename)
|
||||
}
|
||||
name := filename[0 : len(filename)-len(extension)]
|
||||
return fmt.Sprintf("%s (%d)%s", name, mod, extension)
|
||||
}
|
||||
|
||||
func itemType(it provider.ResourceType) string {
|
||||
var itemType = "file"
|
||||
if it == provider.ResourceType_RESOURCE_TYPE_CONTAINER {
|
||||
itemType = "folder"
|
||||
}
|
||||
return itemType
|
||||
}
|
||||
|
||||
func itemsTable(total int) *tablewriter.Table {
|
||||
table := tablewriter.NewTable(os.Stdout, tablewriter.WithHeaderAutoFormat(tw.Off))
|
||||
table.Header([]string{"itemID", "path", "type", "delete at"})
|
||||
table.Footer([]string{"", "", "", "total count: " + strconv.Itoa(total)})
|
||||
return table
|
||||
}
|
||||
|
||||
func cliLogger(verbose bool) zlog.Logger {
|
||||
logLvl := zerolog.ErrorLevel
|
||||
if verbose {
|
||||
logLvl = zerolog.InfoLevel
|
||||
}
|
||||
zerolog.SetGlobalLevel(zerolog.TraceLevel)
|
||||
output := zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: time.RFC3339, NoColor: true}
|
||||
return zlog.Logger{zerolog.New(output).With().Timestamp().Logger().Level(logLvl)}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func Test_modifyFilename(t *testing.T) {
|
||||
type args struct {
|
||||
filename string
|
||||
mod int
|
||||
}
|
||||
tests := []struct {
|
||||
name string
|
||||
args args
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "file",
|
||||
args: args{filename: "file.txt", mod: 1},
|
||||
want: "file (1).txt",
|
||||
},
|
||||
{
|
||||
name: "file with path",
|
||||
args: args{filename: "./file.txt", mod: 1},
|
||||
want: "./file (1).txt",
|
||||
},
|
||||
{
|
||||
name: "file with path 2",
|
||||
args: args{filename: "./subdir/file.tar.gz", mod: 99},
|
||||
want: "./subdir/file (99).tar.gz",
|
||||
},
|
||||
{
|
||||
name: "file with path 3",
|
||||
args: args{filename: "./sub dir/new file.tar.gz", mod: 99},
|
||||
want: "./sub dir/new file (99).tar.gz",
|
||||
},
|
||||
{
|
||||
name: "file without ext",
|
||||
args: args{filename: "./subdir/file", mod: 2},
|
||||
want: "./subdir/file (2)",
|
||||
},
|
||||
{
|
||||
name: "file without ext 2",
|
||||
args: args{filename: "./subdir/file 1", mod: 2},
|
||||
want: "./subdir/file 1 (2)",
|
||||
},
|
||||
{
|
||||
name: "file with emoji",
|
||||
args: args{filename: "./subdir/file 🙂.tar.gz", mod: 3},
|
||||
want: "./subdir/file 🙂 (3).tar.gz",
|
||||
},
|
||||
{
|
||||
name: "file with emoji 2",
|
||||
args: args{filename: "./subdir/file 🙂", mod: 2},
|
||||
want: "./subdir/file 🙂 (2)",
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := modifyFilename(tt.args.filename, tt.args.mod); got != tt.want {
|
||||
t.Errorf("modifyFilename() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/olekukonko/tablewriter"
|
||||
"github.com/olekukonko/tablewriter/tw"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
userpb "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1"
|
||||
"github.com/qsfera/server/pkg/config/configlog"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config/parser"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/event"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/revaconfig"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/events"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/storage"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/storage/fs/registry"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/utils"
|
||||
)
|
||||
|
||||
// Session contains the information of an upload session
|
||||
type Session struct {
|
||||
ID string `json:"id"`
|
||||
Space string `json:"space"`
|
||||
Filename string `json:"filename"`
|
||||
Offset int64 `json:"offset"`
|
||||
Size int64 `json:"size"`
|
||||
Executant userpb.UserId `json:"executant"`
|
||||
SpaceOwner *userpb.UserId `json:"spaceowner,omitempty"`
|
||||
Expires time.Time `json:"expires"`
|
||||
Processing bool `json:"processing"`
|
||||
ScanDate time.Time `json:"virus_scan_date"`
|
||||
ScanResult string `json:"virus_scan_result"`
|
||||
}
|
||||
|
||||
// Uploads is the entry point for the uploads command
|
||||
func Uploads(cfg *config.Config) *cobra.Command {
|
||||
uploadsCmd := &cobra.Command{
|
||||
Use: "uploads",
|
||||
Short: "manage unfinished uploads",
|
||||
}
|
||||
uploadsCmd.AddCommand([]*cobra.Command{
|
||||
ListUploadSessions(cfg),
|
||||
}...)
|
||||
|
||||
return uploadsCmd
|
||||
|
||||
}
|
||||
|
||||
// ListUploadSessions prints a list of upload sessiens
|
||||
func ListUploadSessions(cfg *config.Config) *cobra.Command {
|
||||
listUploadSessionsCmd := &cobra.Command{
|
||||
Use: "sessions",
|
||||
Short: "Print a list of upload sessions",
|
||||
PreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
return configlog.ReturnFatal(parser.ParseConfig(cfg))
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var err error
|
||||
f, ok := registry.NewFuncs[cfg.Driver]
|
||||
if !ok {
|
||||
fmt.Fprintf(os.Stderr, "Unknown filesystem driver '%s'\n", cfg.Driver)
|
||||
os.Exit(1)
|
||||
}
|
||||
drivers := revaconfig.StorageProviderDrivers(cfg)
|
||||
var fsStream events.Stream
|
||||
if cfg.Driver == "posix" {
|
||||
// We need to init the posix driver with 'scanfs' disabled
|
||||
drivers["posix"] = revaconfig.Posix(cfg, false, false)
|
||||
// Also posix refuses to start without an events stream
|
||||
fsStream, err = event.NewStream(cfg)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to create event stream for posix driver: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
fs, err := f(drivers[cfg.Driver].(map[string]any), fsStream, nil)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to initialize filesystem driver '%s'\n", cfg.Driver)
|
||||
return err
|
||||
}
|
||||
|
||||
managingFS, ok := fs.(storage.UploadSessionLister)
|
||||
if !ok {
|
||||
fmt.Fprintf(os.Stderr, "'%s' storage does not support listing upload sessions\n", cfg.Driver)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
restart, _ := cmd.Flags().GetBool("restart")
|
||||
resume, _ := cmd.Flags().GetBool("resume")
|
||||
clean, _ := cmd.Flags().GetBool("clean")
|
||||
renderJson, _ := cmd.Flags().GetBool("json")
|
||||
|
||||
var stream events.Stream
|
||||
if restart || resume {
|
||||
stream, err = event.NewStream(cfg)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to create event stream: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
filter := buildFilter(cmd)
|
||||
uploads, err := managingFS.ListUploadSessions(cmd.Context(), filter)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var (
|
||||
table *tablewriter.Table
|
||||
raw []Session
|
||||
)
|
||||
|
||||
if !renderJson {
|
||||
fmt.Println(buildInfo(filter))
|
||||
|
||||
table = tablewriter.NewTable(os.Stdout, tablewriter.WithHeaderAutoFormat(tw.Off))
|
||||
table.Header([]string{"Space", "Upload Id", "Name", "Offset", "Size", "Executant", "Owner", "Expires", "Processing", "Scan Date", "Scan Result"})
|
||||
}
|
||||
|
||||
for _, u := range uploads {
|
||||
ref := u.Reference()
|
||||
sr, sd := u.ScanData()
|
||||
|
||||
session := Session{
|
||||
Space: ref.GetResourceId().GetSpaceId(),
|
||||
ID: u.ID(),
|
||||
Filename: u.Filename(),
|
||||
Offset: u.Offset(),
|
||||
Size: u.Size(),
|
||||
Executant: u.Executant(),
|
||||
SpaceOwner: u.SpaceOwner(),
|
||||
Expires: u.Expires(),
|
||||
Processing: u.IsProcessing(),
|
||||
ScanDate: sd,
|
||||
ScanResult: sr,
|
||||
}
|
||||
|
||||
if renderJson {
|
||||
raw = append(raw, session)
|
||||
} else {
|
||||
table.Append([]string{
|
||||
session.Space,
|
||||
session.ID,
|
||||
session.Filename,
|
||||
strconv.FormatInt(session.Offset, 10),
|
||||
strconv.FormatInt(session.Size, 10),
|
||||
session.Executant.OpaqueId,
|
||||
session.SpaceOwner.GetOpaqueId(),
|
||||
session.Expires.Format(time.RFC3339),
|
||||
strconv.FormatBool(session.Processing),
|
||||
session.ScanDate.Format(time.RFC3339),
|
||||
session.ScanResult,
|
||||
})
|
||||
}
|
||||
|
||||
switch {
|
||||
case restart:
|
||||
if err := events.Publish(context.Background(), stream, events.RestartPostprocessing{
|
||||
UploadID: u.ID(),
|
||||
Timestamp: utils.TSNow(),
|
||||
}); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to send restart event for upload session '%s'\n", u.ID())
|
||||
// if publishing fails there is no need to try publishing other events - they will fail too.
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
case resume:
|
||||
if err := events.Publish(context.Background(), stream, events.ResumePostprocessing{
|
||||
UploadID: u.ID(),
|
||||
Timestamp: utils.TSNow(),
|
||||
}); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to send resume event for upload session '%s'\n", u.ID())
|
||||
// if publishing fails there is no need to try publishing other events - they will fail too.
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
case clean:
|
||||
if err := u.Purge(cmd.Context()); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to clean upload session '%s'\n", u.ID())
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
if !renderJson {
|
||||
table.Render()
|
||||
return nil
|
||||
}
|
||||
|
||||
j, err := json.Marshal(raw)
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
return err
|
||||
}
|
||||
fmt.Println(string(j))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
listUploadSessionsCmd.Flags().String("id", "", "filter sessions by upload session id")
|
||||
listUploadSessionsCmd.Flags().Bool("processing", false, "filter sessions by processing status")
|
||||
listUploadSessionsCmd.Flags().Bool("expired", false, "filter sessions by expired status")
|
||||
listUploadSessionsCmd.Flags().Bool("has-virus", false, "filter sessions by virus scan result")
|
||||
listUploadSessionsCmd.Flags().Bool("json", false, "output as json")
|
||||
listUploadSessionsCmd.Flags().Bool("restart", false, "send restart event for all listed sessions. Only one of resume/restart/clean can be set.")
|
||||
listUploadSessionsCmd.Flags().Bool("resume", false, "send resume event for all listed sessions. Only one of resume/restart/clean can be set.")
|
||||
listUploadSessionsCmd.Flags().Bool("clean", false, "remove uploads for all listed sessions. Only one of resume/restart/clean can be set.")
|
||||
return listUploadSessionsCmd
|
||||
}
|
||||
|
||||
func buildFilter(cmd *cobra.Command) storage.UploadSessionFilter {
|
||||
filter := storage.UploadSessionFilter{}
|
||||
if cmd.Flag("processing").Changed {
|
||||
processingValue, _ := cmd.Flags().GetBool("processing")
|
||||
filter.Processing = &processingValue
|
||||
}
|
||||
if cmd.Flag("expired").Changed {
|
||||
expiredValue, _ := cmd.Flags().GetBool("expired")
|
||||
filter.Expired = &expiredValue
|
||||
}
|
||||
if cmd.Flag("has-virus").Changed {
|
||||
infectedValue, _ := cmd.Flags().GetBool("has-virus")
|
||||
filter.HasVirus = &infectedValue
|
||||
}
|
||||
if cmd.Flag("id").Changed {
|
||||
idValue, _ := cmd.Flags().GetString("id")
|
||||
if idValue != "" {
|
||||
filter.ID = &idValue
|
||||
}
|
||||
}
|
||||
return filter
|
||||
}
|
||||
|
||||
func buildInfo(filter storage.UploadSessionFilter) string {
|
||||
var b strings.Builder
|
||||
if filter.Processing != nil {
|
||||
if !*filter.Processing {
|
||||
b.WriteString("Not ")
|
||||
}
|
||||
if b.Len() == 0 {
|
||||
b.WriteString("Processing")
|
||||
} else {
|
||||
b.WriteString("processing")
|
||||
}
|
||||
}
|
||||
|
||||
if filter.Expired != nil {
|
||||
if b.Len() != 0 {
|
||||
b.WriteString(", ")
|
||||
}
|
||||
if !*filter.Expired {
|
||||
if b.Len() == 0 {
|
||||
b.WriteString("Not ")
|
||||
} else {
|
||||
b.WriteString("not ")
|
||||
}
|
||||
}
|
||||
if b.Len() == 0 {
|
||||
b.WriteString("Expired")
|
||||
} else {
|
||||
b.WriteString("expired")
|
||||
}
|
||||
}
|
||||
|
||||
if filter.HasVirus != nil {
|
||||
if b.Len() != 0 {
|
||||
b.WriteString(", ")
|
||||
}
|
||||
if !*filter.HasVirus {
|
||||
if b.Len() == 0 {
|
||||
b.WriteString("Not ")
|
||||
} else {
|
||||
b.WriteString("not ")
|
||||
}
|
||||
}
|
||||
if b.Len() == 0 {
|
||||
b.WriteString("Virusinfected")
|
||||
} else {
|
||||
b.WriteString("virusinfected")
|
||||
}
|
||||
}
|
||||
|
||||
if b.Len() == 0 {
|
||||
b.WriteString("Session")
|
||||
} else {
|
||||
b.WriteString(" session")
|
||||
}
|
||||
|
||||
if filter.ID != nil {
|
||||
b.WriteString(" with id '" + *filter.ID + "'")
|
||||
} else {
|
||||
// to make `session` plural
|
||||
b.WriteString("s")
|
||||
}
|
||||
|
||||
b.WriteString(":")
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/opencloud-eu/reva/v2/pkg/storage"
|
||||
"github.com/test-go/testify/require"
|
||||
)
|
||||
|
||||
func TestBuildInfo(t *testing.T) {
|
||||
testCases := []struct {
|
||||
alias string
|
||||
filter storage.UploadSessionFilter
|
||||
expectedInfo string
|
||||
}{
|
||||
{
|
||||
alias: "empty filter",
|
||||
filter: storage.UploadSessionFilter{},
|
||||
expectedInfo: "Sessions:",
|
||||
},
|
||||
{
|
||||
alias: "processing",
|
||||
filter: storage.UploadSessionFilter{Processing: boolPtr(true)},
|
||||
expectedInfo: "Processing sessions:",
|
||||
},
|
||||
{
|
||||
alias: "processing and not expired",
|
||||
filter: storage.UploadSessionFilter{Processing: boolPtr(true), Expired: boolPtr(false)},
|
||||
expectedInfo: "Processing, not expired sessions:",
|
||||
},
|
||||
{
|
||||
alias: "processing and expired",
|
||||
filter: storage.UploadSessionFilter{Processing: boolPtr(true), Expired: boolPtr(true)},
|
||||
expectedInfo: "Processing, expired sessions:",
|
||||
},
|
||||
{
|
||||
alias: "with id",
|
||||
filter: storage.UploadSessionFilter{ID: strPtr("123")},
|
||||
expectedInfo: "Session with id '123':",
|
||||
},
|
||||
{
|
||||
alias: "processing, not expired and not virus infected",
|
||||
filter: storage.UploadSessionFilter{Processing: boolPtr(true), Expired: boolPtr(false), HasVirus: boolPtr(false)},
|
||||
expectedInfo: "Processing, not expired, not virusinfected sessions:",
|
||||
},
|
||||
{
|
||||
alias: "not virusinfected",
|
||||
filter: storage.UploadSessionFilter{HasVirus: boolPtr(false)},
|
||||
expectedInfo: "Not virusinfected sessions:",
|
||||
},
|
||||
{
|
||||
alias: "expired and virusinfected",
|
||||
filter: storage.UploadSessionFilter{Expired: boolPtr(true), HasVirus: boolPtr(true)},
|
||||
expectedInfo: "Expired, virusinfected sessions:",
|
||||
},
|
||||
{
|
||||
alias: "expired and not virus infected",
|
||||
filter: storage.UploadSessionFilter{Expired: boolPtr(true), HasVirus: boolPtr(false)},
|
||||
expectedInfo: "Expired, not virusinfected sessions:",
|
||||
},
|
||||
{
|
||||
alias: "processing, not expired, virus infected and with id (note: this makes no sense)",
|
||||
filter: storage.UploadSessionFilter{Processing: boolPtr(true), Expired: boolPtr(false), HasVirus: boolPtr(true), ID: strPtr("123")},
|
||||
expectedInfo: "Processing, not expired, virusinfected session with id '123':",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range testCases {
|
||||
alias := tc.alias
|
||||
filter := tc.filter
|
||||
expectedInfo := tc.expectedInfo
|
||||
|
||||
t.Run(alias, func(t *testing.T) {
|
||||
require.Equal(t, expectedInfo, buildInfo(filter))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func boolPtr(b bool) *bool {
|
||||
return &b
|
||||
}
|
||||
|
||||
func strPtr(s string) *string {
|
||||
return &s
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/qsfera/server/pkg/registry"
|
||||
"github.com/qsfera/server/pkg/version"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
|
||||
"github.com/olekukonko/tablewriter"
|
||||
"github.com/olekukonko/tablewriter/tw"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// Version prints the service versions of all running instances.
|
||||
func Version(cfg *config.Config) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "version",
|
||||
Short: "print the version of this binary and the running service instances",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
fmt.Println("Version: " + version.GetString())
|
||||
fmt.Printf("Compiled: %s\n", version.Compiled())
|
||||
fmt.Println("")
|
||||
|
||||
reg := registry.GetRegistry()
|
||||
services, err := reg.GetService(cfg.GRPC.Namespace + "." + cfg.Service.Name)
|
||||
if err != nil {
|
||||
fmt.Println(fmt.Errorf("could not get %s services from the registry: %v", cfg.Service.Name, err))
|
||||
return err
|
||||
}
|
||||
|
||||
if len(services) == 0 {
|
||||
fmt.Println("No running " + cfg.Service.Name + " service found.")
|
||||
return nil
|
||||
}
|
||||
|
||||
table := tablewriter.NewTable(os.Stdout, tablewriter.WithHeaderAutoFormat(tw.Off))
|
||||
table.Header([]string{"Version", "Address", "Id"})
|
||||
for _, s := range services {
|
||||
for _, n := range s.Nodes {
|
||||
table.Append([]string{s.Version, n.Address, n.Id})
|
||||
}
|
||||
}
|
||||
table.Render()
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,321 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/qsfera/server/pkg/shared"
|
||||
)
|
||||
|
||||
// Config is the configuration for the storage-users service
|
||||
type Config struct {
|
||||
Commons *shared.Commons `yaml:"-"` // don't use this directly as configuration for a service
|
||||
Service Service `yaml:"-"`
|
||||
LogLevel string `yaml:"loglevel" env:"OC_LOG_LEVEL;STORAGE_USERS_LOG_LEVEL" desc:"The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'." introductionVersion:"1.0.0"`
|
||||
Debug Debug `yaml:"debug"`
|
||||
|
||||
GRPC GRPCConfig `yaml:"grpc"`
|
||||
HTTP HTTPConfig `yaml:"http"`
|
||||
|
||||
TokenManager *TokenManager `yaml:"token_manager"`
|
||||
Reva *shared.Reva `yaml:"reva"`
|
||||
|
||||
SkipUserGroupsInToken bool `yaml:"skip_user_groups_in_token" env:"STORAGE_USERS_SKIP_USER_GROUPS_IN_TOKEN" desc:"Disables the loading of user's group memberships from the reva access token." introductionVersion:"1.0.0"`
|
||||
GracefulShutdownTimeout int `yaml:"graceful_shutdown_timeout" env:"STORAGE_USERS_GRACEFUL_SHUTDOWN_TIMEOUT" desc:"The number of seconds to wait for the 'storage-users' service to shutdown cleanly before exiting with an error that gets logged. Note: This setting is only applicable when running the 'storage-users' service as a standalone service. See the text description for more details." introductionVersion:"1.0.0"`
|
||||
|
||||
Driver string `yaml:"driver" env:"STORAGE_USERS_DRIVER" desc:"The storage driver which should be used by the service. Defaults to 'posix'. Supported values are: 'posix', 'decomposed', 'decomposeds3' and 'owncloudsql'. For backwards compatibility reasons it's also possible to use the 'ocis' and 's3ng' driver and configure them using the 'decomposed'/'decomposeds3' options. The 'posix' driver stores data directly on a POSIX-compliant filesystem. The 'decomposed' driver stores all data (blob and meta data) in a POSIX compliant volume. The 'decomposeds3' driver stores metadata in a POSIX compliant volume and uploads blobs to the s3 bucket." introductionVersion:"1.0.0"`
|
||||
Drivers Drivers `yaml:"drivers"`
|
||||
DataServerURL string `yaml:"data_server_url" env:"STORAGE_USERS_DATA_SERVER_URL" desc:"URL of the data server, needs to be reachable by the data gateway provided by the frontend service or the user if directly exposed." introductionVersion:"1.0.0"`
|
||||
DataGatewayURL string `yaml:"data_gateway_url" env:"STORAGE_USERS_DATA_GATEWAY_URL" desc:"URL of the data gateway server" introductionVersion:"1.0.0"`
|
||||
|
||||
TransferExpires int64 `yaml:"transfer_expires" env:"STORAGE_USERS_TRANSFER_EXPIRES" desc:"The time after which the token for upload postprocessing expires" introductionVersion:"1.0.0"`
|
||||
Events Events `yaml:"events"`
|
||||
FilemetadataCache FilemetadataCache `yaml:"filemetadata_cache"`
|
||||
IDCache IDCache `yaml:"id_cache"`
|
||||
MountID string `yaml:"mount_id" env:"STORAGE_USERS_MOUNT_ID" desc:"Mount ID of this storage." introductionVersion:"1.0.0"`
|
||||
ExposeDataServer bool `yaml:"expose_data_server" env:"STORAGE_USERS_EXPOSE_DATA_SERVER" desc:"Exposes the data server directly to users and bypasses the data gateway. Ensure that the data server address is reachable by users." introductionVersion:"1.0.0"`
|
||||
ReadOnly bool `yaml:"readonly" env:"STORAGE_USERS_READ_ONLY" desc:"Set this storage to be read-only." introductionVersion:"1.0.0"`
|
||||
UploadExpiration int64 `yaml:"upload_expiration" env:"STORAGE_USERS_UPLOAD_EXPIRATION" desc:"Duration in seconds after which uploads will expire. Note that when setting this to a low number, uploads could be cancelled before they are finished and return a 403 to the user." introductionVersion:"1.0.0"`
|
||||
Tasks Tasks `yaml:"tasks"`
|
||||
ServiceAccount ServiceAccount `yaml:"service_account"`
|
||||
|
||||
// CLI
|
||||
RevaGatewayGRPCAddr string `yaml:"gateway_addr" env:"OC_GATEWAY_GRPC_ADDR;STORAGE_USERS_GATEWAY_GRPC_ADDR" desc:"The bind address of the gateway GRPC address." introductionVersion:"1.0.0"`
|
||||
MachineAuthAPIKey string `yaml:"machine_auth_api_key" env:"OC_MACHINE_AUTH_API_KEY;STORAGE_USERS_MACHINE_AUTH_API_KEY" desc:"Machine auth API key used to validate internal requests necessary for the access to resources from other services." introductionVersion:"1.0.0"`
|
||||
CliMaxAttemptsRenameFile int `yaml:"max_attempts_rename_file" env:"STORAGE_USERS_CLI_MAX_ATTEMPTS_RENAME_FILE" desc:"The maximum number of attempts to rename a file when a user restores a file to an existing destination with the same name. The minimum value is 100." introductionVersion:"1.0.0"`
|
||||
|
||||
Context context.Context `yaml:"-"`
|
||||
}
|
||||
|
||||
// Service holds general service configuration
|
||||
type Service struct {
|
||||
Name string `yaml:"-" env:"STORAGE_USERS_SERVICE_NAME" desc:"Service name to use. Change this when starting an additional storage provider with a custom configuration to prevent it from colliding with the default 'storage-users' service." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// Debug is the configuration for the debug server
|
||||
type Debug struct {
|
||||
Addr string `yaml:"addr" env:"STORAGE_USERS_DEBUG_ADDR" desc:"Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed." introductionVersion:"1.0.0"`
|
||||
Token string `yaml:"token" env:"STORAGE_USERS_DEBUG_TOKEN" desc:"Token to secure the metrics endpoint." introductionVersion:"1.0.0"`
|
||||
Pprof bool `yaml:"pprof" env:"STORAGE_USERS_DEBUG_PPROF" desc:"Enables pprof, which can be used for profiling." introductionVersion:"1.0.0"`
|
||||
Zpages bool `yaml:"zpages" env:"STORAGE_USERS_DEBUG_ZPAGES" desc:"Enables zpages, which can be used for collecting and viewing in-memory traces." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// GRPCConfig is the configuration for the grpc server
|
||||
type GRPCConfig struct {
|
||||
Addr string `yaml:"addr" env:"STORAGE_USERS_GRPC_ADDR" desc:"The bind address of the GRPC service." introductionVersion:"1.0.0"`
|
||||
TLS *shared.GRPCServiceTLS `yaml:"tls"`
|
||||
Namespace string `yaml:"-"`
|
||||
Protocol string `yaml:"protocol" env:"OC_GRPC_PROTOCOL;STORAGE_USERS_GRPC_PROTOCOL" desc:"The transport protocol of the GPRC service." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// HTTPConfig is the configuration for the http server
|
||||
type HTTPConfig struct {
|
||||
Addr string `yaml:"addr" env:"STORAGE_USERS_HTTP_ADDR" desc:"The bind address of the HTTP service." introductionVersion:"1.0.0"`
|
||||
Namespace string `yaml:"-"`
|
||||
Protocol string `yaml:"protocol" env:"STORAGE_USERS_HTTP_PROTOCOL" desc:"The transport protocol of the HTTP service." introductionVersion:"1.0.0"`
|
||||
Prefix string
|
||||
CORS CORS `yaml:"cors"`
|
||||
}
|
||||
|
||||
// CORS defines the available cors configuration.
|
||||
type CORS struct {
|
||||
AllowedOrigins []string `yaml:"allow_origins" env:"OC_CORS_ALLOW_ORIGINS;STORAGE_USERS_CORS_ALLOW_ORIGINS" desc:"A list of allowed CORS origins. See following chapter for more details: *Access-Control-Allow-Origin* at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
AllowedMethods []string `yaml:"allow_methods" env:"OC_CORS_ALLOW_METHODS;STORAGE_USERS_CORS_ALLOW_METHODS" desc:"A list of allowed CORS methods. See following chapter for more details: *Access-Control-Request-Method* at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Request-Method. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
AllowedHeaders []string `yaml:"allow_headers" env:"OC_CORS_ALLOW_HEADERS;STORAGE_USERS_CORS_ALLOW_HEADERS" desc:"A list of allowed CORS headers. See following chapter for more details: *Access-Control-Request-Headers* at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Request-Headers. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
AllowCredentials bool `yaml:"allow_credentials" env:"OC_CORS_ALLOW_CREDENTIALS;STORAGE_USERS_CORS_ALLOW_CREDENTIALS" desc:"Allow credentials for CORS.See following chapter for more details: *Access-Control-Allow-Credentials* at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Credentials." introductionVersion:"1.0.0"`
|
||||
ExposedHeaders []string `yaml:"expose_headers" env:"OC_CORS_EXPOSE_HEADERS;STORAGE_USERS_CORS_EXPOSE_HEADERS" desc:"A list of exposed CORS headers. See following chapter for more details: *Access-Control-Expose-Headers* at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Expose-Headers. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
MaxAge uint `yaml:"max_age" env:"OC_CORS_MAX_AGE;STORAGE_USERS_CORS_MAX_AGE" desc:"The max cache duration of preflight headers. See following chapter for more details: *Access-Control-Max-Age* at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Max-Age. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// Drivers combine all storage driver configurations
|
||||
type Drivers struct {
|
||||
Decomposed DecomposedDriver `yaml:"decomposed"`
|
||||
DecomposedS3 DecomposedS3Driver `yaml:"decomposeds3"`
|
||||
OwnCloudSQL OwnCloudSQLDriver `yaml:"owncloudsql"`
|
||||
Posix PosixDriver `yaml:"posix"`
|
||||
|
||||
EOS EOSDriver `yaml:",omitempty"` // not supported by the КуСфера product, therefore not part of docs
|
||||
Local LocalDriver `yaml:",omitempty"` // not supported by the КуСфера product, therefore not part of docs
|
||||
}
|
||||
|
||||
// AsyncPropagatorOptions configures the async propagator
|
||||
type AsyncPropagatorOptions struct {
|
||||
PropagationDelay time.Duration `yaml:"propagation_delay" env:"STORAGE_USERS_ASYNC_PROPAGATOR_PROPAGATION_DELAY" desc:"The delay between a change made to a tree and the propagation start on treesize and treetime. Multiple propagations are computed to a single one. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// DecomposedDriver is the storage driver configuration when using 'decomposed' storage driver
|
||||
type DecomposedDriver struct {
|
||||
Propagator string `yaml:"propagator" env:"OC_DECOMPOSEDFS_PROPAGATOR;STORAGE_USERS_DECOMPOSED_PROPAGATOR" desc:"The propagator used for decomposedfs. At the moment, only 'sync' is fully supported, 'async' is available as an experimental option." introductionVersion:"1.0.0"`
|
||||
AsyncPropagatorOptions AsyncPropagatorOptions `yaml:"async_propagator_options"`
|
||||
// Root is the absolute path to the location of the data
|
||||
Root string `yaml:"root" env:"STORAGE_USERS_DECOMPOSED_ROOT" desc:"The directory where the filesystem storage will store blobs and metadata. If not defined, the root directory derives from $OC_BASE_DATA_PATH/storage/users." introductionVersion:"1.0.0"`
|
||||
UserLayout string `yaml:"user_layout" env:"STORAGE_USERS_DECOMPOSED_USER_LAYOUT" desc:"Template string for the user storage layout in the user directory." introductionVersion:"1.0.0"`
|
||||
PermissionsEndpoint string `yaml:"permissions_endpoint" env:"STORAGE_USERS_PERMISSION_ENDPOINT;STORAGE_USERS_DECOMPOSED_PERMISSIONS_ENDPOINT" desc:"Endpoint of the permissions service. The endpoints can differ for 'decomposed' and 'decomposeds3'." introductionVersion:"1.0.0"`
|
||||
// PersonalSpaceAliasTemplate contains the template used to construct
|
||||
// the personal space alias, eg: `"{{.SpaceType}}/{{.User.Username | lower}}"`
|
||||
PersonalSpaceAliasTemplate string `yaml:"personalspacealias_template" env:"STORAGE_USERS_DECOMPOSED_PERSONAL_SPACE_ALIAS_TEMPLATE" desc:"Template string to construct personal space aliases." introductionVersion:"1.0.0"`
|
||||
PersonalSpacePathTemplate string `yaml:"personalspacepath_template" env:"STORAGE_USERS_DECOMPOSED_PERSONAL_SPACE_PATH_TEMPLATE" desc:"Template string to construct the paths of the personal space roots." introductionVersion:"1.0.0"`
|
||||
// GeneralSpaceAliasTemplate contains the template used to construct
|
||||
// the general space alias, eg: `{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}}`
|
||||
GeneralSpaceAliasTemplate string `yaml:"generalspacealias_template" env:"STORAGE_USERS_DECOMPOSED_GENERAL_SPACE_ALIAS_TEMPLATE" desc:"Template string to construct general space aliases." introductionVersion:"1.0.0"`
|
||||
GeneralSpacePathTemplate string `yaml:"generalspacepath_template" env:"STORAGE_USERS_DECOMPOSED_GENERAL_SPACE_PATH_TEMPLATE" desc:"Template string to construct the paths of the projects space roots." introductionVersion:"1.0.0"`
|
||||
// ShareFolder defines the name of the folder jailing all shares
|
||||
ShareFolder string `yaml:"share_folder" env:"STORAGE_USERS_DECOMPOSED_SHARE_FOLDER" desc:"Name of the folder jailing all shares." introductionVersion:"1.0.0"`
|
||||
MaxAcquireLockCycles int `yaml:"max_acquire_lock_cycles" env:"STORAGE_USERS_DECOMPOSED_MAX_ACQUIRE_LOCK_CYCLES" desc:"When trying to lock files, КуСфера will try this amount of times to acquire the lock before failing. After each try it will wait for an increasing amount of time. Values of 0 or below will be ignored and the default value will be used." introductionVersion:"1.0.0"`
|
||||
LockCycleDurationFactor int `yaml:"lock_cycle_duration_factor" env:"STORAGE_USERS_DECOMPOSED_LOCK_CYCLE_DURATION_FACTOR" desc:"When trying to lock files, КуСфера will multiply the cycle with this factor and use it as a millisecond timeout. Values of 0 or below will be ignored and the default value will be used." introductionVersion:"1.0.0"`
|
||||
MaxConcurrency int `yaml:"max_concurrency" env:"OC_MAX_CONCURRENCY;STORAGE_USERS_DECOMPOSED_MAX_CONCURRENCY" desc:"Maximum number of concurrent go-routines. Higher values can potentially get work done faster but will also cause more load on the system. Values of 0 or below will be ignored and the default value will be used." introductionVersion:"1.0.0"`
|
||||
AsyncUploads bool `yaml:"async_uploads" env:"OC_ASYNC_UPLOADS" desc:"Enable asynchronous file uploads." introductionVersion:"1.0.0"`
|
||||
MaxQuota uint64 `yaml:"max_quota" env:"OC_SPACES_MAX_QUOTA;STORAGE_USERS_DECOMPOSED_MAX_QUOTA" desc:"Set a global max quota for spaces in bytes. A value of 0 equals unlimited. If not using the global OC_SPACES_MAX_QUOTA, you must define the FRONTEND_MAX_QUOTA in the frontend service." introductionVersion:"1.0.0"`
|
||||
DisableVersioning bool `yaml:"disable_versioning" env:"OC_DISABLE_VERSIONING" desc:"Disables versioning of files. When set to true, new uploads with the same filename will overwrite existing files instead of creating a new version." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// DecomposedS3Driver is the storage driver configuration when using 'decomposeds3' storage driver
|
||||
type DecomposedS3Driver struct {
|
||||
Propagator string `yaml:"propagator" env:"OC_DECOMPOSEDFS_PROPAGATOR;STORAGE_USERS_DECOMPOSEDS3_PROPAGATOR" desc:"The propagator used for decomposedfs. At the moment, only 'sync' is fully supported, 'async' is available as an experimental option." introductionVersion:"1.0.0"`
|
||||
AsyncPropagatorOptions AsyncPropagatorOptions `yaml:"async_propagator_options"`
|
||||
// Root is the absolute path to the location of the data
|
||||
Root string `yaml:"root" env:"STORAGE_USERS_DECOMPOSEDS3_ROOT" desc:"The directory where the filesystem storage will store metadata for blobs. If not defined, the root directory derives from $OC_BASE_DATA_PATH/storage/users." introductionVersion:"1.0.0"`
|
||||
UserLayout string `yaml:"user_layout" env:"STORAGE_USERS_DECOMPOSEDS3_USER_LAYOUT" desc:"Template string for the user storage layout in the user directory." introductionVersion:"1.0.0"`
|
||||
PermissionsEndpoint string `yaml:"permissions_endpoint" env:"STORAGE_USERS_PERMISSION_ENDPOINT;STORAGE_USERS_DECOMPOSEDS3_PERMISSIONS_ENDPOINT" desc:"Endpoint of the permissions service. The endpoints can differ for 'decomposed' and 'decomposeds3'." introductionVersion:"1.0.0"`
|
||||
Region string `yaml:"region" env:"STORAGE_USERS_DECOMPOSEDS3_REGION" desc:"Region of the S3 bucket." introductionVersion:"1.0.0"`
|
||||
AccessKey string `yaml:"access_key" env:"STORAGE_USERS_DECOMPOSEDS3_ACCESS_KEY" desc:"Access key for the S3 bucket." introductionVersion:"1.0.0"`
|
||||
SecretKey string `yaml:"secret_key" env:"STORAGE_USERS_DECOMPOSEDS3_SECRET_KEY" desc:"Secret key for the S3 bucket." introductionVersion:"1.0.0"`
|
||||
Endpoint string `yaml:"endpoint" env:"STORAGE_USERS_DECOMPOSEDS3_ENDPOINT" desc:"Endpoint for the S3 bucket." introductionVersion:"1.0.0"`
|
||||
Bucket string `yaml:"bucket" env:"STORAGE_USERS_DECOMPOSEDS3_BUCKET" desc:"Name of the S3 bucket." introductionVersion:"1.0.0"`
|
||||
DisableContentSha256 bool `yaml:"put_object_disable_content_sha254" env:"STORAGE_USERS_DECOMPOSEDS3_PUT_OBJECT_DISABLE_CONTENT_SHA256" desc:"Disable sending content sha256 when copying objects to S3." introductionVersion:"1.0.0"`
|
||||
DisableMultipart bool `yaml:"put_object_disable_multipart" env:"STORAGE_USERS_DECOMPOSEDS3_PUT_OBJECT_DISABLE_MULTIPART" desc:"Disable multipart uploads when copying objects to S3." introductionVersion:"1.0.0"`
|
||||
SendContentMd5 bool `yaml:"put_object_send_content_md5" env:"STORAGE_USERS_DECOMPOSEDS3_PUT_OBJECT_SEND_CONTENT_MD5" desc:"Send a Content-MD5 header when copying objects to S3." introductionVersion:"1.0.0"`
|
||||
ConcurrentStreamParts bool `yaml:"put_object_concurrent_stream_parts" env:"STORAGE_USERS_DECOMPOSEDS3_PUT_OBJECT_CONCURRENT_STREAM_PARTS" desc:"Always precreate parts when copying objects to S3. This is not recommended. It uses a memory buffer. If true, PartSize needs to be set." introductionVersion:"1.0.0"`
|
||||
NumThreads uint `yaml:"put_object_num_threads" env:"STORAGE_USERS_DECOMPOSEDS3_PUT_OBJECT_NUM_THREADS" desc:"Number of concurrent uploads to use when copying objects to S3." introductionVersion:"1.0.0"`
|
||||
PartSize uint64 `yaml:"put_object_part_size" env:"STORAGE_USERS_DECOMPOSEDS3_PUT_OBJECT_PART_SIZE" desc:"Part size for concurrent uploads to S3. If no value or 0 is set, the library automatically calculates the part size according to the total size of the file to be uploaded. The value range is min 5MB and max 5GB." introductionVersion:"1.0.0"`
|
||||
// PersonalSpaceAliasTemplate contains the template used to construct
|
||||
// the personal space alias, eg: `"{{.SpaceType}}/{{.User.Username | lower}}"`
|
||||
PersonalSpaceAliasTemplate string `yaml:"personalspacealias_template" env:"STORAGE_USERS_DECOMPOSEDS3_PERSONAL_SPACE_ALIAS_TEMPLATE" desc:"Template string to construct personal space aliases." introductionVersion:"1.0.0"`
|
||||
PersonalSpacePathTemplate string `yaml:"personalspacepath_template" env:"STORAGE_USERS_DECOMPOSEDS3_PERSONAL_SPACE_PATH_TEMPLATE" desc:"Template string to construct the paths of the personal space roots." introductionVersion:"1.0.0"`
|
||||
// GeneralSpaceAliasTemplate contains the template used to construct
|
||||
// the general space alias, eg: `{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}}`
|
||||
GeneralSpaceAliasTemplate string `yaml:"generalspacealias_template" env:"STORAGE_USERS_DECOMPOSEDS3_GENERAL_SPACE_ALIAS_TEMPLATE" desc:"Template string to construct general space aliases." introductionVersion:"1.0.0"`
|
||||
GeneralSpacePathTemplate string `yaml:"generalspacepath_template" env:"STORAGE_USERS_DECOMPOSEDS3_GENERAL_SPACE_PATH_TEMPLATE" desc:"Template string to construct the paths of the projects space roots." introductionVersion:"1.0.0"`
|
||||
// ShareFolder defines the name of the folder jailing all shares
|
||||
ShareFolder string `yaml:"share_folder" env:"STORAGE_USERS_DECOMPOSEDS3_SHARE_FOLDER" desc:"Name of the folder jailing all shares." introductionVersion:"1.0.0"`
|
||||
MaxAcquireLockCycles int `yaml:"max_acquire_lock_cycles" env:"STORAGE_USERS_DECOMPOSEDS3_MAX_ACQUIRE_LOCK_CYCLES" desc:"When trying to lock files, КуСфера will try this amount of times to acquire the lock before failing. After each try it will wait for an increasing amount of time. Values of 0 or below will be ignored and the default value of 20 will be used." introductionVersion:"1.0.0"`
|
||||
LockCycleDurationFactor int `yaml:"lock_cycle_duration_factor" env:"STORAGE_USERS_DECOMPOSEDS3_LOCK_CYCLE_DURATION_FACTOR" desc:"When trying to lock files, КуСфера will multiply the cycle with this factor and use it as a millisecond timeout. Values of 0 or below will be ignored and the default value of 30 will be used." introductionVersion:"1.0.0"`
|
||||
MaxConcurrency int `yaml:"max_concurrency" env:"OC_MAX_CONCURRENCY;STORAGE_USERS_DECOMPOSEDS3_MAX_CONCURRENCY" desc:"Maximum number of concurrent go-routines. Higher values can potentially get work done faster but will also cause more load on the system. Values of 0 or below will be ignored and the default value of 100 will be used." introductionVersion:"1.0.0"`
|
||||
AsyncUploads bool `yaml:"async_uploads" env:"OC_ASYNC_UPLOADS" desc:"Enable asynchronous file uploads." introductionVersion:"1.0.0"`
|
||||
DisableVersioning bool `yaml:"disable_versioning" env:"OC_DISABLE_VERSIONING" desc:"Disables versioning of files. When set to true, new uploads with the same filename will overwrite existing files instead of creating a new version." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// OwnCloudSQLDriver is the storage driver configuration when using 'owncloudsql' storage driver
|
||||
type OwnCloudSQLDriver struct {
|
||||
// Root is the absolute path to the location of the data
|
||||
Root string `yaml:"root" env:"STORAGE_USERS_OWNCLOUDSQL_DATADIR" desc:"The directory where the filesystem storage will store SQL migration data. If not defined, the root directory derives from $OC_BASE_DATA_PATH/storage/owncloud." introductionVersion:"1.0.0"`
|
||||
// ShareFolder defines the name of the folder jailing all shares
|
||||
ShareFolder string `yaml:"share_folder" env:"STORAGE_USERS_OWNCLOUDSQL_SHARE_FOLDER" desc:"Name of the folder jailing all shares." introductionVersion:"1.0.0"`
|
||||
UserLayout string `yaml:"user_layout" env:"STORAGE_USERS_OWNCLOUDSQL_LAYOUT" desc:"Path layout to use to navigate into a users folder in an owncloud data directory" introductionVersion:"1.0.0"`
|
||||
UploadInfoDir string `yaml:"upload_info_dir" env:"STORAGE_USERS_OWNCLOUDSQL_UPLOADINFO_DIR" desc:"The directory where the filesystem will store uploads temporarily. If not defined, the root directory derives from $OC_BASE_DATA_PATH/storage/uploadinfo." introductionVersion:"1.0.0"`
|
||||
DBUsername string `yaml:"db_username" env:"STORAGE_USERS_OWNCLOUDSQL_DB_USERNAME" desc:"Username for the database." introductionVersion:"1.0.0"`
|
||||
DBPassword string `yaml:"db_password" env:"STORAGE_USERS_OWNCLOUDSQL_DB_PASSWORD" desc:"Password for the database." introductionVersion:"1.0.0"`
|
||||
DBHost string `yaml:"db_host" env:"STORAGE_USERS_OWNCLOUDSQL_DB_HOST" desc:"Hostname or IP of the database server." introductionVersion:"1.0.0"`
|
||||
DBPort int `yaml:"db_port" env:"STORAGE_USERS_OWNCLOUDSQL_DB_PORT" desc:"Port that the database server is listening on." introductionVersion:"1.0.0"`
|
||||
DBName string `yaml:"db_name" env:"STORAGE_USERS_OWNCLOUDSQL_DB_NAME" desc:"Name of the database to be used." introductionVersion:"1.0.0"`
|
||||
UsersProviderEndpoint string `yaml:"users_provider_endpoint" env:"STORAGE_USERS_OWNCLOUDSQL_USERS_PROVIDER_ENDPOINT" desc:"Endpoint of the users provider." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// PosixDriver is the storage driver configuration when using 'posix' storage driver
|
||||
type PosixDriver struct {
|
||||
// Root is the absolute path to the location of the data
|
||||
Root string `yaml:"root" env:"STORAGE_USERS_POSIX_ROOT" desc:"The directory where the filesystem storage will store its data. If not defined, the root directory derives from $OC_BASE_DATA_PATH/storage/users." introductionVersion:"1.0.0"`
|
||||
Propagator string `yaml:"propagator" env:"OC_DECOMPOSEDFS_PROPAGATOR;STORAGE_USERS_POSIX_PROPAGATOR" desc:"The propagator used for the posix driver. At the moment, only 'sync' is fully supported, 'async' is available as an experimental option." introductionVersion:"2.0.0"`
|
||||
AsyncPropagatorOptions AsyncPropagatorOptions `yaml:"async_propagator_options"`
|
||||
PersonalSpaceAliasTemplate string `yaml:"personalspacealias_template" env:"STORAGE_USERS_POSIX_PERSONAL_SPACE_ALIAS_TEMPLATE" desc:"Template string to construct personal space aliases." introductionVersion:"1.0.0"`
|
||||
PersonalSpacePathTemplate string `yaml:"personalspacepath_template" env:"STORAGE_USERS_POSIX_PERSONAL_SPACE_PATH_TEMPLATE" desc:"Template string to construct the paths of the personal space roots." introductionVersion:"1.0.0"`
|
||||
GeneralSpaceAliasTemplate string `yaml:"generalspacealias_template" env:"STORAGE_USERS_POSIX_GENERAL_SPACE_ALIAS_TEMPLATE" desc:"Template string to construct general space aliases." introductionVersion:"1.0.0"`
|
||||
GeneralSpacePathTemplate string `yaml:"generalspacepath_template" env:"STORAGE_USERS_POSIX_GENERAL_SPACE_PATH_TEMPLATE" desc:"Template string to construct the paths of the projects space roots." introductionVersion:"1.0.0"`
|
||||
PermissionsEndpoint string `yaml:"permissions_endpoint" env:"STORAGE_USERS_PERMISSION_ENDPOINT;STORAGE_USERS_POSIX_PERMISSIONS_ENDPOINT" desc:"Endpoint of the permissions service. The endpoints can differ for 'decomposed', 'posix' and 'decomposeds3'." introductionVersion:"1.0.0"`
|
||||
AsyncUploads bool `yaml:"async_uploads" env:"OC_ASYNC_UPLOADS" desc:"Enable asynchronous file uploads." introductionVersion:"1.0.0"`
|
||||
ScanDebounceDelay time.Duration `yaml:"scan_debounce_delay" env:"STORAGE_USERS_POSIX_SCAN_DEBOUNCE_DELAY" desc:"The time in milliseconds to wait before scanning the filesystem for changes after a change has been detected." introductionVersion:"1.0.0"`
|
||||
MaxQuota uint64 `yaml:"max_quota" env:"OC_SPACES_MAX_QUOTA;STORAGE_USERS_POSIX_MAX_QUOTA" desc:"Set a global max quota for spaces in bytes. A value of 0 equals unlimited. If not using the global OC_SPACES_MAX_QUOTA, you must define the FRONTEND_MAX_QUOTA in the frontend service." introductionVersion:"2.0.0"`
|
||||
MaxAcquireLockCycles int `yaml:"max_acquire_lock_cycles" env:"STORAGE_USERS_POSIX_MAX_ACQUIRE_LOCK_CYCLES" desc:"When trying to lock files, КуСфера will try this amount of times to acquire the lock before failing. After each try it will wait for an increasing amount of time. Values of 0 or below will be ignored and the default value will be used." introductionVersion:"2.0.0"`
|
||||
LockCycleDurationFactor int `yaml:"lock_cycle_duration_factor" env:"STORAGE_USERS_POSIX_LOCK_CYCLE_DURATION_FACTOR" desc:"When trying to lock files, КуСфера will multiply the cycle with this factor and use it as a millisecond timeout. Values of 0 or below will be ignored and the default value will be used." introductionVersion:"2.0.0"`
|
||||
MaxConcurrency int `yaml:"max_concurrency" env:"OC_MAX_CONCURRENCY;STORAGE_USERS_POSIX_MAX_CONCURRENCY" desc:"Maximum number of concurrent go-routines. Higher values can potentially get work done faster but will also cause more load on the system. Values of 0 or below will be ignored and the default value will be used." introductionVersion:"2.0.0"`
|
||||
DisableVersioning bool `yaml:"disable_versioning" env:"OC_DISABLE_VERSIONING" desc:"Disables versioning of files. When set to true, new uploads with the same filename will overwrite existing files instead of creating a new version." introductionVersion:"2.0.0"`
|
||||
|
||||
UseSpaceGroups bool `yaml:"use_space_groups" env:"STORAGE_USERS_POSIX_USE_SPACE_GROUPS" desc:"Use space groups to manage permissions on spaces." introductionVersion:"1.0.0"`
|
||||
|
||||
EnableFSRevisions bool `yaml:"enable_fs_revisions" env:"STORAGE_USERS_POSIX_ENABLE_FS_REVISIONS" desc:"Allow for generating revisions from changes done to the local storage. Note: This doubles the number of bytes stored on disk because a copy of the current revision is stored to be turned into a revision later." introductionVersion:"1.0.0"`
|
||||
|
||||
ScanFS bool `yaml:"scan_fs" env:"STORAGE_USERS_POSIX_SCAN_FS" desc:"Scan the filesystem at startup for changes and update the metadata accordingly." introductionVersion:"6.2.0"`
|
||||
WatchFS bool `yaml:"watch_fs" env:"STORAGE_USERS_POSIX_WATCH_FS" desc:"Enable the filesystem watcher to detect changes to the filesystem. This is used to detect changes to the filesystem and update the metadata accordingly." introductionVersion:"2.0.0"`
|
||||
WatchType string `yaml:"watch_type" env:"STORAGE_USERS_POSIX_WATCH_TYPE" desc:"Type of the watcher to use for getting notified about changes to the filesystem. Currently available options are 'inotifywait' (default), 'cephfs', 'gpfswatchfolder' and 'gpfsfileauditlogging'." introductionVersion:"1.0.0"`
|
||||
WatchPath string `yaml:"watch_path" env:"STORAGE_USERS_POSIX_WATCH_PATH" desc:"Path to the watch directory/file. Only applies to the 'gpfsfileauditlogging' and 'inotifywait' watcher, in which case it is the path of the file audit log file/base directory to watch." introductionVersion:"1.0.0"`
|
||||
WatchNotificationBrokers string `yaml:"watch_notification_brokers" env:"STORAGE_USERS_POSIX_WATCH_NOTIFICATION_BROKERS,STORAGE_USERS_POSIX_WATCH_FOLDER_KAFKA_BROKERS" desc:"Comma-separated list of kafka brokers to read the watchfolder events from." introductionVersion:"1.0.0" deprecationVersion:"4.0.0" deprecationInfo:"STORAGE_USERS_POSIX_WATCH_FOLDER_KAFKA_BROKERS is deprecated and will be removed in a future version. Please use STORAGE_USERS_POSIX_WATCH_NOTIFICATION_BROKERS instead."`
|
||||
WatchRoot string `yaml:"watch_root" env:"STORAGE_USERS_POSIX_WATCH_ROOT" desc:"Path to the watch root directory. Event paths will be considered relative to this path. Only applies to the 'gpswatchfolder' and 'cephfs' watchers." introductionVersion:"4.0.0"`
|
||||
InotifyStatsFrequency time.Duration `yaml:"inotify_stats_frequency" env:"STORAGE_USERS_POSIX_INOTIFY_STATS_FREQUENCY" desc:"Frequency to log inotify stats." introductionVersion:"4.0.0"`
|
||||
}
|
||||
|
||||
// Events combines the configuration options for the event bus.
|
||||
type Events struct {
|
||||
Addr string `yaml:"endpoint" env:"OC_EVENTS_ENDPOINT;STORAGE_USERS_EVENTS_ENDPOINT" desc:"The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture." introductionVersion:"1.0.0"`
|
||||
ClusterID string `yaml:"cluster" env:"OC_EVENTS_CLUSTER;STORAGE_USERS_EVENTS_CLUSTER" desc:"The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system." introductionVersion:"1.0.0"`
|
||||
TLSInsecure bool `yaml:"tls_insecure" env:"OC_INSECURE;OC_EVENTS_TLS_INSECURE;STORAGE_USERS_EVENTS_TLS_INSECURE" desc:"Whether to verify the server TLS certificates." introductionVersion:"1.0.0"`
|
||||
TLSRootCaCertPath string `yaml:"tls_root_ca_cert_path" env:"OC_EVENTS_TLS_ROOT_CA_CERTIFICATE;STORAGE_USERS_EVENTS_TLS_ROOT_CA_CERTIFICATE" desc:"The root CA certificate used to validate the server's TLS certificate. If provided STORAGE_USERS_EVENTS_TLS_INSECURE will be seen as false." introductionVersion:"1.0.0"`
|
||||
EnableTLS bool `yaml:"enable_tls" env:"OC_EVENTS_ENABLE_TLS;STORAGE_USERS_EVENTS_ENABLE_TLS" desc:"Enable TLS for the connection to the events broker. The events broker is the КуСфера service which receives and delivers events between the services." introductionVersion:"1.0.0"`
|
||||
NumConsumers int `yaml:"num_consumers" env:"STORAGE_USERS_EVENTS_NUM_CONSUMERS" desc:"The amount of concurrent event consumers to start. Event consumers are used for post-processing files. Multiple consumers increase parallelisation, but will also increase CPU and memory demands. The setting has no effect when the OC_ASYNC_UPLOADS is set to false. The default and minimum value is 1." introductionVersion:"1.0.0"`
|
||||
AuthUsername string `yaml:"username" env:"OC_EVENTS_AUTH_USERNAME;STORAGE_USERS_EVENTS_AUTH_USERNAME" desc:"The username to authenticate with the events broker. The events broker is the КуСфера service which receives and delivers events between the services." introductionVersion:"1.0.0"`
|
||||
AuthPassword string `yaml:"password" env:"OC_EVENTS_AUTH_PASSWORD;STORAGE_USERS_EVENTS_AUTH_PASSWORD" desc:"The password to authenticate with the events broker. The events broker is the КуСфера service which receives and delivers events between the services." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// FilemetadataCache holds cache config
|
||||
type FilemetadataCache struct {
|
||||
Store string `yaml:"store" env:"OC_CACHE_STORE;STORAGE_USERS_FILEMETADATA_CACHE_STORE" desc:"The type of the cache store. Supported values are: 'memory', 'redis-sentinel', 'nats-js-kv', 'noop'. See the text description for details." introductionVersion:"1.0.0"`
|
||||
Nodes []string `yaml:"nodes" env:"OC_CACHE_STORE_NODES;STORAGE_USERS_FILEMETADATA_CACHE_STORE_NODES" desc:"A list of nodes to access the configured store. This has no effect when 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
Database string `yaml:"database" env:"OC_CACHE_DATABASE" desc:"The database name the configured store should use." introductionVersion:"1.0.0"`
|
||||
TTL time.Duration `yaml:"ttl" env:"OC_CACHE_TTL;STORAGE_USERS_FILEMETADATA_CACHE_TTL" desc:"Default time to live for user info in the user info cache. Only applied when access tokens has no expiration. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
DisablePersistence bool `yaml:"disable_persistence" env:"OC_CACHE_DISABLE_PERSISTENCE;STORAGE_USERS_FILEMETADATA_CACHE_DISABLE_PERSISTENCE" desc:"Disables persistence of the cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false." introductionVersion:"1.0.0"`
|
||||
AuthUsername string `yaml:"username" env:"OC_CACHE_AUTH_USERNAME;STORAGE_USERS_FILEMETADATA_CACHE_AUTH_USERNAME" desc:"The username to authenticate with the cache store. Only applies when store type 'nats-js-kv' is configured." introductionVersion:"1.0.0"`
|
||||
AuthPassword string `yaml:"password" env:"OC_CACHE_AUTH_PASSWORD;STORAGE_USERS_FILEMETADATA_CACHE_AUTH_PASSWORD" desc:"The password to authenticate with the cache store. Only applies when store type 'nats-js-kv' is configured." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// IDCache holds cache config
|
||||
type IDCache struct {
|
||||
Store string `yaml:"store" env:"OC_CACHE_STORE;STORAGE_USERS_ID_CACHE_STORE" desc:"The type of the cache store. Supported values are: 'memory', 'redis-sentinel', 'nats-js-kv', 'noop'. See the text description for details." introductionVersion:"1.0.0"`
|
||||
Nodes []string `yaml:"nodes" env:"OC_CACHE_STORE_NODES;STORAGE_USERS_ID_CACHE_STORE_NODES" desc:"A list of nodes to access the configured store. This has no effect when 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
Database string `yaml:"database" env:"OC_CACHE_DATABASE" desc:"The database name the configured store should use." introductionVersion:"1.0.0"`
|
||||
DisablePersistence bool `yaml:"disable_persistence" env:"OC_CACHE_DISABLE_PERSISTENCE;STORAGE_USERS_ID_CACHE_DISABLE_PERSISTENCE" desc:"Disables persistence of the cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false." introductionVersion:"1.0.0"`
|
||||
AuthUsername string `yaml:"username" env:"OC_CACHE_AUTH_USERNAME;STORAGE_USERS_ID_CACHE_AUTH_USERNAME" desc:"The username to authenticate with the cache store. Only applies when store type 'nats-js-kv' is configured." introductionVersion:"1.0.0"`
|
||||
AuthPassword string `yaml:"password" env:"OC_CACHE_AUTH_PASSWORD;STORAGE_USERS_ID_CACHE_AUTH_PASSWORD" desc:"The password to authenticate with the cache store. Only applies when store type 'nats-js-kv' is configured." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// EOSDriver is the storage driver configuration when using 'eos' storage driver
|
||||
type EOSDriver struct {
|
||||
// Root is the absolute path to the location of the data
|
||||
Root string `yaml:"root"`
|
||||
// ShadowNamespace for storing shadow data
|
||||
ShadowNamespace string `yaml:"shadow_namespace"`
|
||||
// UploadsNamespace for storing upload data
|
||||
UploadsNamespace string `yaml:"uploads_namespace"`
|
||||
// Location of the eos binary.
|
||||
// Default is /usr/bin/eos.
|
||||
EosBinary string `yaml:"eos_binary"`
|
||||
// Location of the xrdcopy binary.
|
||||
// Default is /usr/bin/xrdcopy.
|
||||
XrdcopyBinary string `yaml:"xrd_copy_binary"`
|
||||
// URL of the Master EOS MGM.
|
||||
// Default is root://eos-example.org
|
||||
MasterURL string `yaml:"master_url"`
|
||||
// URL of the Slave EOS MGM.
|
||||
// Default is root://eos-example.org
|
||||
SlaveURL string `yaml:"slave_url"`
|
||||
// Location on the local fs where to store reads.
|
||||
// Defaults to os.TempDir()
|
||||
CacheDirectory string `yaml:"cache_directory"`
|
||||
// SecProtocol specifies the xrootd security protocol to use between the server and EOS.
|
||||
SecProtocol string `yaml:"sec_protocol"`
|
||||
// Keytab specifies the location of the keytab to use to authenticate to EOS.
|
||||
Keytab string `yaml:"keytab"`
|
||||
// SingleUsername is the username to use when SingleUserMode is enabled
|
||||
SingleUsername string `yaml:"single_username"`
|
||||
// Enables logging of the commands executed
|
||||
// Defaults to false
|
||||
EnableLogging bool `yaml:"enable_logging"`
|
||||
// ShowHiddenSysFiles shows internal EOS files like
|
||||
// .sys.v# and .sys.a# files.
|
||||
ShowHiddenSysFiles bool `yaml:"shadow_hidden_files"`
|
||||
// ForceSingleUserMode will force connections to EOS to use SingleUsername
|
||||
ForceSingleUserMode bool `yaml:"force_single_user_mode"`
|
||||
// UseKeyTabAuth changes will authenticate requests by using an EOS keytab.
|
||||
UseKeytab bool `yaml:"user_keytab"`
|
||||
// gateway service to use for uid lookups
|
||||
GatewaySVC string `yaml:"gateway_svc"`
|
||||
// ShareFolder defines the name of the folder jailing all shares
|
||||
ShareFolder string `yaml:"share_folder"`
|
||||
GRPCURI string
|
||||
UserLayout string
|
||||
}
|
||||
|
||||
// LocalDriver is the storage driver configuration when using 'local' storage driver
|
||||
type LocalDriver struct {
|
||||
// Root is the absolute path to the location of the data
|
||||
Root string `yaml:"root"`
|
||||
// ShareFolder defines the name of the folder jailing all shares
|
||||
ShareFolder string `yaml:"share_folder"`
|
||||
UserLayout string `yaml:"user_layout"`
|
||||
}
|
||||
|
||||
// Tasks wraps task configurations
|
||||
type Tasks struct {
|
||||
PurgeTrashBin PurgeTrashBin `yaml:"purge_trash_bin"`
|
||||
}
|
||||
|
||||
// PurgeTrashBin contains all necessary configurations to clean up the respective trash cans
|
||||
type PurgeTrashBin struct {
|
||||
UserID string `yaml:"user_id" env:"OC_ADMIN_USER_ID;STORAGE_USERS_PURGE_TRASH_BIN_USER_ID" desc:"ID of the user who collects all necessary information for deletion. Consider that the UUID can be encoded in some LDAP deployment configurations like in .ldif files. These need to be decoded beforehand." introductionVersion:"1.0.0"`
|
||||
PersonalDeleteBefore time.Duration `yaml:"personal_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the personal trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
ProjectDeleteBefore time.Duration `yaml:"project_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the project trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
|
||||
}
|
||||
|
||||
// ServiceAccount is the configuration for the used service account
|
||||
type ServiceAccount struct {
|
||||
ServiceAccountID string `yaml:"service_account_id" env:"OC_SERVICE_ACCOUNT_ID;STORAGE_USERS_SERVICE_ACCOUNT_ID" desc:"The ID of the service account the service should use. See the 'auth-service' service description for more details." introductionVersion:"1.0.0"`
|
||||
ServiceAccountSecret string `yaml:"service_account_secret" env:"OC_SERVICE_ACCOUNT_SECRET;STORAGE_USERS_SERVICE_ACCOUNT_SECRET" desc:"The service account secret." introductionVersion:"1.0.0"`
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
package defaults
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/qsfera/server/pkg/config/defaults"
|
||||
"github.com/qsfera/server/pkg/shared"
|
||||
"github.com/qsfera/server/pkg/structs"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
)
|
||||
|
||||
// FullDefaultConfig returns a fully initialized default configuration
|
||||
func FullDefaultConfig() *config.Config {
|
||||
cfg := DefaultConfig()
|
||||
EnsureDefaults(cfg)
|
||||
Sanitize(cfg)
|
||||
return cfg
|
||||
}
|
||||
|
||||
// DefaultConfig returns a basic default configuration
|
||||
func DefaultConfig() *config.Config {
|
||||
return &config.Config{
|
||||
Debug: config.Debug{
|
||||
Addr: "127.0.0.1:9159",
|
||||
Token: "",
|
||||
Pprof: false,
|
||||
Zpages: false,
|
||||
},
|
||||
GRPC: config.GRPCConfig{
|
||||
Addr: "127.0.0.1:9157",
|
||||
Namespace: "qsfera.api",
|
||||
Protocol: "tcp",
|
||||
},
|
||||
HTTP: config.HTTPConfig{
|
||||
Addr: "127.0.0.1:9158",
|
||||
Namespace: "qsfera.web",
|
||||
Protocol: "tcp",
|
||||
Prefix: "data",
|
||||
CORS: config.CORS{
|
||||
AllowedOrigins: []string{"https://localhost:9200"},
|
||||
AllowedMethods: []string{
|
||||
"POST",
|
||||
"HEAD",
|
||||
"PATCH",
|
||||
"OPTIONS",
|
||||
"GET",
|
||||
"DELETE",
|
||||
},
|
||||
AllowedHeaders: []string{
|
||||
"Authorization",
|
||||
"Origin",
|
||||
"X-Requested-With",
|
||||
"X-Request-Id",
|
||||
"X-HTTP-Method-Override",
|
||||
"Content-Type",
|
||||
"Upload-Length",
|
||||
"Upload-Offset",
|
||||
"Tus-Resumable",
|
||||
"Upload-Metadata",
|
||||
"Upload-Defer-Length",
|
||||
"Upload-Concat",
|
||||
"Upload-Incomplete",
|
||||
"Upload-Draft-Interop-Version",
|
||||
},
|
||||
AllowCredentials: false,
|
||||
ExposedHeaders: []string{
|
||||
"Upload-Offset",
|
||||
"Location",
|
||||
"Upload-Length",
|
||||
"Tus-Version",
|
||||
"Tus-Resumable",
|
||||
"Tus-Max-Size",
|
||||
"Tus-Extension",
|
||||
"Upload-Metadata",
|
||||
"Upload-Defer-Length",
|
||||
"Upload-Concat",
|
||||
"Upload-Incomplete",
|
||||
"Upload-Draft-Interop-Version",
|
||||
},
|
||||
MaxAge: 86400,
|
||||
},
|
||||
},
|
||||
Service: config.Service{
|
||||
Name: "storage-users",
|
||||
},
|
||||
Reva: shared.DefaultRevaConfig(),
|
||||
DataServerURL: "http://localhost:9158/data",
|
||||
DataGatewayURL: "http://localhost:9140/data",
|
||||
RevaGatewayGRPCAddr: "127.0.0.1:9142",
|
||||
TransferExpires: 86400,
|
||||
UploadExpiration: 24 * 60 * 60,
|
||||
GracefulShutdownTimeout: 30,
|
||||
Driver: "posix",
|
||||
Drivers: config.Drivers{
|
||||
OwnCloudSQL: config.OwnCloudSQLDriver{
|
||||
Root: filepath.Join(defaults.BaseDataPath(), "storage", "owncloud"),
|
||||
ShareFolder: "/Shares",
|
||||
UserLayout: "{{.Username}}",
|
||||
UploadInfoDir: filepath.Join(defaults.BaseDataPath(), "storage", "uploadinfo"),
|
||||
DBUsername: "owncloud",
|
||||
DBPassword: "owncloud",
|
||||
DBHost: "",
|
||||
DBPort: 3306,
|
||||
DBName: "owncloud",
|
||||
UsersProviderEndpoint: "qsfera.api.users",
|
||||
},
|
||||
DecomposedS3: config.DecomposedS3Driver{
|
||||
Propagator: "sync",
|
||||
Root: filepath.Join(defaults.BaseDataPath(), "storage", "users"),
|
||||
ShareFolder: "/Shares",
|
||||
UserLayout: "{{.Id.OpaqueId}}",
|
||||
Region: "default",
|
||||
SendContentMd5: true,
|
||||
ConcurrentStreamParts: false,
|
||||
NumThreads: 4,
|
||||
PersonalSpaceAliasTemplate: "{{.SpaceType}}/{{.User.Username | lower}}",
|
||||
PersonalSpacePathTemplate: "",
|
||||
GeneralSpaceAliasTemplate: "{{.SpaceType}}/{{.SpaceName | replace \" \" \"-\" | lower}}",
|
||||
GeneralSpacePathTemplate: "",
|
||||
PermissionsEndpoint: "qsfera.api.settings",
|
||||
MaxAcquireLockCycles: 20,
|
||||
MaxConcurrency: 5,
|
||||
LockCycleDurationFactor: 30,
|
||||
DisableMultipart: false,
|
||||
PartSize: 0,
|
||||
AsyncUploads: true,
|
||||
},
|
||||
Decomposed: config.DecomposedDriver{
|
||||
Propagator: "sync",
|
||||
Root: filepath.Join(defaults.BaseDataPath(), "storage", "users"),
|
||||
ShareFolder: "/Shares",
|
||||
UserLayout: "{{.Id.OpaqueId}}",
|
||||
PersonalSpaceAliasTemplate: "{{.SpaceType}}/{{.User.Username | lower}}",
|
||||
PersonalSpacePathTemplate: "",
|
||||
GeneralSpaceAliasTemplate: "{{.SpaceType}}/{{.SpaceName | replace \" \" \"-\" | lower}}",
|
||||
GeneralSpacePathTemplate: "",
|
||||
PermissionsEndpoint: "qsfera.api.settings",
|
||||
MaxAcquireLockCycles: 20,
|
||||
MaxConcurrency: 5,
|
||||
LockCycleDurationFactor: 30,
|
||||
AsyncUploads: true,
|
||||
},
|
||||
Posix: config.PosixDriver{
|
||||
UseSpaceGroups: false,
|
||||
Root: filepath.Join(defaults.BaseDataPath(), "storage", "users"),
|
||||
PersonalSpaceAliasTemplate: "{{.SpaceType}}/{{.User.Username | lower}}",
|
||||
PersonalSpacePathTemplate: "users/{{.User.Id.OpaqueId}}",
|
||||
GeneralSpaceAliasTemplate: "{{.SpaceType}}/{{.SpaceName | replace \" \" \"-\" | lower}}",
|
||||
GeneralSpacePathTemplate: "projects/{{.SpaceId}}",
|
||||
PermissionsEndpoint: "qsfera.api.settings",
|
||||
AsyncUploads: true,
|
||||
ScanDebounceDelay: 1 * time.Second,
|
||||
ScanFS: true,
|
||||
WatchFS: false,
|
||||
EnableFSRevisions: false,
|
||||
InotifyStatsFrequency: 5 * time.Minute,
|
||||
},
|
||||
},
|
||||
Events: config.Events{
|
||||
Addr: "127.0.0.1:9233",
|
||||
ClusterID: "qsfera-cluster",
|
||||
EnableTLS: false,
|
||||
},
|
||||
FilemetadataCache: config.FilemetadataCache{
|
||||
Store: "memory",
|
||||
Nodes: []string{"127.0.0.1:9233"},
|
||||
Database: "storage-users",
|
||||
TTL: 24 * time.Hour,
|
||||
},
|
||||
IDCache: config.IDCache{
|
||||
Store: "nats-js-kv",
|
||||
Nodes: []string{"127.0.0.1:9233"},
|
||||
Database: "ids-storage-users",
|
||||
},
|
||||
Tasks: config.Tasks{
|
||||
PurgeTrashBin: config.PurgeTrashBin{
|
||||
ProjectDeleteBefore: 30 * 24 * time.Hour,
|
||||
PersonalDeleteBefore: 30 * 24 * time.Hour,
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// EnsureDefaults adds default values to the configuration if they are not set yet
|
||||
func EnsureDefaults(cfg *config.Config) {
|
||||
if cfg.LogLevel == "" {
|
||||
cfg.LogLevel = "error"
|
||||
}
|
||||
|
||||
if cfg.Reva == nil && cfg.Commons != nil {
|
||||
cfg.Reva = structs.CopyOrZeroValue(cfg.Commons.Reva)
|
||||
}
|
||||
|
||||
if cfg.TokenManager == nil && cfg.Commons != nil && cfg.Commons.TokenManager != nil {
|
||||
cfg.TokenManager = &config.TokenManager{
|
||||
JWTSecret: cfg.Commons.TokenManager.JWTSecret,
|
||||
}
|
||||
} else if cfg.TokenManager == nil {
|
||||
cfg.TokenManager = &config.TokenManager{}
|
||||
}
|
||||
|
||||
if cfg.GRPC.TLS == nil && cfg.Commons != nil {
|
||||
cfg.GRPC.TLS = structs.CopyOrZeroValue(cfg.Commons.GRPCServiceTLS)
|
||||
}
|
||||
|
||||
if cfg.Tasks.PurgeTrashBin.UserID == "" && cfg.Commons != nil {
|
||||
cfg.Tasks.PurgeTrashBin.UserID = cfg.Commons.AdminUserID
|
||||
}
|
||||
|
||||
if (cfg.Commons != nil && cfg.Commons.QsferaURL != "") &&
|
||||
(cfg.HTTP.CORS.AllowedOrigins == nil ||
|
||||
len(cfg.HTTP.CORS.AllowedOrigins) == 1 &&
|
||||
cfg.HTTP.CORS.AllowedOrigins[0] == "https://localhost:9200") {
|
||||
cfg.HTTP.CORS.AllowedOrigins = []string{cfg.Commons.QsferaURL}
|
||||
}
|
||||
}
|
||||
|
||||
// Sanitize sanitized the configuration
|
||||
func Sanitize(cfg *config.Config) {
|
||||
// nothing to sanitize here atm
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
occfg "github.com/qsfera/server/pkg/config"
|
||||
defaults2 "github.com/qsfera/server/pkg/config/defaults"
|
||||
"github.com/qsfera/server/pkg/shared"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config/defaults"
|
||||
|
||||
"github.com/qsfera/server/pkg/config/envdecode"
|
||||
)
|
||||
|
||||
// ParseConfig loads configuration from known paths.
|
||||
func ParseConfig(cfg *config.Config) error {
|
||||
err := occfg.BindSourcesToStructs(cfg.Service.Name, cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defaults.EnsureDefaults(cfg)
|
||||
|
||||
// load all env variables relevant to the config in the current context.
|
||||
if err := envdecode.Decode(cfg); err != nil {
|
||||
// no environment variable set for this config is an expected "error"
|
||||
if !errors.Is(err, envdecode.ErrNoTargetFieldsAreSet) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
defaults.Sanitize(cfg)
|
||||
|
||||
return Validate(cfg)
|
||||
}
|
||||
|
||||
func Validate(cfg *config.Config) error {
|
||||
if cfg.TokenManager.JWTSecret == "" {
|
||||
return shared.MissingJWTTokenError(cfg.Service.Name)
|
||||
}
|
||||
|
||||
if cfg.MountID == "" {
|
||||
return fmt.Errorf("The storage users mount ID has not been configured for %s. "+
|
||||
"Make sure your %s config contains the proper values "+
|
||||
"(e.g. by running qsfera init or setting it manually in "+
|
||||
"the config/corresponding environment variable).",
|
||||
"storage-users", defaults2.BaseConfigPath())
|
||||
}
|
||||
|
||||
if cfg.ServiceAccount.ServiceAccountID == "" {
|
||||
return shared.MissingServiceAccountID(cfg.Service.Name)
|
||||
}
|
||||
if cfg.ServiceAccount.ServiceAccountSecret == "" {
|
||||
return shared.MissingServiceAccountSecret(cfg.Service.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
package config
|
||||
|
||||
// TokenManager is the config for using the reva token manager
|
||||
type TokenManager struct {
|
||||
JWTSecret string `yaml:"jwt_secret" env:"OC_JWT_SECRET;STORAGE_USERS_JWT_SECRET" desc:"The secret to mint and validate jwt tokens." introductionVersion:"1.0.0"`
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package event
|
||||
|
||||
import (
|
||||
"github.com/qsfera/server/pkg/generators"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/events/stream"
|
||||
"go-micro.dev/v4/events"
|
||||
)
|
||||
|
||||
// NewStream prepares the requested nats stream and returns it.
|
||||
func NewStream(cfg *config.Config) (events.Stream, error) {
|
||||
connName := generators.GenerateConnectionName(cfg.Service.Name, generators.NTypeBus)
|
||||
return stream.NatsFromConfig(connName, false, stream.NatsConfig{
|
||||
Endpoint: cfg.Events.Addr,
|
||||
Cluster: cfg.Events.ClusterID,
|
||||
EnableTLS: cfg.Events.EnableTLS,
|
||||
TLSInsecure: cfg.Events.TLSInsecure,
|
||||
TLSRootCACertificate: cfg.Events.TLSRootCaCertPath,
|
||||
AuthUsername: cfg.Events.AuthUsername,
|
||||
AuthPassword: cfg.Events.AuthPassword,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package event
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
apiGateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
|
||||
"github.com/qsfera/server/pkg/log"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/task"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/events"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
|
||||
)
|
||||
|
||||
const (
|
||||
consumerGroup = "storage-users"
|
||||
)
|
||||
|
||||
// Service wraps all common logic that is needed to react to incoming events.
|
||||
type Service struct {
|
||||
gatewaySelector pool.Selectable[apiGateway.GatewayAPIClient]
|
||||
eventStream events.Stream
|
||||
logger log.Logger
|
||||
config config.Config
|
||||
ctx context.Context
|
||||
}
|
||||
|
||||
// NewService prepares and returns a Service implementation.
|
||||
func NewService(ctx context.Context, gatewaySelector pool.Selectable[apiGateway.GatewayAPIClient], eventStream events.Stream, logger log.Logger, conf config.Config) (Service, error) {
|
||||
svc := Service{
|
||||
gatewaySelector: gatewaySelector,
|
||||
eventStream: eventStream,
|
||||
logger: logger,
|
||||
config: conf,
|
||||
ctx: ctx,
|
||||
}
|
||||
|
||||
return svc, nil
|
||||
}
|
||||
|
||||
// Run to fulfil Runner interface
|
||||
func (s Service) Run() error {
|
||||
ch, err := events.Consume(s.eventStream, consumerGroup, PurgeTrashBin{})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-s.ctx.Done():
|
||||
s.logger.Info().Str("service", s.config.Service.Name).Msg("Context canceled. Shutting down event handler")
|
||||
return nil
|
||||
case e, more := <-ch:
|
||||
if !more {
|
||||
s.logger.Info().Str("service", s.config.Service.Name).Msg("Event channel closed. Shutting down event handler")
|
||||
// the channel was closed we can stop here
|
||||
return nil
|
||||
}
|
||||
s.handleEvent(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s Service) handleEvent(e events.Event) {
|
||||
var errs []error
|
||||
|
||||
switch ev := e.Event.(type) {
|
||||
case PurgeTrashBin:
|
||||
executionTime := ev.ExecutionTime
|
||||
if executionTime.IsZero() {
|
||||
executionTime = time.Now()
|
||||
}
|
||||
|
||||
tasks := map[task.SpaceType]time.Time{
|
||||
task.Project: executionTime.Add(-s.config.Tasks.PurgeTrashBin.ProjectDeleteBefore),
|
||||
task.Personal: executionTime.Add(-s.config.Tasks.PurgeTrashBin.PersonalDeleteBefore),
|
||||
}
|
||||
|
||||
for spaceType, deleteBefore := range tasks {
|
||||
// skip task execution if the deleteBefore time is the same as the now time,
|
||||
// which indicates that the duration configuration for this space type is set to 0 which is the equivalent to disabled.
|
||||
if deleteBefore.Equal(executionTime) {
|
||||
continue
|
||||
}
|
||||
|
||||
if err := task.PurgeTrashBin(s.config.ServiceAccount.ServiceAccountID, deleteBefore, spaceType, s.gatewaySelector, s.config.ServiceAccount.ServiceAccountSecret); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
for _, err := range errs {
|
||||
s.logger.Error().Err(err).Interface("event", e).Msg("Error running PurgeTrashBin task")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package event
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"time"
|
||||
|
||||
apiUser "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1"
|
||||
)
|
||||
|
||||
// PurgeTrashBin wraps all needed information to purge a trash-bin
|
||||
type PurgeTrashBin struct {
|
||||
ExecutantID *apiUser.UserId
|
||||
ExecutionTime time.Time
|
||||
}
|
||||
|
||||
// Unmarshal to fulfill umarshaller interface
|
||||
func (PurgeTrashBin) Unmarshal(v []byte) (any, error) {
|
||||
e := PurgeTrashBin{}
|
||||
err := json.Unmarshal(v, &e)
|
||||
return e, err
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
// Package revaconfig contains the config for the reva service
|
||||
package revaconfig
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/qsfera/server/pkg/generators"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
)
|
||||
|
||||
// StorageUsersConfigFromStruct will adapt a КуСфера config struct into a reva mapstructure to start a reva service.
|
||||
func StorageUsersConfigFromStruct(cfg *config.Config) map[string]any {
|
||||
rcfg := map[string]any{
|
||||
"core": map[string]any{
|
||||
"graceful_shutdown_timeout": cfg.GracefulShutdownTimeout,
|
||||
},
|
||||
"shared": map[string]any{
|
||||
"jwt_secret": cfg.TokenManager.JWTSecret,
|
||||
"gatewaysvc": cfg.Reva.Address,
|
||||
"skip_user_groups_in_token": cfg.SkipUserGroupsInToken,
|
||||
"grpc_client_options": cfg.Reva.GetGRPCClientConfig(),
|
||||
"multi_tenant_enabled": cfg.Commons.MultiTenantEnabled,
|
||||
},
|
||||
"grpc": map[string]any{
|
||||
"network": cfg.GRPC.Protocol,
|
||||
"address": cfg.GRPC.Addr,
|
||||
"tls_settings": map[string]any{
|
||||
"enabled": cfg.GRPC.TLS.Enabled,
|
||||
"certificate": cfg.GRPC.TLS.Cert,
|
||||
"key": cfg.GRPC.TLS.Key,
|
||||
},
|
||||
// TODO build services dynamically
|
||||
"services": map[string]any{
|
||||
"storageprovider": map[string]any{
|
||||
"driver": cfg.Driver,
|
||||
"drivers": StorageProviderDrivers(cfg),
|
||||
"mount_id": cfg.MountID,
|
||||
"expose_data_server": cfg.ExposeDataServer,
|
||||
"data_server_url": cfg.DataServerURL,
|
||||
"upload_expiration": cfg.UploadExpiration,
|
||||
"events": map[string]any{
|
||||
"nats_address": cfg.Events.Addr,
|
||||
"nats_clusterid": cfg.Events.ClusterID,
|
||||
"tls_insecure": cfg.Events.TLSInsecure,
|
||||
"tls_root_ca_cert": cfg.Events.TLSRootCaCertPath,
|
||||
"nats_enable_tls": cfg.Events.EnableTLS,
|
||||
"nats_username": cfg.Events.AuthUsername,
|
||||
"nats_password": cfg.Events.AuthPassword,
|
||||
},
|
||||
},
|
||||
},
|
||||
"interceptors": map[string]any{
|
||||
"eventsmiddleware": map[string]any{
|
||||
"group": "sharing",
|
||||
"type": "nats",
|
||||
"address": cfg.Events.Addr,
|
||||
"clusterID": cfg.Events.ClusterID,
|
||||
"tls-insecure": cfg.Events.TLSInsecure,
|
||||
"tls-root-ca-cert": cfg.Events.TLSRootCaCertPath,
|
||||
"enable-tls": cfg.Events.EnableTLS,
|
||||
"name": generators.GenerateConnectionName(cfg.Service.Name, generators.NTypeBus),
|
||||
"username": cfg.Events.AuthUsername,
|
||||
"password": cfg.Events.AuthPassword,
|
||||
},
|
||||
"prometheus": map[string]any{
|
||||
"namespace": "qsfera",
|
||||
"subsystem": "storage_users",
|
||||
},
|
||||
},
|
||||
},
|
||||
"http": map[string]any{
|
||||
"network": cfg.HTTP.Protocol,
|
||||
"address": cfg.HTTP.Addr,
|
||||
"middlewares": map[string]any{
|
||||
"requestid": map[string]any{},
|
||||
},
|
||||
// TODO build services dynamically
|
||||
"services": map[string]any{
|
||||
"dataprovider": map[string]any{
|
||||
"prefix": cfg.HTTP.Prefix,
|
||||
"driver": cfg.Driver,
|
||||
"drivers": DataProviderDrivers(cfg),
|
||||
"nats_address": cfg.Events.Addr,
|
||||
"nats_clusterID": cfg.Events.ClusterID,
|
||||
"nats_tls_insecure": cfg.Events.TLSInsecure,
|
||||
"nats_root_ca_cert_path": cfg.Events.TLSRootCaCertPath,
|
||||
"nats_enable_tls": cfg.Events.EnableTLS,
|
||||
"nats_username": cfg.Events.AuthUsername,
|
||||
"nats_password": cfg.Events.AuthPassword,
|
||||
"data_txs": map[string]any{
|
||||
"simple": map[string]any{
|
||||
"cache_store": "noop",
|
||||
"cache_database": "system",
|
||||
"cache_table": "stat",
|
||||
},
|
||||
"spaces": map[string]any{
|
||||
"cache_store": "noop",
|
||||
"cache_database": "system",
|
||||
"cache_table": "stat",
|
||||
},
|
||||
"tus": map[string]any{
|
||||
"cache_store": "noop",
|
||||
"cache_database": "system",
|
||||
"cache_table": "stat",
|
||||
"cors_enabled": true,
|
||||
// allow_origin is configured as a regex in tusd, so we concatenate the configured values into a regex
|
||||
"cors_allow_origin": "(" + strings.ReplaceAll(strings.Join(cfg.HTTP.CORS.AllowedOrigins, "|"), "*", ".*") + ")",
|
||||
"cors_allow_credentials": cfg.HTTP.CORS.AllowCredentials,
|
||||
"cors_allow_methods": strings.Join(cfg.HTTP.CORS.AllowedMethods, ","),
|
||||
"cors_allow_headers": strings.Join(cfg.HTTP.CORS.AllowedHeaders, ","),
|
||||
"cors_max_age": strconv.FormatUint(uint64(cfg.HTTP.CORS.MaxAge), 10),
|
||||
"cors_expose_headers": strings.Join(cfg.HTTP.CORS.ExposedHeaders, ","),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
if cfg.ReadOnly {
|
||||
gcfg := rcfg["grpc"].(map[string]any)
|
||||
gcfg["interceptors"] = map[string]any{
|
||||
"readonly": map[string]any{},
|
||||
}
|
||||
}
|
||||
return rcfg
|
||||
}
|
||||
@@ -0,0 +1,393 @@
|
||||
package revaconfig
|
||||
|
||||
import (
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
)
|
||||
|
||||
// EOS is the config mapping for the EOS storage driver
|
||||
func EOS(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"namespace": cfg.Drivers.EOS.Root,
|
||||
"shadow_namespace": cfg.Drivers.EOS.ShadowNamespace,
|
||||
"uploads_namespace": cfg.Drivers.EOS.UploadsNamespace,
|
||||
"share_folder": cfg.Drivers.EOS.ShareFolder,
|
||||
"eos_binary": cfg.Drivers.EOS.EosBinary,
|
||||
"xrdcopy_binary": cfg.Drivers.EOS.XrdcopyBinary,
|
||||
"master_url": cfg.Drivers.EOS.MasterURL,
|
||||
"slave_url": cfg.Drivers.EOS.SlaveURL,
|
||||
"cache_directory": cfg.Drivers.EOS.CacheDirectory,
|
||||
"sec_protocol": cfg.Drivers.EOS.SecProtocol,
|
||||
"keytab": cfg.Drivers.EOS.Keytab,
|
||||
"single_username": cfg.Drivers.EOS.SingleUsername,
|
||||
"enable_logging": cfg.Drivers.EOS.EnableLogging,
|
||||
"show_hidden_sys_files": cfg.Drivers.EOS.ShowHiddenSysFiles,
|
||||
"force_single_user_mode": cfg.Drivers.EOS.ForceSingleUserMode,
|
||||
"use_keytab": cfg.Drivers.EOS.UseKeytab,
|
||||
"gatewaysvc": cfg.Drivers.EOS.GatewaySVC,
|
||||
}
|
||||
}
|
||||
|
||||
// EOSHome is the config mapping for the EOSHome storage driver
|
||||
func EOSHome(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"namespace": cfg.Drivers.EOS.Root,
|
||||
"shadow_namespace": cfg.Drivers.EOS.ShadowNamespace,
|
||||
"uploads_namespace": cfg.Drivers.EOS.UploadsNamespace,
|
||||
"share_folder": cfg.Drivers.EOS.ShareFolder,
|
||||
"eos_binary": cfg.Drivers.EOS.EosBinary,
|
||||
"xrdcopy_binary": cfg.Drivers.EOS.XrdcopyBinary,
|
||||
"master_url": cfg.Drivers.EOS.MasterURL,
|
||||
"slave_url": cfg.Drivers.EOS.SlaveURL,
|
||||
"cache_directory": cfg.Drivers.EOS.CacheDirectory,
|
||||
"sec_protocol": cfg.Drivers.EOS.SecProtocol,
|
||||
"keytab": cfg.Drivers.EOS.Keytab,
|
||||
"single_username": cfg.Drivers.EOS.SingleUsername,
|
||||
"user_layout": cfg.Drivers.EOS.UserLayout,
|
||||
"enable_logging": cfg.Drivers.EOS.EnableLogging,
|
||||
"show_hidden_sys_files": cfg.Drivers.EOS.ShowHiddenSysFiles,
|
||||
"force_single_user_mode": cfg.Drivers.EOS.ForceSingleUserMode,
|
||||
"use_keytab": cfg.Drivers.EOS.UseKeytab,
|
||||
"gatewaysvc": cfg.Drivers.EOS.GatewaySVC,
|
||||
}
|
||||
}
|
||||
|
||||
// EOSGRPC is the config mapping for the EOSGRPC storage driver
|
||||
func EOSGRPC(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"namespace": cfg.Drivers.EOS.Root,
|
||||
"shadow_namespace": cfg.Drivers.EOS.ShadowNamespace,
|
||||
"share_folder": cfg.Drivers.EOS.ShareFolder,
|
||||
"eos_binary": cfg.Drivers.EOS.EosBinary,
|
||||
"xrdcopy_binary": cfg.Drivers.EOS.XrdcopyBinary,
|
||||
"master_url": cfg.Drivers.EOS.MasterURL,
|
||||
"master_grpc_uri": cfg.Drivers.EOS.GRPCURI,
|
||||
"slave_url": cfg.Drivers.EOS.SlaveURL,
|
||||
"cache_directory": cfg.Drivers.EOS.CacheDirectory,
|
||||
"sec_protocol": cfg.Drivers.EOS.SecProtocol,
|
||||
"keytab": cfg.Drivers.EOS.Keytab,
|
||||
"single_username": cfg.Drivers.EOS.SingleUsername,
|
||||
"user_layout": cfg.Drivers.EOS.UserLayout,
|
||||
"enable_logging": cfg.Drivers.EOS.EnableLogging,
|
||||
"show_hidden_sys_files": cfg.Drivers.EOS.ShowHiddenSysFiles,
|
||||
"force_single_user_mode": cfg.Drivers.EOS.ForceSingleUserMode,
|
||||
"use_keytab": cfg.Drivers.EOS.UseKeytab,
|
||||
"enable_home": false,
|
||||
"gatewaysvc": cfg.Drivers.EOS.GatewaySVC,
|
||||
}
|
||||
}
|
||||
|
||||
// Local is the config mapping for the Local storage driver
|
||||
func Local(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"root": cfg.Drivers.Local.Root,
|
||||
"share_folder": cfg.Drivers.Local.ShareFolder,
|
||||
}
|
||||
}
|
||||
|
||||
// Posix is the config mapping for the Posix storage driver
|
||||
func Posix(cfg *config.Config, enableFSScan, enableFSWatch bool) map[string]any {
|
||||
return map[string]any{
|
||||
"root": cfg.Drivers.Posix.Root,
|
||||
"personalspacepath_template": cfg.Drivers.Posix.PersonalSpacePathTemplate,
|
||||
"personalspacealias_template": cfg.Drivers.Posix.PersonalSpaceAliasTemplate,
|
||||
"generalspacepath_template": cfg.Drivers.Posix.GeneralSpacePathTemplate,
|
||||
"generalspacealias_template": cfg.Drivers.Posix.GeneralSpaceAliasTemplate,
|
||||
"permissionssvc": cfg.Drivers.Posix.PermissionsEndpoint,
|
||||
"permissionssvc_tls_mode": cfg.Commons.GRPCClientTLS.Mode,
|
||||
"treetime_accounting": true,
|
||||
"treesize_accounting": true,
|
||||
"asyncfileuploads": cfg.Drivers.Posix.AsyncUploads,
|
||||
"scan_debounce_delay": cfg.Drivers.Posix.ScanDebounceDelay,
|
||||
"max_quota": cfg.Drivers.Posix.MaxQuota,
|
||||
"disable_versioning": cfg.Drivers.Posix.DisableVersioning,
|
||||
"multi_tenant_enabled": cfg.Commons.MultiTenantEnabled,
|
||||
"propagator": cfg.Drivers.Posix.Propagator,
|
||||
"async_propagator_options": map[string]any{
|
||||
"propagation_delay": cfg.Drivers.Posix.AsyncPropagatorOptions.PropagationDelay,
|
||||
},
|
||||
"max_acquire_lock_cycles": cfg.Drivers.Posix.MaxAcquireLockCycles,
|
||||
"lock_cycle_duration_factor": cfg.Drivers.Posix.LockCycleDurationFactor,
|
||||
"max_concurrency": cfg.Drivers.Posix.MaxConcurrency,
|
||||
"idcache": map[string]any{
|
||||
"cache_store": cfg.IDCache.Store,
|
||||
"cache_nodes": cfg.IDCache.Nodes,
|
||||
"cache_database": cfg.IDCache.Database,
|
||||
"cache_disable_persistence": cfg.IDCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.IDCache.AuthUsername,
|
||||
"cache_auth_password": cfg.IDCache.AuthPassword,
|
||||
},
|
||||
"filemetadatacache": map[string]any{
|
||||
"cache_store": cfg.FilemetadataCache.Store,
|
||||
"cache_nodes": cfg.FilemetadataCache.Nodes,
|
||||
"cache_database": cfg.FilemetadataCache.Database,
|
||||
"cache_ttl": cfg.FilemetadataCache.TTL,
|
||||
"cache_disable_persistence": cfg.FilemetadataCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.FilemetadataCache.AuthUsername,
|
||||
"cache_auth_password": cfg.FilemetadataCache.AuthPassword,
|
||||
},
|
||||
"events": map[string]any{
|
||||
"numconsumers": cfg.Events.NumConsumers,
|
||||
},
|
||||
"tokens": map[string]any{
|
||||
"transfer_shared_secret": cfg.Commons.TransferSecret,
|
||||
"transfer_expires": cfg.TransferExpires,
|
||||
"download_endpoint": cfg.DataServerURL,
|
||||
"datagateway_endpoint": cfg.DataGatewayURL,
|
||||
},
|
||||
"use_space_groups": cfg.Drivers.Posix.UseSpaceGroups,
|
||||
"enable_fs_revisions": cfg.Drivers.Posix.EnableFSRevisions,
|
||||
"scan_fs": enableFSScan,
|
||||
"watch_fs": enableFSWatch,
|
||||
"watch_type": cfg.Drivers.Posix.WatchType,
|
||||
"watch_path": cfg.Drivers.Posix.WatchPath,
|
||||
"watch_notification_brokers": cfg.Drivers.Posix.WatchNotificationBrokers,
|
||||
"watch_root": cfg.Drivers.Posix.WatchRoot,
|
||||
"inotify_stats_frequency": cfg.Drivers.Posix.InotifyStatsFrequency,
|
||||
}
|
||||
}
|
||||
|
||||
// LocalHome is the config mapping for the LocalHome storage driver
|
||||
func LocalHome(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"root": cfg.Drivers.Local.Root,
|
||||
"share_folder": cfg.Drivers.Local.ShareFolder,
|
||||
"user_layout": cfg.Drivers.Local.UserLayout,
|
||||
}
|
||||
}
|
||||
|
||||
// OwnCloudSQL is the config mapping for the OwnCloudSQL storage driver
|
||||
func OwnCloudSQL(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"datadirectory": cfg.Drivers.OwnCloudSQL.Root,
|
||||
"upload_info_dir": cfg.Drivers.OwnCloudSQL.UploadInfoDir,
|
||||
"share_folder": cfg.Drivers.OwnCloudSQL.ShareFolder,
|
||||
"user_layout": cfg.Drivers.OwnCloudSQL.UserLayout,
|
||||
"enable_home": false,
|
||||
"dbusername": cfg.Drivers.OwnCloudSQL.DBUsername,
|
||||
"dbpassword": cfg.Drivers.OwnCloudSQL.DBPassword,
|
||||
"dbhost": cfg.Drivers.OwnCloudSQL.DBHost,
|
||||
"dbport": cfg.Drivers.OwnCloudSQL.DBPort,
|
||||
"dbname": cfg.Drivers.OwnCloudSQL.DBName,
|
||||
"userprovidersvc": cfg.Drivers.OwnCloudSQL.UsersProviderEndpoint,
|
||||
"tokens": map[string]any{
|
||||
"download_endpoint": cfg.DataServerURL,
|
||||
"datagateway_endpoint": cfg.DataGatewayURL,
|
||||
"transfer_shared_secret": cfg.Commons.TransferSecret,
|
||||
"transfer_expires": cfg.TransferExpires,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// Decomposed is the config mapping for the Decomposed storage driver
|
||||
func Decomposed(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"metadata_backend": "messagepack",
|
||||
"propagator": cfg.Drivers.Decomposed.Propagator,
|
||||
"async_propagator_options": map[string]any{
|
||||
"propagation_delay": cfg.Drivers.Decomposed.AsyncPropagatorOptions.PropagationDelay,
|
||||
},
|
||||
"root": cfg.Drivers.Decomposed.Root,
|
||||
"user_layout": cfg.Drivers.Decomposed.UserLayout,
|
||||
"share_folder": cfg.Drivers.Decomposed.ShareFolder,
|
||||
"personalspacealias_template": cfg.Drivers.Decomposed.PersonalSpaceAliasTemplate,
|
||||
"personalspacepath_template": cfg.Drivers.Decomposed.PersonalSpacePathTemplate,
|
||||
"generalspacealias_template": cfg.Drivers.Decomposed.GeneralSpaceAliasTemplate,
|
||||
"generalspacepath_template": cfg.Drivers.Decomposed.GeneralSpacePathTemplate,
|
||||
"treetime_accounting": true,
|
||||
"treesize_accounting": true,
|
||||
"permissionssvc": cfg.Drivers.Decomposed.PermissionsEndpoint,
|
||||
"permissionssvc_tls_mode": cfg.Commons.GRPCClientTLS.Mode,
|
||||
"max_acquire_lock_cycles": cfg.Drivers.Decomposed.MaxAcquireLockCycles,
|
||||
"lock_cycle_duration_factor": cfg.Drivers.Decomposed.LockCycleDurationFactor,
|
||||
"max_concurrency": cfg.Drivers.Decomposed.MaxConcurrency,
|
||||
"asyncfileuploads": cfg.Drivers.Decomposed.AsyncUploads,
|
||||
"max_quota": cfg.Drivers.Decomposed.MaxQuota,
|
||||
"disable_versioning": cfg.Drivers.Decomposed.DisableVersioning,
|
||||
"multi_tenant_enabled": cfg.Commons.MultiTenantEnabled,
|
||||
"filemetadatacache": map[string]any{
|
||||
"cache_store": cfg.FilemetadataCache.Store,
|
||||
"cache_nodes": cfg.FilemetadataCache.Nodes,
|
||||
"cache_database": cfg.FilemetadataCache.Database,
|
||||
"cache_ttl": cfg.FilemetadataCache.TTL,
|
||||
"cache_disable_persistence": cfg.FilemetadataCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.FilemetadataCache.AuthUsername,
|
||||
"cache_auth_password": cfg.FilemetadataCache.AuthPassword,
|
||||
},
|
||||
"idcache": map[string]any{
|
||||
"cache_store": cfg.IDCache.Store,
|
||||
"cache_nodes": cfg.IDCache.Nodes,
|
||||
"cache_database": cfg.IDCache.Database,
|
||||
"cache_disable_persistence": cfg.IDCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.IDCache.AuthUsername,
|
||||
"cache_auth_password": cfg.IDCache.AuthPassword,
|
||||
},
|
||||
"events": map[string]any{
|
||||
"numconsumers": cfg.Events.NumConsumers,
|
||||
},
|
||||
"tokens": map[string]any{
|
||||
"transfer_shared_secret": cfg.Commons.TransferSecret,
|
||||
"transfer_expires": cfg.TransferExpires,
|
||||
"download_endpoint": cfg.DataServerURL,
|
||||
"datagateway_endpoint": cfg.DataGatewayURL,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// DecomposedsNoEvents is the config mapping for the Decomposed storage driver emitting no events
|
||||
func DecomposedNoEvents(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"metadata_backend": "messagepack",
|
||||
"propagator": cfg.Drivers.Decomposed.Propagator,
|
||||
"async_propagator_options": map[string]any{
|
||||
"propagation_delay": cfg.Drivers.Decomposed.AsyncPropagatorOptions.PropagationDelay,
|
||||
},
|
||||
"root": cfg.Drivers.Decomposed.Root,
|
||||
"user_layout": cfg.Drivers.Decomposed.UserLayout,
|
||||
"share_folder": cfg.Drivers.Decomposed.ShareFolder,
|
||||
"personalspacealias_template": cfg.Drivers.Decomposed.PersonalSpaceAliasTemplate,
|
||||
"personalspacepath_template": cfg.Drivers.Decomposed.PersonalSpacePathTemplate,
|
||||
"generalspacealias_template": cfg.Drivers.Decomposed.GeneralSpaceAliasTemplate,
|
||||
"generalspacepath_template": cfg.Drivers.Decomposed.GeneralSpacePathTemplate,
|
||||
"treetime_accounting": true,
|
||||
"treesize_accounting": true,
|
||||
"permissionssvc": cfg.Drivers.Decomposed.PermissionsEndpoint,
|
||||
"permissionssvc_tls_mode": cfg.Commons.GRPCClientTLS.Mode,
|
||||
"max_acquire_lock_cycles": cfg.Drivers.Decomposed.MaxAcquireLockCycles,
|
||||
"lock_cycle_duration_factor": cfg.Drivers.Decomposed.LockCycleDurationFactor,
|
||||
"max_concurrency": cfg.Drivers.Decomposed.MaxConcurrency,
|
||||
"max_quota": cfg.Drivers.Decomposed.MaxQuota,
|
||||
"disable_versioning": cfg.Drivers.Decomposed.DisableVersioning,
|
||||
"multi_tenant_enabled": cfg.Commons.MultiTenantEnabled,
|
||||
"filemetadatacache": map[string]any{
|
||||
"cache_store": cfg.FilemetadataCache.Store,
|
||||
"cache_nodes": cfg.FilemetadataCache.Nodes,
|
||||
"cache_database": cfg.FilemetadataCache.Database,
|
||||
"cache_ttl": cfg.FilemetadataCache.TTL,
|
||||
"cache_disable_persistence": cfg.FilemetadataCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.FilemetadataCache.AuthUsername,
|
||||
"cache_auth_password": cfg.FilemetadataCache.AuthPassword,
|
||||
},
|
||||
"idcache": map[string]any{
|
||||
"cache_store": cfg.IDCache.Store,
|
||||
"cache_nodes": cfg.IDCache.Nodes,
|
||||
"cache_database": cfg.IDCache.Database,
|
||||
"cache_disable_persistence": cfg.IDCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.IDCache.AuthUsername,
|
||||
"cache_auth_password": cfg.IDCache.AuthPassword,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// DecomposedS3 is the config mapping for the decomposeds3 storage driver
|
||||
func DecomposedS3(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"metadata_backend": "messagepack",
|
||||
"propagator": cfg.Drivers.DecomposedS3.Propagator,
|
||||
"async_propagator_options": map[string]any{
|
||||
"propagation_delay": cfg.Drivers.DecomposedS3.AsyncPropagatorOptions.PropagationDelay,
|
||||
},
|
||||
"root": cfg.Drivers.DecomposedS3.Root,
|
||||
"user_layout": cfg.Drivers.DecomposedS3.UserLayout,
|
||||
"share_folder": cfg.Drivers.DecomposedS3.ShareFolder,
|
||||
"personalspacealias_template": cfg.Drivers.DecomposedS3.PersonalSpaceAliasTemplate,
|
||||
"personalspacepath_template": cfg.Drivers.DecomposedS3.PersonalSpacePathTemplate,
|
||||
"generalspacealias_template": cfg.Drivers.DecomposedS3.GeneralSpaceAliasTemplate,
|
||||
"generalspacepath_template": cfg.Drivers.DecomposedS3.GeneralSpacePathTemplate,
|
||||
"treetime_accounting": true,
|
||||
"treesize_accounting": true,
|
||||
"permissionssvc": cfg.Drivers.DecomposedS3.PermissionsEndpoint,
|
||||
"permissionssvc_tls_mode": cfg.Commons.GRPCClientTLS.Mode,
|
||||
"s3.region": cfg.Drivers.DecomposedS3.Region,
|
||||
"s3.access_key": cfg.Drivers.DecomposedS3.AccessKey,
|
||||
"s3.secret_key": cfg.Drivers.DecomposedS3.SecretKey,
|
||||
"s3.endpoint": cfg.Drivers.DecomposedS3.Endpoint,
|
||||
"s3.bucket": cfg.Drivers.DecomposedS3.Bucket,
|
||||
"s3.disable_content_sha254": cfg.Drivers.DecomposedS3.DisableContentSha256,
|
||||
"s3.disable_multipart": cfg.Drivers.DecomposedS3.DisableMultipart,
|
||||
"s3.send_content_md5": cfg.Drivers.DecomposedS3.SendContentMd5,
|
||||
"s3.concurrent_stream_parts": cfg.Drivers.DecomposedS3.ConcurrentStreamParts,
|
||||
"s3.num_threads": cfg.Drivers.DecomposedS3.NumThreads,
|
||||
"s3.part_size": cfg.Drivers.DecomposedS3.PartSize,
|
||||
"max_acquire_lock_cycles": cfg.Drivers.DecomposedS3.MaxAcquireLockCycles,
|
||||
"lock_cycle_duration_factor": cfg.Drivers.DecomposedS3.LockCycleDurationFactor,
|
||||
"max_concurrency": cfg.Drivers.DecomposedS3.MaxConcurrency,
|
||||
"disable_versioning": cfg.Drivers.DecomposedS3.DisableVersioning,
|
||||
"multi_tenant_enabled": cfg.Commons.MultiTenantEnabled,
|
||||
"asyncfileuploads": cfg.Drivers.DecomposedS3.AsyncUploads,
|
||||
"filemetadatacache": map[string]any{
|
||||
"cache_store": cfg.FilemetadataCache.Store,
|
||||
"cache_nodes": cfg.FilemetadataCache.Nodes,
|
||||
"cache_database": cfg.FilemetadataCache.Database,
|
||||
"cache_ttl": cfg.FilemetadataCache.TTL,
|
||||
"cache_disable_persistence": cfg.FilemetadataCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.FilemetadataCache.AuthUsername,
|
||||
"cache_auth_password": cfg.FilemetadataCache.AuthPassword,
|
||||
},
|
||||
"idcache": map[string]any{
|
||||
"cache_store": cfg.IDCache.Store,
|
||||
"cache_nodes": cfg.IDCache.Nodes,
|
||||
"cache_database": cfg.IDCache.Database,
|
||||
"cache_disable_persistence": cfg.IDCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.IDCache.AuthUsername,
|
||||
"cache_auth_password": cfg.IDCache.AuthPassword,
|
||||
},
|
||||
"events": map[string]any{
|
||||
"numconsumers": cfg.Events.NumConsumers,
|
||||
},
|
||||
"tokens": map[string]any{
|
||||
"transfer_shared_secret": cfg.Commons.TransferSecret,
|
||||
"transfer_expires": cfg.TransferExpires,
|
||||
"download_endpoint": cfg.DataServerURL,
|
||||
"datagateway_endpoint": cfg.DataGatewayURL,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// DecomposedS3NoEvents is the config mapping for the decomposeds3 storage driver emitting no events
|
||||
func DecomposedS3NoEvents(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"metadata_backend": "messagepack",
|
||||
"propagator": cfg.Drivers.DecomposedS3.Propagator,
|
||||
"async_propagator_options": map[string]any{
|
||||
"propagation_delay": cfg.Drivers.DecomposedS3.AsyncPropagatorOptions.PropagationDelay,
|
||||
},
|
||||
"root": cfg.Drivers.DecomposedS3.Root,
|
||||
"user_layout": cfg.Drivers.DecomposedS3.UserLayout,
|
||||
"share_folder": cfg.Drivers.DecomposedS3.ShareFolder,
|
||||
"personalspacealias_template": cfg.Drivers.Decomposed.PersonalSpaceAliasTemplate,
|
||||
"personalspacepath_template": cfg.Drivers.Decomposed.PersonalSpacePathTemplate,
|
||||
"generalspacealias_template": cfg.Drivers.Decomposed.GeneralSpaceAliasTemplate,
|
||||
"generalspacepath_template": cfg.Drivers.Decomposed.GeneralSpacePathTemplate,
|
||||
"treetime_accounting": true,
|
||||
"treesize_accounting": true,
|
||||
"permissionssvc": cfg.Drivers.DecomposedS3.PermissionsEndpoint,
|
||||
"permissionssvc_tls_mode": cfg.Commons.GRPCClientTLS.Mode,
|
||||
"s3.region": cfg.Drivers.DecomposedS3.Region,
|
||||
"s3.access_key": cfg.Drivers.DecomposedS3.AccessKey,
|
||||
"s3.secret_key": cfg.Drivers.DecomposedS3.SecretKey,
|
||||
"s3.endpoint": cfg.Drivers.DecomposedS3.Endpoint,
|
||||
"s3.bucket": cfg.Drivers.DecomposedS3.Bucket,
|
||||
"max_acquire_lock_cycles": cfg.Drivers.DecomposedS3.MaxAcquireLockCycles,
|
||||
"max_concurrency": cfg.Drivers.DecomposedS3.MaxConcurrency,
|
||||
"disable_versioning": cfg.Drivers.DecomposedS3.DisableVersioning,
|
||||
"multi_tenant_enabled": cfg.Commons.MultiTenantEnabled,
|
||||
"lock_cycle_duration_factor": cfg.Drivers.DecomposedS3.LockCycleDurationFactor,
|
||||
"filemetadatacache": map[string]any{
|
||||
"cache_store": cfg.FilemetadataCache.Store,
|
||||
"cache_nodes": cfg.FilemetadataCache.Nodes,
|
||||
"cache_database": cfg.FilemetadataCache.Database,
|
||||
"cache_ttl": cfg.FilemetadataCache.TTL,
|
||||
"cache_disable_persistence": cfg.FilemetadataCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.FilemetadataCache.AuthUsername,
|
||||
"cache_auth_password": cfg.FilemetadataCache.AuthPassword,
|
||||
},
|
||||
"idcache": map[string]any{
|
||||
"cache_store": cfg.IDCache.Store,
|
||||
"cache_nodes": cfg.IDCache.Nodes,
|
||||
"cache_database": cfg.IDCache.Database,
|
||||
"cache_disable_persistence": cfg.IDCache.DisablePersistence,
|
||||
"cache_auth_username": cfg.IDCache.AuthUsername,
|
||||
"cache_auth_password": cfg.IDCache.AuthPassword,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package revaconfig
|
||||
|
||||
import (
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
)
|
||||
|
||||
// StorageProviderDrivers are the drivers for the storage provider
|
||||
func StorageProviderDrivers(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"eos": EOS(cfg),
|
||||
"eoshome": EOSHome(cfg),
|
||||
"eosgrpc": EOSGRPC(cfg),
|
||||
"local": Local(cfg),
|
||||
"localhome": LocalHome(cfg),
|
||||
"owncloudsql": OwnCloudSQL(cfg),
|
||||
"decomposed": DecomposedNoEvents(cfg),
|
||||
"decomposeds3": DecomposedS3NoEvents(cfg),
|
||||
"posix": Posix(cfg, cfg.Drivers.Posix.ScanFS, cfg.Drivers.Posix.WatchFS),
|
||||
|
||||
"ocis": Decomposed(cfg), // deprecated: use decomposed
|
||||
"s3ng": DecomposedS3NoEvents(cfg), // deprecated: use decomposeds3
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
// DataProviderDrivers are the drivers for the storage provider
|
||||
func DataProviderDrivers(cfg *config.Config) map[string]any {
|
||||
return map[string]any{
|
||||
"eos": EOS(cfg),
|
||||
"eoshome": EOSHome(cfg),
|
||||
"eosgrpc": EOSGRPC(cfg),
|
||||
"local": Local(cfg),
|
||||
"localhome": LocalHome(cfg),
|
||||
"owncloudsql": OwnCloudSQL(cfg),
|
||||
"decomposed": Decomposed(cfg),
|
||||
"decomposeds3": DecomposedS3(cfg),
|
||||
"posix": Posix(cfg, false, false),
|
||||
|
||||
"ocis": Decomposed(cfg), // deprecated: use decomposed
|
||||
"s3ng": DecomposedS3NoEvents(cfg), // deprecated: use decomposeds3
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package debug
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/qsfera/server/pkg/log"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/config"
|
||||
)
|
||||
|
||||
// Option defines a single option function.
|
||||
type Option func(o *Options)
|
||||
|
||||
// Options defines the available options for this package.
|
||||
type Options struct {
|
||||
Logger log.Logger
|
||||
Context context.Context
|
||||
Config *config.Config
|
||||
}
|
||||
|
||||
// newOptions initializes the available default options.
|
||||
func newOptions(opts ...Option) Options {
|
||||
opt := Options{}
|
||||
|
||||
for _, o := range opts {
|
||||
o(&opt)
|
||||
}
|
||||
|
||||
return opt
|
||||
}
|
||||
|
||||
// Logger provides a function to set the logger option.
|
||||
func Logger(val log.Logger) Option {
|
||||
return func(o *Options) {
|
||||
o.Logger = val
|
||||
}
|
||||
}
|
||||
|
||||
// Context provides a function to set the context option.
|
||||
func Context(val context.Context) Option {
|
||||
return func(o *Options) {
|
||||
o.Context = val
|
||||
}
|
||||
}
|
||||
|
||||
// Config provides a function to set the config option.
|
||||
func Config(val *config.Config) Option {
|
||||
return func(o *Options) {
|
||||
o.Config = val
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package debug
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/qsfera/server/pkg/checks"
|
||||
"github.com/qsfera/server/pkg/handlers"
|
||||
"github.com/qsfera/server/pkg/service/debug"
|
||||
"github.com/qsfera/server/pkg/version"
|
||||
)
|
||||
|
||||
// Server initializes the debug service and server.
|
||||
func Server(opts ...Option) (*http.Server, error) {
|
||||
options := newOptions(opts...)
|
||||
|
||||
readyHandler := handlers.NewCheckHandler(handlers.NewCheckHandlerConfiguration().
|
||||
WithLogger(options.Logger).
|
||||
WithCheck("nats reachability", checks.NewNatsCheck(options.Config.Events.Addr)).
|
||||
WithCheck("grpc reachability", checks.NewGRPCCheck(options.Config.GRPC.Addr)),
|
||||
)
|
||||
|
||||
return debug.NewService(
|
||||
debug.Logger(options.Logger),
|
||||
debug.Context(options.Context),
|
||||
debug.Name(options.Config.Service.Name),
|
||||
debug.Version(version.GetString()),
|
||||
debug.Address(options.Config.Debug.Addr),
|
||||
debug.Token(options.Config.Debug.Token),
|
||||
debug.Pprof(options.Config.Debug.Pprof),
|
||||
debug.Zpages(options.Config.Debug.Zpages),
|
||||
debug.Ready(readyHandler),
|
||||
//debug.CorsAllowedOrigins(options.Config.HTTP.CORS.AllowedOrigins),
|
||||
//debug.CorsAllowedMethods(options.Config.HTTP.CORS.AllowedMethods),
|
||||
//debug.CorsAllowedHeaders(options.Config.HTTP.CORS.AllowedHeaders),
|
||||
//debug.CorsAllowCredentials(options.Config.HTTP.CORS.AllowCredentials),
|
||||
), nil
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
package task
|
||||
|
||||
// SpaceType represents known space types
|
||||
type SpaceType string
|
||||
|
||||
const (
|
||||
// Personal represents a space of type personal
|
||||
Personal SpaceType = "personal"
|
||||
// Project represents a space of type project
|
||||
Project SpaceType = "project"
|
||||
)
|
||||
@@ -0,0 +1,25 @@
|
||||
package task_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/qsfera/server/pkg/registry"
|
||||
mRegistry "go-micro.dev/v4/registry"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func init() {
|
||||
r := registry.GetRegistry(registry.Inmemory())
|
||||
service := registry.BuildGRPCService("qsfera.api.gateway", "", "", "")
|
||||
service.Nodes = []*mRegistry.Node{{
|
||||
Address: "any",
|
||||
}}
|
||||
|
||||
_ = r.Register(service)
|
||||
}
|
||||
func TestTask(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Task Suite")
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package task
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
apiGateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
|
||||
apiRpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1"
|
||||
apiProvider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/errtypes"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/utils"
|
||||
)
|
||||
|
||||
// PurgeTrashBin can be used to purge space trash-bin's,
|
||||
// the provided executantID must have space access.
|
||||
// removeBefore specifies how long an item must be in the trash-bin to be deleted,
|
||||
// items that stay there for a shorter time are ignored and kept in place.
|
||||
func PurgeTrashBin(serviceAccountID string, deleteBefore time.Time, spaceType SpaceType, gatewaySelector pool.Selectable[apiGateway.GatewayAPIClient], serviceAccountSecret string) error {
|
||||
gatewayClient, err := gatewaySelector.Next()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ctx, err := utils.GetServiceUserContext(serviceAccountID, gatewayClient, serviceAccountSecret)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
gatewayClient, err = gatewaySelector.Next()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
listStorageSpacesResponse, err := gatewayClient.ListStorageSpaces(ctx, &apiProvider.ListStorageSpacesRequest{
|
||||
Filters: []*apiProvider.ListStorageSpacesRequest_Filter{
|
||||
{
|
||||
Type: apiProvider.ListStorageSpacesRequest_Filter_TYPE_SPACE_TYPE,
|
||||
Term: &apiProvider.ListStorageSpacesRequest_Filter_SpaceType{
|
||||
SpaceType: string(spaceType),
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, storageSpace := range listStorageSpacesResponse.StorageSpaces {
|
||||
if typ := storageSpace.GetSpaceType(); typ != "personal" && typ != "project" {
|
||||
// ignore spaces that are neither personal nor project
|
||||
continue
|
||||
}
|
||||
storageSpaceReference := &apiProvider.Reference{
|
||||
ResourceId: storageSpace.GetRoot(),
|
||||
}
|
||||
|
||||
gatewayClient, err = gatewaySelector.Next()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
listRecycleResponse, err := gatewayClient.ListRecycle(ctx, &apiProvider.ListRecycleRequest{Ref: storageSpaceReference})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, recycleItem := range listRecycleResponse.GetRecycleItems() {
|
||||
doDelete := utils.TSToUnixNano(recycleItem.DeletionTime) < utils.TSToUnixNano(utils.TimeToTS(deleteBefore))
|
||||
if !doDelete {
|
||||
continue
|
||||
}
|
||||
|
||||
gatewayClient, err = gatewaySelector.Next()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
purgeRecycleResponse, err := gatewayClient.PurgeRecycle(ctx, &apiProvider.PurgeRecycleRequest{
|
||||
Ref: storageSpaceReference,
|
||||
Key: recycleItem.Key,
|
||||
})
|
||||
|
||||
if purgeRecycleResponse.GetStatus().GetCode() != apiRpc.Code_CODE_OK {
|
||||
return errtypes.NewErrtypeFromStatus(purgeRecycleResponse.Status)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package task_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
apiGateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
|
||||
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
|
||||
apiUser "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1"
|
||||
apiRpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1"
|
||||
apiProvider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
|
||||
apiTypes "github.com/cs3org/go-cs3apis/cs3/types/v1beta1"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"github.com/qsfera/server/services/storage-users/pkg/task"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/status"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
|
||||
"github.com/opencloud-eu/reva/v2/pkg/utils"
|
||||
cs3mocks "github.com/opencloud-eu/reva/v2/tests/cs3mocks/mocks"
|
||||
"github.com/stretchr/testify/mock"
|
||||
"google.golang.org/grpc"
|
||||
)
|
||||
|
||||
func MustMarshal(v any) []byte {
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
var _ = Describe("trash", func() {
|
||||
var (
|
||||
gatewayClient *cs3mocks.GatewayAPIClient
|
||||
gatewaySelector pool.Selectable[gateway.GatewayAPIClient]
|
||||
ctx context.Context
|
||||
now time.Time
|
||||
genericError error
|
||||
user *apiUser.User
|
||||
getUserResponse *apiUser.GetUserResponse
|
||||
authenticateResponse *apiGateway.AuthenticateResponse
|
||||
listStorageSpacesResponse *apiProvider.ListStorageSpacesResponse
|
||||
personalSpace *apiProvider.StorageSpace
|
||||
projectSpace *apiProvider.StorageSpace
|
||||
virtualSpace *apiProvider.StorageSpace
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
pool.RemoveSelector("GatewaySelector" + "qsfera.api.gateway")
|
||||
gatewayClient = &cs3mocks.GatewayAPIClient{}
|
||||
gatewaySelector = pool.GetSelector[gateway.GatewayAPIClient](
|
||||
"GatewaySelector",
|
||||
"qsfera.api.gateway",
|
||||
func(cc grpc.ClientConnInterface) gateway.GatewayAPIClient {
|
||||
return gatewayClient
|
||||
},
|
||||
)
|
||||
|
||||
ctx = context.Background()
|
||||
now = time.Now()
|
||||
genericError = errors.New("any")
|
||||
getUserResponse = &apiUser.GetUserResponse{
|
||||
Status: status.NewOK(ctx),
|
||||
}
|
||||
authenticateResponse = &apiGateway.AuthenticateResponse{
|
||||
Status: status.NewOK(ctx),
|
||||
Token: "",
|
||||
}
|
||||
listStorageSpacesResponse = &apiProvider.ListStorageSpacesResponse{
|
||||
Status: status.NewOK(ctx),
|
||||
StorageSpaces: []*apiProvider.StorageSpace{},
|
||||
}
|
||||
personalSpace = &apiProvider.StorageSpace{
|
||||
SpaceType: "personal",
|
||||
Id: &apiProvider.StorageSpaceId{
|
||||
OpaqueId: "personal",
|
||||
},
|
||||
Root: &apiProvider.ResourceId{
|
||||
OpaqueId: "personal",
|
||||
},
|
||||
}
|
||||
projectSpace = &apiProvider.StorageSpace{
|
||||
SpaceType: "project",
|
||||
Id: &apiProvider.StorageSpaceId{
|
||||
OpaqueId: "project",
|
||||
},
|
||||
Root: &apiProvider.ResourceId{
|
||||
OpaqueId: "project",
|
||||
},
|
||||
Opaque: &apiTypes.Opaque{},
|
||||
}
|
||||
// virtual is here as an example,
|
||||
// the task ignores all space types expect `project` and `personal`.
|
||||
virtualSpace = &apiProvider.StorageSpace{
|
||||
SpaceType: "virtual",
|
||||
Id: &apiProvider.StorageSpaceId{
|
||||
OpaqueId: "virtual",
|
||||
},
|
||||
Root: &apiProvider.ResourceId{
|
||||
OpaqueId: "virtual",
|
||||
},
|
||||
}
|
||||
user = &apiUser.User{
|
||||
Id: &apiUser.UserId{
|
||||
OpaqueId: "user",
|
||||
},
|
||||
}
|
||||
|
||||
})
|
||||
|
||||
Describe("PurgeTrashBin", func() {
|
||||
It("throws an error if the user cannot authenticate", func() {
|
||||
gatewayClient.On("GetUser", mock.Anything, mock.Anything).Return(getUserResponse, nil)
|
||||
gatewayClient.On("Authenticate", mock.Anything, mock.Anything).Return(nil, genericError)
|
||||
|
||||
err := task.PurgeTrashBin("service-user-id", now, task.Project, gatewaySelector, "")
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
It("throws an error if space listing fails", func() {
|
||||
gatewayClient.On("GetUser", mock.Anything, mock.Anything).Return(getUserResponse, nil)
|
||||
gatewayClient.On("Authenticate", mock.Anything, mock.Anything).Return(authenticateResponse, nil)
|
||||
gatewayClient.On("ListStorageSpaces", mock.Anything, mock.Anything).Return(nil, genericError)
|
||||
|
||||
err := task.PurgeTrashBin("service-user-id", now, task.Project, gatewaySelector, "")
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
It("only deletes items older than the specified period", func() {
|
||||
var (
|
||||
recycleItems = map[string][]*apiProvider.RecycleItem{
|
||||
"personal": {
|
||||
{Key: "now", DeletionTime: utils.TimeToTS(now)},
|
||||
{Key: "after", DeletionTime: utils.TimeToTS(now.Add(1 * time.Second))},
|
||||
{Key: "before", DeletionTime: utils.TimeToTS(now.Add(-1 * time.Second))},
|
||||
},
|
||||
"project": {
|
||||
{Key: "now", DeletionTime: utils.TimeToTS(now)},
|
||||
{Key: "after", DeletionTime: utils.TimeToTS(now.Add(1 * time.Minute))},
|
||||
{Key: "before", DeletionTime: utils.TimeToTS(now.Add(-1 * time.Minute))},
|
||||
},
|
||||
"virtual": {
|
||||
{Key: "now", DeletionTime: utils.TimeToTS(now)},
|
||||
{Key: "after", DeletionTime: utils.TimeToTS(now.Add(1 * time.Hour))},
|
||||
{Key: "before", DeletionTime: utils.TimeToTS(now.Add(-1 * time.Hour))},
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
personalSpace.Owner = user
|
||||
listStorageSpacesResponse.StorageSpaces = []*apiProvider.StorageSpace{
|
||||
personalSpace,
|
||||
projectSpace,
|
||||
virtualSpace,
|
||||
}
|
||||
projectSpace.Opaque.Map = map[string]*apiTypes.OpaqueEntry{
|
||||
"grants": {
|
||||
Decoder: "json",
|
||||
Value: MustMarshal(map[string]*apiProvider.ResourcePermissions{
|
||||
"admin": {
|
||||
Delete: true,
|
||||
},
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
gatewayClient.On("GetUser", mock.Anything, mock.Anything).Return(getUserResponse, nil)
|
||||
gatewayClient.On("Authenticate", mock.Anything, mock.Anything).Return(authenticateResponse, nil)
|
||||
gatewayClient.On("ListStorageSpaces", mock.Anything, mock.Anything).Return(listStorageSpacesResponse, nil)
|
||||
gatewayClient.On("ListRecycle", mock.Anything, mock.Anything).Return(
|
||||
func(_ context.Context, req *apiProvider.ListRecycleRequest, _ ...grpc.CallOption) *apiProvider.ListRecycleResponse {
|
||||
return &apiProvider.ListRecycleResponse{
|
||||
RecycleItems: recycleItems[req.Ref.ResourceId.OpaqueId],
|
||||
}
|
||||
}, nil,
|
||||
)
|
||||
gatewayClient.On("PurgeRecycle", mock.Anything, mock.Anything).Return(
|
||||
func(_ context.Context, req *apiProvider.PurgeRecycleRequest, _ ...grpc.CallOption) *apiProvider.PurgeRecycleResponse {
|
||||
var items []*apiProvider.RecycleItem
|
||||
for _, item := range recycleItems[req.Ref.ResourceId.OpaqueId] {
|
||||
if req.Key == item.Key {
|
||||
continue
|
||||
}
|
||||
|
||||
items = append(items, item)
|
||||
}
|
||||
|
||||
recycleItems[req.Ref.ResourceId.OpaqueId] = items
|
||||
|
||||
return &apiProvider.PurgeRecycleResponse{
|
||||
Status: &apiRpc.Status{
|
||||
Code: apiRpc.Code_CODE_OK,
|
||||
},
|
||||
}
|
||||
}, nil,
|
||||
)
|
||||
|
||||
err := task.PurgeTrashBin("service-user-id", now, task.Project, gatewaySelector, "")
|
||||
Expect(err).To(BeNil())
|
||||
Expect(recycleItems["personal"]).To(HaveLen(2))
|
||||
Expect(recycleItems["project"]).To(HaveLen(2))
|
||||
// virtual spaces are ignored
|
||||
Expect(recycleItems["virtual"]).To(HaveLen(3))
|
||||
})
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user