Initial QSfera import

This commit is contained in:
Курнат Андрей
2026-06-07 10:20:04 +03:00
commit 2315f25754
16485 changed files with 4826827 additions and 0 deletions
+11
View File
@@ -0,0 +1,11 @@
SHELL := bash
NAME := policies
ifneq (, $(shell command -v go 2> /dev/null)) # suppress `command not found warnings` for non go targets in CI
include ../../.bingo/Variables.mk
endif
include ../../.make/default.mk
include ../../.make/go.mk
include ../../.make/release.mk
include ../../.make/docs.mk
+167
View File
@@ -0,0 +1,167 @@
# Policies
The policies service provides a new gRPC API which can be used to check whether a requested operation is allowed or not. To do so, Open Policy Agent (OPA) is used to define the set of rules of what is permitted and what is not.
Policies are written in the [rego query language](https://www.openpolicyagent.org/docs/latest/policy-language/). The location of the rego files can be configured via yaml, a configuration via environment variables is not possible.
## General Information
The policies service consists of the following modules:
* Proxy authorization (middleware)
* Event authorization (async post-processing)
* gRPC API (can be used by other services)
To configure the policies service, three environment variables need to be defined:
* `POLICIES_ENGINE_TIMEOUT`
* `POLICIES_POSTPROCESSING_QUERY`
* `PROXY_POLICIES_QUERY`
Note that each query setting defines the [Complete Rules](https://www.openpolicyagent.org/docs/latest/#complete-rules) variable defined in the rego rule set the corresponding step uses for the evaluation. If the variable is mistyped or not found, the evaluation defaults to deny. Individual query definitions can be defined for each module.
To activate the policies service for a module, it must be started with a yaml configuration that points to one or more rego files. Note that if the service is scaled horizontally, each instance should have access to the same rego files to avoid unpredictable results. If a file path has been configured but the file is not present or accessible, the evaluation defaults to deny.
When using async post-processing which is done via the postprocessing service, the value `policies` must be added to the `POSTPROCESSING_STEPS` configuration in postprocessing service in the order where the evaluation should take place.
variable defined in the Rego rule set the corresponding step uses for the evaluation. If the variable is mistyped or not found, the evaluation defaults to deny. Individual query definitions can be defined for each module.
To activate the policies service for a module, it must be started with a yaml configuration that points to at least one Rego file that contains the complete rule variable to be queried. Note that if the service is scaled horizontally, each instance should have access to the same Rego files to avoid unpredictable results. If a file path has been configured but the file it is not present or accessible, the evaluation defaults to deny.
When using async post-processing via the postprocessing service, the value `policies` must be added to the `POSTPROCESSING_STEPS` configuration in the order in which the evaluation should take place. Example: First check if a file contains questionable content via policies. If it looks okay, continue to check for viruses.
For configuration examples, the [Example Policies](#example-policies) from below are used.
## Modules
### gRPC API
The gRPC API can be used by any other internal service. It can also be used for example by third parties to find out if an action is allowed or not. This layer is already used by the proxy middleware. There is no configuration necessary, because the query setting (complete rule variable) must be part of the request.
### Proxy Middleware
The proxy service already includes a middleware which uses the internal [gRPC API](#grpc-api) to evaluate the policies. Since the proxy is in heavy use and every HTTP request is processed here, only simple and quick decisions should be evaluated. More complex queries such as file content evaluation are _strongly_ discouraged.
If the evaluation in the proxy results in a "denied" outcome, the response will return a `403 Permission Denied` with the following response body
```json
{
"error":
{
"code": "deniedByPolicy",
"message": "Operation denied due to security policies",
"innererror":
{
"date": "2023-09-19T13:22:20Z",
"filename": "File",
"method": "POST",
"path": "/dav/spaces/some-space-id/Folder/",
"request-id": "9CFCE925-F9D9-4F26-AB3B-2C1C40A9CD0C"
}
}
}
```
### Event Service (Postprocessing)
This layer is event-based and part of the postprocessing service. Since processing at this point is asynchronous, the operations can also take longer and be more expensive, like evaluating the contents of a file.
## Defining Policies to Evaluate
Each module can have as many policy files as needed for evaluation. Files can also include other files if necessary. To use policies, they have to be saved to a location that is accessible to the policies service. As a good starting point, take the config directory and use a subdirectory collecting all the `.rego` files, though any other directory can be defined. The config directory is already accessible by all services and usually is included in a xref:maintenance/b-r/backup.adoc[backup] plan.
If this is done, it's required to configure the policies service to use these files:
NOTE: It is important that *all* necessary files are added to the list of files the policies service uses.
```yaml
policies:
engine:
policies:
- your_path_to_policies/proxy.rego
- your_path_to_policies/postprocessing.rego
- your_path_to_policies/util.rego
```
Once the references to policy files are configured correctly, the `_QUERY` configuration needs to be defined for the proxy middleware and for the events service.
## Setting the Query Configuration
To define a value for the query evaluation, the following scheme is necessary:
`data.<package-name>.<complete-rule-variable-name>`
* The keyword `data` is mandatory and must be present.
* The `package-name` is defined in one .rego file like `package postprocessing`. It is not related to the filename. For more details, see the [packages](https://www.openpolicyagent.org/docs/latest/policy-language/#packages) documentation.
* The `complete-rule-variable-name` is the variable providing the result of the evaluation.
* Exact one of the defined files, which is responsible for returning the evaluation result, must contain the combination of `<package-name>` and `<complete-rule-variable-name>`.
### Proxy
Note that this setting has to be part of the proxy configuration.
```yaml
proxy:
policies_middleware:
query: data.proxy.granted
```
The same can be achieved by setting the following environment variable:
```shell
export PROXY_POLICIES_QUERY=data.proxy.granted
```
### Postprocessing
```yaml
policies:
postprocessing:
query: data.postprocessing.granted
```
The same can be achieved by setting the following environment variable:
```shell
export POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted
```
As soon as that query is configured, the postprocessing service must be informed to use the policies step by setting the environment variable:
```shell
export POSTPROCESSING_STEPS=policies
```
Note that additional steps can be configured and their position in the list defines the order of processing. For details see the postprocessing service documentation.
## Rego Key Match
To identify available keys for OPA, you need to look at [engine.go](https://github.com/qsfera/server/blob/main/services/policies/pkg/engine/engine.go) and the [policies.swagger.json](https://github.com/qsfera/server/blob/master/protogen/gen/qsfera/services/policies/v0/policies.swagger.json) file. Note that which keys are available depends on from which module it is used.
## Extend Mimetype File Extension Mapping
In the extended set of the rego query language, it is possible to get a list of associated file extensions based on a mimetype, for example `qsfera.mimetype.extensions("application/pdf")`.
The list of mappings is restricted by default and is provided by the host system КуСфера is installed on.
In order to extend this list, КуСфера must be provided with the path to a custom `mime.types` file that maps mimetypes to extensions.
The location for the file must be accessible by all instances of the policy service. As a rule of thumb, use the directory where the КуСфера configuration files are stored.
Note that existing mappings from the host are extended by the definitions from the mime types file, but not replaced.
The path to that file can be provided via a yaml configuration or an environment variable. Note to replace the `OC_CONFIG_DIR` string by an existing path.
```shell
export POLICIES_ENGINE_MIMES=OC_CONFIG_DIR/mime.types
```
```yaml
policies:
engine:
mimes: OC_CONFIG_DIR/mime.types
```
A good example of how such a file should be formatted can be found in the [Apache svn repository](https://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types).
## Example Policies
The policies service contains a set of preconfigured example policies. See the [devtools policie](https://github.com/qsfera/server/tree/main/devtools/deployments/service_policies/policies/) directory for details. The contained policies disallow КуСфера to create certain file types, both via the proxy middleware and the events service via postprocessing.
@@ -0,0 +1,54 @@
package command
import (
"fmt"
"net/http"
"github.com/qsfera/server/pkg/config/configlog"
"github.com/qsfera/server/pkg/log"
"github.com/qsfera/server/services/policies/pkg/config"
"github.com/qsfera/server/services/policies/pkg/config/parser"
"github.com/spf13/cobra"
)
// Health is the entrypoint for the health command.
func Health(cfg *config.Config) *cobra.Command {
return &cobra.Command{
Use: "health",
Short: "check health status",
PreRunE: func(cmd *cobra.Command, args []string) error {
return configlog.ReturnError(parser.ParseConfig(cfg))
},
RunE: func(cmd *cobra.Command, args []string) error {
logger := log.Configure(cfg.Service.Name, cfg.Commons, cfg.LogLevel)
resp, err := http.Get(
fmt.Sprintf(
"http://%s/healthz",
cfg.Debug.Addr,
),
)
if err != nil {
logger.Fatal().
Err(err).
Msg("Failed to request health check")
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
logger.Fatal().
Int("code", resp.StatusCode).
Msg("Health seems to be in bad state")
}
logger.Debug().
Int("code", resp.StatusCode).
Msg("Health got a good state")
return nil
},
}
}
@@ -0,0 +1,31 @@
package command
import (
"os"
"github.com/qsfera/server/pkg/clihelper"
"github.com/qsfera/server/services/policies/pkg/config"
"github.com/spf13/cobra"
)
// GetCommands provides all commands for this service
func GetCommands(cfg *config.Config) []*cobra.Command {
return []*cobra.Command{
Server(cfg),
Health(cfg),
Version(cfg),
}
}
// Execute is the entry point for the policies command.
func Execute(cfg *config.Config) error {
app := clihelper.DefaultApp(&cobra.Command{
Use: "policies",
Short: "Serve policies for КуСфера",
})
app.AddCommand(GetCommands(cfg)...)
app.SetArgs(os.Args[1:])
return app.ExecuteContext(cfg.Context)
}
@@ -0,0 +1,146 @@
package command
import (
"context"
"fmt"
"os/signal"
"github.com/qsfera/server/pkg/config/configlog"
"github.com/qsfera/server/pkg/generators"
"github.com/qsfera/server/pkg/log"
"github.com/qsfera/server/pkg/runner"
"github.com/qsfera/server/pkg/service/grpc"
"github.com/qsfera/server/pkg/tracing"
"github.com/qsfera/server/pkg/version"
svcProtogen "github.com/qsfera/server/protogen/gen/qsfera/services/policies/v0"
"github.com/qsfera/server/services/policies/pkg/config"
"github.com/qsfera/server/services/policies/pkg/config/parser"
"github.com/qsfera/server/services/policies/pkg/engine/opa"
"github.com/qsfera/server/services/policies/pkg/server/debug"
svcEvent "github.com/qsfera/server/services/policies/pkg/service/event"
svcGRPC "github.com/qsfera/server/services/policies/pkg/service/grpc"
"github.com/opencloud-eu/reva/v2/pkg/events/stream"
"github.com/spf13/cobra"
)
// Server is the entrypoint for the server command.
func Server(cfg *config.Config) *cobra.Command {
return &cobra.Command{
Use: "server",
Short: fmt.Sprintf("start the %s service without runtime (unsupervised mode)", "authz"),
PreRunE: func(cmd *cobra.Command, args []string) error {
return configlog.ReturnFatal(parser.ParseConfig(cfg))
},
RunE: func(cmd *cobra.Command, args []string) error {
var cancel context.CancelFunc
if cfg.Context == nil {
cfg.Context, cancel = signal.NotifyContext(context.Background(), runner.StopSignals...)
defer cancel()
}
ctx := cfg.Context
logger := log.Configure(cfg.Service.Name, cfg.Commons, cfg.LogLevel).SubloggerWithRequestID(ctx)
traceProvider, err := tracing.GetTraceProvider(cmd.Context(), cfg.Commons.TracesExporter, cfg.Service.Name)
if err != nil {
return err
}
e, err := opa.NewOPA(cfg.Engine.Timeout, logger, cfg.Engine)
if err != nil {
return err
}
gr := runner.NewGroup()
{
grpcClient, err := grpc.NewClient(
append(
grpc.GetClientOptions(cfg.GRPCClientTLS),
grpc.WithTraceProvider(traceProvider),
)...,
)
if err != nil {
return err
}
svc, err := grpc.NewServiceWithClient(
grpcClient,
grpc.Logger(logger),
grpc.TLSEnabled(cfg.GRPC.TLS.Enabled),
grpc.TLSCert(
cfg.GRPC.TLS.Cert,
cfg.GRPC.TLS.Key,
),
grpc.Name(cfg.Service.Name),
grpc.Context(ctx),
grpc.Address(cfg.GRPC.Addr),
grpc.Namespace(cfg.GRPC.Namespace),
grpc.Version(version.GetString()),
grpc.TraceProvider(traceProvider),
)
if err != nil {
return err
}
grpcSvc, err := svcGRPC.New(e)
if err != nil {
return err
}
if err := svcProtogen.RegisterPoliciesProviderHandler(
svc.Server(),
grpcSvc,
); err != nil {
return err
}
gr.Add(runner.NewGoMicroGrpcServerRunner(cfg.Service.Name+".grpc", svc))
}
{
connName := generators.GenerateConnectionName(cfg.Service.Name, generators.NTypeBus)
bus, err := stream.NatsFromConfig(connName, false, stream.NatsConfig(cfg.Events))
if err != nil {
return err
}
eventSvc, err := svcEvent.New(ctx, bus, logger, traceProvider, e, cfg.Postprocessing.Query)
if err != nil {
return err
}
gr.Add(runner.New(cfg.Service.Name+".svc", func() error {
return eventSvc.Run()
}, func() {
eventSvc.Close()
}))
}
{
debugServer, err := debug.Server(
debug.Logger(logger),
debug.Context(ctx),
debug.Config(cfg),
)
if err != nil {
logger.Info().Err(err).Str("transport", "debug").Msg("Failed to initialize server")
return err
}
gr.Add(runner.NewGolangHttpServerRunner(cfg.Service.Name+".debug", debugServer))
}
grResults := gr.Run(ctx)
// return the first non-nil error found in the results
for _, grResult := range grResults {
if grResult.RunnerError != nil {
return grResult.RunnerError
}
}
return nil
},
}
}
@@ -0,0 +1,49 @@
package command
import (
"fmt"
"os"
"github.com/qsfera/server/pkg/registry"
"github.com/qsfera/server/pkg/version"
"github.com/qsfera/server/services/policies/pkg/config"
"github.com/olekukonko/tablewriter"
"github.com/olekukonko/tablewriter/tw"
"github.com/spf13/cobra"
)
// Version prints the service versions of all running instances.
func Version(cfg *config.Config) *cobra.Command {
return &cobra.Command{
Use: "version",
Short: "print the version of this binary and the running service instances",
RunE: func(cmd *cobra.Command, args []string) error {
fmt.Println("Version: " + version.GetString())
fmt.Printf("Compiled: %s\n", version.Compiled())
fmt.Println("")
reg := registry.GetRegistry()
services, err := reg.GetService(cfg.GRPC.Namespace + "." + cfg.Service.Name)
if err != nil {
fmt.Println(fmt.Errorf("could not get %s services from the registry: %v", cfg.Service.Name, err))
return err
}
if len(services) == 0 {
fmt.Println("No running " + cfg.Service.Name + " service found.")
return nil
}
table := tablewriter.NewTable(os.Stdout, tablewriter.WithHeaderAutoFormat(tw.Off))
table.Header([]string{"Version", "Address", "Id"})
for _, s := range services {
for _, n := range s.Nodes {
table.Append([]string{s.Version, n.Address, n.Id})
}
}
table.Render()
return nil
},
}
}
@@ -0,0 +1,66 @@
package config
import (
"context"
"time"
"github.com/qsfera/server/pkg/shared"
)
// Config combines all available configuration parts.
type Config struct {
Commons *shared.Commons `yaml:"-"` // don't use this directly as configuration for a service
GRPC GRPC `yaml:"grpc"`
Service Service `yaml:"-"`
Debug Debug `yaml:"debug"`
Events Events `yaml:"events"`
GRPCClientTLS *shared.GRPCClientTLS `yaml:"grpc_client_tls"`
Context context.Context `yaml:"-"`
LogLevel string `yaml:"loglevel" env:"OC_LOG_LEVEL;POLICIES_LOG_LEVEL" desc:"The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'." introductionVersion:"1.0.0"`
Engine Engine `yaml:"engine"`
Postprocessing Postprocessing `yaml:"postprocessing"`
}
// Service defines the available service configuration.
type Service struct {
Name string `yaml:"-"`
}
// GRPC defines the available grpc configuration.
type GRPC struct {
Addr string `yaml:"addr" env:"POLICIES_GRPC_ADDR" desc:"The bind address of the GRPC service." introductionVersion:"1.0.0"`
Namespace string `yaml:"-"`
TLS *shared.GRPCServiceTLS `yaml:"tls"`
}
// Engine configures the policy engine.
type Engine struct {
Timeout time.Duration `yaml:"timeout" env:"POLICIES_ENGINE_TIMEOUT" desc:"Sets the timeout the rego expression evaluation can take. Rules default to deny if the timeout was reached. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
Policies []string `yaml:"policies"`
// Mimes file path, RFC 4288
Mimes string `yaml:"mimes" env:"POLICIES_ENGINE_MIMES" desc:"Sets the mimes file path which maps mimetypes to associated file extensions. See the text description for details." introductionVersion:"1.0.0"`
}
// Postprocessing defines the config options for the postprocessing policy handling.
type Postprocessing struct {
Query string `yaml:"query" env:"POLICIES_POSTPROCESSING_QUERY" desc:"Defines the 'Complete Rules' variable defined in the rego rule set this step uses for its evaluation. Defaults to deny if the variable was not found." introductionVersion:"1.0.0"`
}
// Events combines the configuration options for the event bus.
type Events struct {
Endpoint string `yaml:"endpoint" env:"OC_EVENTS_ENDPOINT;POLICIES_EVENTS_ENDPOINT" desc:"The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture." introductionVersion:"1.0.0"`
Cluster string `yaml:"cluster" env:"OC_EVENTS_CLUSTER;POLICIES_EVENTS_CLUSTER" desc:"The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system." introductionVersion:"1.0.0"`
TLSInsecure bool `yaml:"tls_insecure" env:"OC_INSECURE;OC_EVENTS_TLS_INSECURE;POLICIES_EVENTS_TLS_INSECURE" desc:"Whether the server should skip the client certificate verification during the TLS handshake." introductionVersion:"1.0.0"`
TLSRootCACertificate string `yaml:"tls_root_ca_certificate" env:"OC_EVENTS_TLS_ROOT_CA_CERTIFICATE;POLICIES_EVENTS_TLS_ROOT_CA_CERTIFICATE" desc:"The root CA certificate used to validate the server's TLS certificate. If provided POLICIES_EVENTS_TLS_INSECURE will be seen as false." introductionVersion:"1.0.0"`
EnableTLS bool `yaml:"enable_tls" env:"OC_EVENTS_ENABLE_TLS;POLICIES_EVENTS_ENABLE_TLS" desc:"Enable TLS for the connection to the events broker. The events broker is the КуСфера service which receives and delivers events between the services." introductionVersion:"1.0.0"`
AuthUsername string `yaml:"username" env:"OC_EVENTS_AUTH_USERNAME;POLICIES_EVENTS_AUTH_USERNAME" desc:"The username to authenticate with the events broker. The events broker is the КуСфера service which receives and delivers events between the services." introductionVersion:"1.0.0"`
AuthPassword string `yaml:"password" env:"OC_EVENTS_AUTH_PASSWORD;POLICIES_EVENTS_AUTH_PASSWORD" desc:"The password to authenticate with the events broker. The events broker is the КуСфера service which receives and delivers events between the services." introductionVersion:"1.0.0"`
}
// Debug defines the available debug configuration.
type Debug struct {
Addr string `yaml:"addr" env:"POLICIES_DEBUG_ADDR" desc:"Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed." introductionVersion:"1.0.0"`
Token string `yaml:"token" env:"POLICIES_DEBUG_TOKEN" desc:"Token to secure the metrics endpoint." introductionVersion:"1.0.0"`
Pprof bool `yaml:"pprof" env:"POLICIES_DEBUG_PPROF" desc:"Enables pprof, which can be used for profiling." introductionVersion:"1.0.0"`
Zpages bool `yaml:"zpages" env:"POLICIES_DEBUG_ZPAGES" desc:"Enables zpages, which can be used for collecting and viewing in-memory traces." introductionVersion:"1.0.0"`
}
@@ -0,0 +1,59 @@
package defaults
import (
"time"
"github.com/qsfera/server/pkg/structs"
"github.com/qsfera/server/services/policies/pkg/config"
)
// FullDefaultConfig returns a fully initialized default configuration which is needed for doc generation.
func FullDefaultConfig() *config.Config {
cfg := DefaultConfig()
EnsureDefaults(cfg)
Sanitize(cfg)
return cfg
}
// DefaultConfig returns the default config
func DefaultConfig() *config.Config {
return &config.Config{
Service: config.Service{
Name: "policies",
},
Debug: config.Debug{
Addr: "127.0.0.1:9129",
Token: "",
Pprof: false,
Zpages: false,
},
GRPC: config.GRPC{
Addr: "127.0.0.1:9125",
Namespace: "qsfera.api",
},
Events: config.Events{
Endpoint: "127.0.0.1:9233",
Cluster: "qsfera-cluster",
EnableTLS: false,
},
Engine: config.Engine{
Timeout: 10 * time.Second,
},
}
}
func EnsureDefaults(cfg *config.Config) {
if cfg.LogLevel == "" {
cfg.LogLevel = "error"
}
if cfg.GRPCClientTLS == nil && cfg.Commons != nil {
cfg.GRPCClientTLS = structs.CopyOrZeroValue(cfg.Commons.GRPCClientTLS)
}
if cfg.GRPC.TLS == nil && cfg.Commons != nil {
cfg.GRPC.TLS = structs.CopyOrZeroValue(cfg.Commons.GRPCServiceTLS)
}
}
func Sanitize(_ *config.Config) {}
@@ -0,0 +1,37 @@
package parser
import (
"errors"
occfg "github.com/qsfera/server/pkg/config"
"github.com/qsfera/server/services/policies/pkg/config"
"github.com/qsfera/server/services/policies/pkg/config/defaults"
"github.com/qsfera/server/pkg/config/envdecode"
)
// ParseConfig loads configuration from known paths.
func ParseConfig(cfg *config.Config) error {
err := occfg.BindSourcesToStructs(cfg.Service.Name, cfg)
if err != nil {
return err
}
defaults.EnsureDefaults(cfg)
// load all env variables relevant to the config in the current context.
if err := envdecode.Decode(cfg); err != nil {
// no environment variable set for this config is an expected "error"
if !errors.Is(err, envdecode.ErrNoTargetFieldsAreSet) {
return err
}
}
defaults.Sanitize(cfg)
return Validate(cfg)
}
func Validate(cfg *config.Config) error {
return nil
}
@@ -0,0 +1,74 @@
package engine
import (
"context"
"encoding/json"
user "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
v0 "github.com/qsfera/server/protogen/gen/qsfera/messages/policies/v0"
"google.golang.org/protobuf/encoding/protojson"
)
// Engine defines the granted handlers.
type Engine interface {
Evaluate(ctx context.Context, query string, env Environment) (bool, error)
}
type (
// Stage defines the used auth stage
Stage string
)
var (
// StagePP defines the post-processing stage
StagePP Stage = "pp"
// StageHTTP defines the http stage
StageHTTP Stage = "http"
)
// Resource contains resource information and is used as part of the evaluated environment.
type Resource struct {
ID provider.ResourceId `json:"resource_id"`
Name string `json:"name"`
URL string `json:"url"`
Size uint64 `json:"size"`
}
// Request contains request information and is used as part of the evaluated environment.
type Request struct {
Method string `json:"method"`
Path string `json:"path"`
}
// Environment contains every data that is needed to decide if the request should pass or not
type Environment struct {
Stage Stage `json:"stage"`
User user.User `json:"user"`
Request Request `json:"request"`
Resource Resource `json:"resource"`
}
// NewEnvironmentFromPB converts a PBEnvironment to Environment.
func NewEnvironmentFromPB(pEnv *v0.Environment) (Environment, error) {
env := Environment{}
rData, err := protojson.Marshal(pEnv)
if err != nil {
return env, err
}
if err := json.Unmarshal(rData, &env); err != nil {
return env, err
}
switch pEnv.Stage {
case v0.Stage_STAGE_HTTP:
env.Stage = StageHTTP
case v0.Stage_STAGE_PP:
env.Stage = StagePP
}
return env, nil
}
@@ -0,0 +1,13 @@
package engine_test
import (
"testing"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
func TestEngine(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Engine Suite")
}
@@ -0,0 +1,25 @@
package engine_test
import (
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
pMessage "github.com/qsfera/server/protogen/gen/qsfera/messages/policies/v0"
"github.com/qsfera/server/services/policies/pkg/engine"
)
var _ = Describe("Engine", func() {
DescribeTable("NewEnvironmentFromPB",
func(incomingStage pMessage.Stage, outgoinStage engine.Stage) {
pEnv := &pMessage.Environment{
Stage: incomingStage,
}
env, err := engine.NewEnvironmentFromPB(pEnv)
Expect(err).ToNot(HaveOccurred())
Expect(env.Stage).To(Equal(outgoinStage))
},
Entry("http stage", pMessage.Stage_STAGE_HTTP, engine.StageHTTP),
Entry("pp stage", pMessage.Stage_STAGE_PP, engine.StagePP),
)
})
@@ -0,0 +1,79 @@
package opa
import (
"context"
"io"
"os"
"time"
"github.com/open-policy-agent/opa/rego"
"github.com/open-policy-agent/opa/topdown/print"
"github.com/qsfera/server/pkg/log"
"github.com/qsfera/server/services/policies/pkg/config"
"github.com/qsfera/server/services/policies/pkg/engine"
)
// OPA wraps open policy agent makes it possible to ask if an action is granted.
type OPA struct {
printHook print.Hook
policies []string
timeout time.Duration
options []func(r *rego.Rego)
}
// NewOPA returns a ready to use opa engine.
func NewOPA(timeout time.Duration, logger log.Logger, conf config.Engine) (OPA, error) {
var mtReader io.ReadCloser
if conf.Mimes != "" {
var err error
mtReader, err = os.Open(conf.Mimes)
if err != nil {
return OPA{}, err
}
defer mtReader.Close()
}
rfMimetypeExtensions, err := RFMimetypeExtensions(mtReader)
if err != nil {
return OPA{}, err
}
return OPA{
policies: conf.Policies,
timeout: timeout,
printHook: logPrinter{logger: logger},
options: []func(r *rego.Rego){
RFMimetypeDetect,
RFResourceDownload,
rfMimetypeExtensions,
},
}, nil
}
// Evaluate evaluates the opa policies and returns the result.
func (o OPA) Evaluate(ctx context.Context, qs string, env engine.Environment) (bool, error) {
ctx, cancel := context.WithTimeout(ctx, o.timeout)
defer cancel()
q, err := rego.New(
append([]func(r *rego.Rego){
rego.Query(qs),
rego.Load(o.policies, nil),
rego.EnablePrintStatements(true),
rego.PrintHook(o.printHook),
}, o.options...)...,
).PrepareForEval(ctx)
if err != nil {
return false, err
}
result, err := q.Eval(ctx, rego.EvalInput(env))
if err != nil {
return false, err
}
return result.Allowed(), nil
}
@@ -0,0 +1,16 @@
package opa
import (
"github.com/open-policy-agent/opa/topdown/print"
"github.com/qsfera/server/pkg/log"
)
type logPrinter struct {
logger log.Logger
}
func (lp logPrinter) Print(_ print.Context, msg string) error {
lp.logger.Info().Msg(msg)
return nil
}
@@ -0,0 +1,13 @@
package opa_test
import (
"testing"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
func TestOpa(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Opa Suite")
}
@@ -0,0 +1,97 @@
package opa
import (
"bufio"
"io"
"mime"
"strings"
"github.com/gabriel-vasile/mimetype"
"github.com/open-policy-agent/opa/ast"
"github.com/open-policy-agent/opa/rego"
"github.com/open-policy-agent/opa/types"
)
// RFMimetypeExtensions extends the rego dictionary with the possibility of mapping mimetypes to file extensions.
// Be careful calling this multiple times with individual readers, the mime store is global,
// which results in one global store which holds all known mimetype mappings at once.
//
// Rego: `qsfera.mimetype.extensions("application/pdf")`
// Result `[.pdf]`
func RFMimetypeExtensions(f io.Reader) (func(*rego.Rego), error) {
if f != nil {
scanner := bufio.NewScanner(f)
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) <= 1 || fields[0][0] == '#' {
continue
}
mimeType := fields[0]
for _, ext := range fields[1:] {
if ext[0] == '#' {
break
}
if err := mime.AddExtensionType("."+ext, mimeType); err != nil {
return nil, err
}
}
}
if err := scanner.Err(); err != nil {
return nil, err
}
}
return rego.Function1(
&rego.Function{
Name: "qsfera.mimetype.extensions",
Decl: types.NewFunction(types.Args(types.S), types.A),
Memoize: true,
Nondeterministic: true,
},
func(_ rego.BuiltinContext, a *ast.Term) (*ast.Term, error) {
var mt string
if err := ast.As(a.Value, &mt); err != nil {
return nil, err
}
detectedExtensions, err := mime.ExtensionsByType(mt)
if err != nil {
return nil, err
}
var mimeTerms []*ast.Term
for _, extension := range detectedExtensions {
mimeTerms = append(mimeTerms, ast.NewTerm(ast.String(extension)))
}
return ast.ArrayTerm(mimeTerms...), nil
},
), nil
}
// RFMimetypeDetect extends the rego dictionary with the possibility to detect mimetypes.
// Be careful, the list of known mimetypes is limited.
//
// Rego: `qsfera.mimetype.extensions(".txt")`
// Result `text/plain`
var RFMimetypeDetect = rego.Function1(
&rego.Function{
Name: "qsfera.mimetype.detect",
Decl: types.NewFunction(types.Args(types.A), types.S),
Memoize: true,
Nondeterministic: true,
},
func(_ rego.BuiltinContext, a *ast.Term) (*ast.Term, error) {
var body []byte
if err := ast.As(a.Value, &body); err != nil {
return nil, err
}
mimetype := mimetype.Detect(body).String()
return ast.StringTerm(strings.Split(mimetype, ";")[0]), nil
},
)
@@ -0,0 +1,57 @@
package opa_test
import (
"context"
"io"
"strings"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/open-policy-agent/opa/rego"
"github.com/qsfera/server/services/policies/pkg/engine/opa"
)
var _ = Describe("opa qsfera mimetype functions", func() {
Describe("qsfera.mimetype.detect", func() {
It("detects the mimetype", func() {
r := rego.New(rego.Query(`qsfera.mimetype.detect("")`), opa.RFMimetypeDetect)
rs, err := r.Eval(context.Background())
Expect(err).ToNot(HaveOccurred())
Expect(rs[0].Expressions[0].String()).To(Equal("text/plain"))
})
})
Describe("qsfera.mimetype.extensions", func() {
DescribeTable("resolves extensions by mimetype",
func(mimetype string, expectations []string, f io.Reader) {
rfMimetypeExtensions, err := opa.RFMimetypeExtensions(f)
Expect(err).ToNot(HaveOccurred())
r := rego.New(rego.Query(`qsfera.mimetype.extensions("`+mimetype+`")`), rfMimetypeExtensions)
rs, err := r.Eval(context.Background())
Expect(err).ToNot(HaveOccurred())
got := rs[0].Expressions[0].String()
if len(expectations) == 0 {
Expect(got).To(Equal("[]"))
}
for i, expectation := range expectations {
if i+1 != len(expectations) {
expectation += " "
}
Expect(string(got[0])).To(Equal("["))
Expect(strings.Contains(got, expectation)).To(BeTrue())
Expect(string(got[len(got)-1])).To(Equal("]"))
}
},
Entry("With default mimetype", "application/pdf", []string{".pdf"}, nil),
Entry("With unknown mimetype", "qsfera/with.custom.mt", []string{}, nil),
Entry("With custom mimetype", "qsfera/with.custom.mt", []string{".with.custom.mt"}, strings.NewReader("qsfera/with.custom.mt with.custom.mt")),
Entry("With multiple custom mimetypes", "qsfera/with.multiple.custom.mt", []string{".with.multiple.custom.1.mt", ".with.multiple.custom.2.mt"}, strings.NewReader("qsfera/with.multiple.custom.mt with.multiple.custom.1.mt with.multiple.custom.2.mt")),
Entry("With custom ignored mimetype", "qsfera/with.multiple.custom.ignored.mt", []string{}, strings.NewReader("#qsfera/with.multiple.custom.ignored.mt with.multiple.custom.ignored.mt")),
)
})
})
@@ -0,0 +1,60 @@
package opa
import (
"bytes"
"fmt"
"net/http"
"github.com/open-policy-agent/opa/ast"
"github.com/open-policy-agent/opa/rego"
"github.com/open-policy-agent/opa/types"
"github.com/opencloud-eu/reva/v2/pkg/rhttp"
)
// RFResourceDownload extends the rego dictionary with the possibility to download qsfera resources.
//
// Rego: `qsfera.resource.download("qsfera/path/0034892347349827")`
// Result: bytes
var RFResourceDownload = rego.Function1(
&rego.Function{
Name: "qsfera.resource.download",
Decl: types.NewFunction(types.Args(types.S), types.A),
Memoize: true,
Nondeterministic: true,
},
func(_ rego.BuiltinContext, a *ast.Term) (*ast.Term, error) {
var url string
if err := ast.As(a.Value, &url); err != nil {
return nil, err
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return nil, err
}
client := rhttp.GetHTTPClient(rhttp.Insecure(true))
res, err := client.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("unexpected status code from Download %v", res.StatusCode)
}
buf := new(bytes.Buffer)
if _, err := buf.ReadFrom(res.Body); err != nil {
return nil, err
}
v, err := ast.InterfaceToValue(buf.Bytes())
if err != nil {
return nil, err
}
return ast.NewTerm(v), nil
},
)
@@ -0,0 +1,36 @@
package opa_test
import (
"context"
"encoding/base64"
"net/http"
"net/http/httptest"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/open-policy-agent/opa/rego"
"github.com/qsfera/server/services/policies/pkg/engine/opa"
)
var _ = Describe("opa qsfera resource functions", func() {
Describe("qsfera.resource.download", func() {
It("downloads reva resources", func() {
ts := []byte("Lorem Ipsum is simply dummy text of the printing and typesetting")
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write(ts)
}))
defer srv.Close()
r := rego.New(rego.Query(`qsfera.resource.download("`+srv.URL+`")`), opa.RFResourceDownload)
rs, err := r.Eval(context.Background())
Expect(err).ToNot(HaveOccurred())
data, err := base64.StdEncoding.DecodeString(rs[0].Expressions[0].String())
Expect(err).ToNot(HaveOccurred())
Expect(data).To(Equal(ts))
})
})
})
@@ -0,0 +1,50 @@
package debug
import (
"context"
"github.com/qsfera/server/pkg/log"
"github.com/qsfera/server/services/policies/pkg/config"
)
// Option defines a single option function.
type Option func(o *Options)
// Options defines the available options for this package.
type Options struct {
Logger log.Logger
Context context.Context
Config *config.Config
}
// newOptions initializes the available default options.
func newOptions(opts ...Option) Options {
opt := Options{}
for _, o := range opts {
o(&opt)
}
return opt
}
// Logger provides a function to set the logger option.
func Logger(val log.Logger) Option {
return func(o *Options) {
o.Logger = val
}
}
// Context provides a function to set the context option.
func Context(val context.Context) Option {
return func(o *Options) {
o.Context = val
}
}
// Config provides a function to set the config option.
func Config(val *config.Config) Option {
return func(o *Options) {
o.Config = val
}
}
@@ -0,0 +1,34 @@
package debug
import (
"net/http"
"github.com/qsfera/server/pkg/checks"
"github.com/qsfera/server/pkg/handlers"
"github.com/qsfera/server/pkg/service/debug"
"github.com/qsfera/server/pkg/version"
)
// Server initializes the debug service and server.
func Server(opts ...Option) (*http.Server, error) {
options := newOptions(opts...)
healthHandlerConfiguration := handlers.NewCheckHandlerConfiguration().
WithLogger(options.Logger).
WithCheck("grpc reachability", checks.NewGRPCCheck(options.Config.GRPC.Addr))
readyHandlerConfiguration := healthHandlerConfiguration.
WithCheck("nats reachability", checks.NewNatsCheck(options.Config.Events.Endpoint))
return debug.NewService(
debug.Logger(options.Logger),
debug.Name(options.Config.Service.Name),
debug.Version(version.GetString()),
debug.Address(options.Config.Debug.Addr),
debug.Token(options.Config.Debug.Token),
debug.Pprof(options.Config.Debug.Pprof),
debug.Zpages(options.Config.Debug.Zpages),
debug.Health(handlers.NewCheckHandler(healthHandlerConfiguration)),
debug.Ready(handlers.NewCheckHandler(readyHandlerConfiguration)),
), nil
}
@@ -0,0 +1,136 @@
package eventSVC
import (
"context"
"sync/atomic"
"github.com/qsfera/server/pkg/log"
"github.com/qsfera/server/services/policies/pkg/engine"
"github.com/opencloud-eu/reva/v2/pkg/events"
"go.opentelemetry.io/otel/trace"
)
// Service defines the service handlers.
type Service struct {
ctx context.Context
query string
log log.Logger
stream events.Stream
engine engine.Engine
tp trace.TracerProvider
stopCh chan struct{}
stopped *atomic.Bool
}
// New returns a service implementation for Service.
func New(ctx context.Context, stream events.Stream, logger log.Logger, tp trace.TracerProvider, engine engine.Engine, query string) (Service, error) {
svc := Service{
ctx: ctx,
log: logger,
query: query,
tp: tp,
engine: engine,
stream: stream,
stopCh: make(chan struct{}, 1),
stopped: new(atomic.Bool),
}
return svc, nil
}
// Run to fulfil Runner interface
func (s Service) Run() error {
ch, err := events.Consume(s.stream, "policies", events.StartPostprocessingStep{})
if err != nil {
return err
}
EventLoop:
for {
select {
case <-s.stopCh:
break EventLoop
case e, ok := <-ch:
if !ok {
break EventLoop
}
err := s.processEvent(e)
if err != nil {
return err
}
if s.stopped.Load() {
break EventLoop
}
}
}
return nil
}
// Close will make the policies service to stop processing, so the `Run`
// method can finish.
// TODO: Underlying services can't be stopped. This means that some goroutines
// will get stuck trying to push events through a channel nobody is reading
// from, so resources won't be freed and there will be memory leaks. For now,
// if the service is stopped, you should close the app soon after.
func (s Service) Close() {
if s.stopped.CompareAndSwap(false, true) {
close(s.stopCh)
}
}
func (s Service) processEvent(e events.Event) error {
ctx := e.GetTraceContext(s.ctx)
ctx, span := s.tp.Tracer("policies").Start(ctx, "processEvent")
defer span.End()
switch ev := e.Event.(type) {
case events.StartPostprocessingStep:
if ev.StepToStart != events.PPStepPolicies {
return nil
}
outcome := events.PPOutcomeContinue
if s.query != "" {
env := engine.Environment{
Stage: engine.StagePP,
Resource: engine.Resource{
Name: ev.Filename,
URL: ev.URL,
Size: ev.Filesize,
},
}
if ev.ExecutingUser != nil {
env.User = *ev.ExecutingUser
}
if ev.ResourceID != nil {
env.Resource.ID = *ev.ResourceID
}
result, err := s.engine.Evaluate(context.TODO(), s.query, env)
if err != nil {
s.log.Error().Err(err).Msg("unable evaluate policy")
}
if !result {
outcome = events.PPOutcomeDelete
}
}
if err := events.Publish(ctx, s.stream, events.PostprocessingStepFinished{
Outcome: outcome,
UploadID: ev.UploadID,
ExecutingUser: ev.ExecutingUser,
Filename: ev.Filename,
FinishedStep: ev.StepToStart,
}); err != nil {
return err
}
}
return nil
}
@@ -0,0 +1,35 @@
package grpcSVC
import (
"context"
v0 "github.com/qsfera/server/protogen/gen/qsfera/services/policies/v0"
"github.com/qsfera/server/services/policies/pkg/engine"
)
// Service defines the service handlers.
type Service struct {
engine engine.Engine
}
// New returns a service implementation for Service.
func New(engine engine.Engine) (Service, error) {
svc := Service{
engine: engine,
}
return svc, nil
}
// Evaluate exposes the engine policy evaluation.
func (s Service) Evaluate(ctx context.Context, request *v0.EvaluateRequest, response *v0.EvaluateResponse) error {
env, err := engine.NewEnvironmentFromPB(request.Environment)
if err != nil {
return err
}
result, err := s.engine.Evaluate(ctx, request.Query, env)
response.Result = result
return err
}