Introduce Policies-Service (#5716)

* add policies service
add policies proxy middleware
add policies event service
add policies grpc service
prepare ci and git environments (ci, make, readme, doc)

* add webfinger to the drone conf

* fix docs
remove not used virus scan postprocessing step

* relocate example rego file
implicitly enable and disable proxy and postprocessing policy checking by setting the query.
update configuration descriptions

* move policies
update readme

* use converter func to convert pp environment to actual environment
expose and test custom rego functions
add engine unit tests
add opa unit tests
update policies readme

Co-authored-by: Martin <github@diemattels.at>

* relocate sample policies to the deployments folder
change and document policies service port

* update index.md and small fix

* add health command
add version command
add debug server

---------

Co-authored-by: Martin <github@diemattels.at>
This commit is contained in:
Florian Schade
2023-03-14 16:08:22 +01:00
committed by GitHub
co-authored by Martin
parent d06d2012be
commit f38a9f4385
48 changed files with 3106 additions and 284 deletions
+37
View File
@@ -0,0 +1,37 @@
SHELL := bash
NAME := policies
include ../../.make/recursion.mk
############ tooling ############
ifneq (, $(shell command -v go 2> /dev/null)) # suppress `command not found warnings` for non go targets in CI
include ../../.bingo/Variables.mk
endif
############ go tooling ############
include ../../.make/go.mk
############ release ############
include ../../.make/release.mk
############ docs generate ############
include ../../.make/docs.mk
.PHONY: docs-generate
docs-generate: config-docs-generate
############ generate ############
include ../../.make/generate.mk
.PHONY: ci-go-generate
ci-go-generate: # CI runs ci-node-generate automatically before this target
.PHONY: ci-node-generate
ci-node-generate:
############ licenses ############
.PHONY: ci-node-check-licenses
ci-node-check-licenses:
.PHONY: ci-node-save-licenses
ci-node-save-licenses:
+73
View File
@@ -0,0 +1,73 @@
# Policies Service
The policies service provides a new grpc api which can be used to return whether a requested operation is allowed or not. To do so, Open Policy Agent (OPA) is used to determine the set of rules of what is permitted and what is not.
Policies are written in the [rego query language](https://www.openpolicyagent.org/docs/latest/policy-language/). The location of the rego files can be configured via yaml, a configuration via environment variables is not possible.
The Policies Service consists of the following modules:
* Proxy Authorization (middleware)
* Event Authorization (async post-processing)
* GRPC API (can be used from other services)
To configure the Policies Service, three environment variables need to be defined:
* `POLICIES_ENGINE_TIMEOUT`
* `POLICIES_POSTPROCESSING_QUERY`
* `PROXY_POLICIES_QUERY`
Note that each query setting defines the [Complete Rules](https://www.openpolicyagent.org/docs/latest/#complete-rules) variable defined in the rego rule set the corresponding step uses for the evaluation. If the variable is mistyped or not found, the evaluation defaults to deny. Individual query definitions can be defined for each module.
To activate a the policies service for a module, it must be started with a yaml configuration that points to one or more rego files. Note that if the service is scaled horizontally, each instance should have access to the same rego files to avoid unpredictable results. If a file path has been configured but the file it is not present or accessible, the evaluation defaults to deny.
When using async post-processing which is done via the postprocessing service, the value `policies` must be added to the `POSTPROCESSING_STEPS` configuration in postprocessing service in the order where the evaluation should take place.
## Modules
### GRPC Service
This service can be used from any other internal service. It can also be used for example by third parties to find out if an action is allowed or not. This layer is already used by the proxy middleware.
### Event Service
This layer is event-based and part of the postprocessing service. Since processing at this point is asynchronous, the operations can also take longer and be more expensive, like evaluating the bytes of a file.
### Proxy Middleware
The [ocis proxy](../proxy) already includes such a middleware which uses the [GRPC service](#grpc-service) to evaluate the policies by using a configurable query. Since the Proxy is in heavy use and every request is processed here, only simple and quick decisions should be evaluated. More complex queries such as file evaluation are strongly discouraged.
## Example Policies
The policies service contains a set of pre-configured example policies. Those policies can be found in the [examples directory](../../deployments/examples/service_policies/policies). The contained policies disallows ocis to create certain filetypes, both for the proxy middleware and the events service.
To use the example policies, it's required to configure ocis to use these files which can be done by adding:
```yaml
policies:
engine:
policies:
- YOUR_PATH/examples/policies/proxy.rego
- YOUR_PATH/examples/policies/postprocessing.rego
- YOUR_PATH/examples/policies/utils.rego
```
Once the policies are configured correctly, the _QUERY configuration needs to be defined for the proxy middleware and for the events service.
### Proxy
```yaml
proxy:
policies_middleware:
query: data.proxy.granted
```
The same can be achieved by setting the `PROXY_POLICIES_QUERY=data.proxy.granted` environment variable.
### ASYNC Postprocessing
```yaml
policies:
postprocessing:
query: data.postprocessing.granted
```
The same can be achieved by setting the `POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted` environment variable. As soon as that query is configured correctly, postprocessing must be informed to use the policies step by setting the environment variable `POSTPROCESSING_STEPS=policies`. Note that additional steps can be configured and their appearance defines the order of processing. For details see the postprocessing service documentation.
+14
View File
@@ -0,0 +1,14 @@
package main
import (
"os"
"github.com/owncloud/ocis/v2/services/policies/pkg/command"
"github.com/owncloud/ocis/v2/services/policies/pkg/config/defaults"
)
func main() {
if err := command.Execute(defaults.DefaultConfig()); err != nil {
os.Exit(1)
}
}
+60
View File
@@ -0,0 +1,60 @@
package command
import (
"fmt"
"github.com/owncloud/ocis/v2/ocis-pkg/log"
"net/http"
"github.com/owncloud/ocis/v2/ocis-pkg/config/configlog"
"github.com/owncloud/ocis/v2/services/policies/pkg/config"
"github.com/owncloud/ocis/v2/services/policies/pkg/config/parser"
"github.com/urfave/cli/v2"
)
// Health is the entrypoint for the health command.
func Health(cfg *config.Config) *cli.Command {
return &cli.Command{
Name: "health",
Usage: "check health status",
Category: "info",
Before: func(c *cli.Context) error {
return configlog.ReturnError(parser.ParseConfig(cfg))
},
Action: func(c *cli.Context) error {
logger := log.NewLogger(
log.Name(cfg.Service.Name),
log.Level(cfg.Log.Level),
log.Pretty(cfg.Log.Pretty),
log.Color(cfg.Log.Color),
log.File(cfg.Log.File),
)
resp, err := http.Get(
fmt.Sprintf(
"http://%s/healthz",
cfg.Debug.Addr,
),
)
if err != nil {
logger.Fatal().
Err(err).
Msg("Failed to request health check")
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
logger.Fatal().
Int("code", resp.StatusCode).
Msg("Health seems to be in bad state")
}
logger.Debug().
Int("code", resp.StatusCode).
Msg("Health got a good state")
return nil
},
}
}
+54
View File
@@ -0,0 +1,54 @@
package command
import (
"context"
"os"
"github.com/owncloud/ocis/v2/ocis-pkg/clihelper"
ociscfg "github.com/owncloud/ocis/v2/ocis-pkg/config"
"github.com/owncloud/ocis/v2/services/policies/pkg/config"
"github.com/thejerf/suture/v4"
"github.com/urfave/cli/v2"
)
// GetCommands provides all commands for this service
func GetCommands(cfg *config.Config) cli.Commands {
return []*cli.Command{
Server(cfg),
Health(cfg),
Version(cfg),
}
}
// Execute is the entry point for the policies command.
func Execute(cfg *config.Config) error {
app := clihelper.DefaultApp(&cli.App{
Name: "policies",
Usage: "Serve ownCloud policies for oCIS",
Commands: GetCommands(cfg),
})
return app.Run(os.Args)
}
// SutureService allows for the web command to be embedded and supervised by a suture supervisor tree.
type SutureService struct {
cfg *config.Config
}
// NewSutureService creates a new web.SutureService
func NewSutureService(cfg *ociscfg.Config) suture.Service {
cfg.Policies.Commons = cfg.Commons
return SutureService{
cfg: cfg.Policies,
}
}
func (s SutureService) Serve(ctx context.Context) error {
s.cfg.Context = ctx
if err := Execute(s.cfg); err != nil {
return err
}
return nil
}
+193
View File
@@ -0,0 +1,193 @@
package command
import (
"context"
"crypto/tls"
"crypto/x509"
"fmt"
"io"
"net/http"
"os"
"github.com/cs3org/reva/v2/pkg/events/stream"
"github.com/go-micro/plugins/v4/events/natsjs"
"github.com/oklog/run"
"github.com/owncloud/ocis/v2/ocis-pkg/config/configlog"
ociscrypto "github.com/owncloud/ocis/v2/ocis-pkg/crypto"
"github.com/owncloud/ocis/v2/ocis-pkg/log"
"github.com/owncloud/ocis/v2/ocis-pkg/service/debug"
"github.com/owncloud/ocis/v2/ocis-pkg/service/grpc"
"github.com/owncloud/ocis/v2/ocis-pkg/version"
svcProtogen "github.com/owncloud/ocis/v2/protogen/gen/ocis/services/policies/v0"
"github.com/owncloud/ocis/v2/services/policies/pkg/config"
"github.com/owncloud/ocis/v2/services/policies/pkg/config/parser"
"github.com/owncloud/ocis/v2/services/policies/pkg/engine"
svcEvent "github.com/owncloud/ocis/v2/services/policies/pkg/service/event"
svcGRPC "github.com/owncloud/ocis/v2/services/policies/pkg/service/grpc"
"github.com/urfave/cli/v2"
)
// Server is the entrypoint for the server command.
func Server(cfg *config.Config) *cli.Command {
return &cli.Command{
Name: "server",
Usage: fmt.Sprintf("start the %s service without runtime (unsupervised mode)", "authz"),
Category: "server",
Before: func(c *cli.Context) error {
return configlog.ReturnFatal(parser.ParseConfig(cfg))
},
Action: func(c *cli.Context) error {
var (
gr = run.Group{}
ctx, cancel = func() (context.Context, context.CancelFunc) {
if cfg.Context == nil {
return context.WithCancel(context.Background())
}
return context.WithCancel(cfg.Context)
}()
logger = log.NewLogger(
log.Name(cfg.Service.Name),
log.Level(cfg.Log.Level),
log.Pretty(cfg.Log.Pretty),
log.Color(cfg.Log.Color),
log.File(cfg.Log.File),
)
)
defer cancel()
e, err := engine.NewOPA(cfg.Engine.Timeout, cfg.Engine)
if err != nil {
return err
}
{
err = grpc.Configure(grpc.GetClientOptions(cfg.GRPCClientTLS)...)
if err != nil {
return err
}
svc, err := grpc.NewService(
grpc.Logger(logger),
grpc.TLSEnabled(cfg.GRPC.TLS.Enabled),
grpc.TLSCert(
cfg.GRPC.TLS.Cert,
cfg.GRPC.TLS.Key,
),
grpc.Name(cfg.Service.Name),
grpc.Context(ctx),
grpc.Address(cfg.GRPC.Addr),
grpc.Namespace(cfg.GRPC.Namespace),
grpc.Version(version.GetString()),
)
if err != nil {
return err
}
grpcSvc, err := svcGRPC.New(e)
if err != nil {
return err
}
if err := svcProtogen.RegisterPoliciesProviderHandler(
svc.Server(),
grpcSvc,
); err != nil {
return err
}
gr.Add(svc.Run, func(_ error) {
cancel()
})
}
{
var tlsConf *tls.Config
if cfg.Events.EnableTLS {
var rootCAPool *x509.CertPool
if cfg.Events.TLSRootCACertificate != "" {
rootCrtFile, err := os.Open(cfg.Events.TLSRootCACertificate)
if err != nil {
return err
}
rootCAPool, err = ociscrypto.NewCertPoolFromPEM(rootCrtFile)
if err != nil {
return err
}
cfg.Events.TLSInsecure = false
}
tlsConf = &tls.Config{
RootCAs: rootCAPool,
}
}
bus, err := stream.Nats(
natsjs.TLSConfig(tlsConf),
natsjs.Address(cfg.Events.Endpoint),
natsjs.ClusterID(cfg.Events.Cluster),
)
if err != nil {
return err
}
eventSvc, err := svcEvent.New(bus, logger, e, cfg.Postprocessing.Query)
if err != nil {
return err
}
gr.Add(eventSvc.Run, func(_ error) {
cancel()
})
}
{
server := debug.NewService(
debug.Logger(logger),
debug.Name(cfg.Service.Name),
debug.Version(version.GetString()),
debug.Address(cfg.Debug.Addr),
debug.Token(cfg.Debug.Token),
debug.Pprof(cfg.Debug.Pprof),
debug.Zpages(cfg.Debug.Zpages),
debug.Health(
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
// TODO: check if services are up and running
_, err := io.WriteString(w, http.StatusText(http.StatusOK))
// io.WriteString should not fail but if it does we want to know.
if err != nil {
panic(err)
}
},
),
debug.Ready(
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
// TODO: check if services are up and running
_, err := io.WriteString(w, http.StatusText(http.StatusOK))
// io.WriteString should not fail but if it does we want to know.
if err != nil {
panic(err)
}
},
),
)
gr.Add(server.ListenAndServe, func(_ error) {
_ = server.Shutdown(ctx)
cancel()
})
}
return gr.Run()
},
}
}
+50
View File
@@ -0,0 +1,50 @@
package command
import (
"fmt"
"os"
"github.com/owncloud/ocis/v2/ocis-pkg/registry"
"github.com/owncloud/ocis/v2/ocis-pkg/version"
tw "github.com/olekukonko/tablewriter"
"github.com/owncloud/ocis/v2/services/policies/pkg/config"
"github.com/urfave/cli/v2"
)
// Version prints the service versions of all running instances.
func Version(cfg *config.Config) *cli.Command {
return &cli.Command{
Name: "version",
Usage: "print the version of this binary and the running service instances",
Category: "info",
Action: func(c *cli.Context) error {
fmt.Println("Version: " + version.GetString())
fmt.Printf("Compiled: %s\n", version.Compiled())
fmt.Println("")
reg := registry.GetRegistry()
services, err := reg.GetService(cfg.GRPC.Namespace + "." + cfg.Service.Name)
if err != nil {
fmt.Println(fmt.Errorf("could not get %s services from the registry: %v", cfg.Service.Name, err))
return err
}
if len(services) == 0 {
fmt.Println("No running " + cfg.Service.Name + " service found.")
return nil
}
table := tw.NewWriter(os.Stdout)
table.SetHeader([]string{"Version", "Address", "Id"})
table.SetAutoFormatHeaders(false)
for _, s := range services {
for _, n := range s.Nodes {
table.Append([]string{s.Version, n.Address, n.Id})
}
}
table.Render()
return nil
},
}
}
+77
View File
@@ -0,0 +1,77 @@
package config
import (
"context"
"github.com/owncloud/ocis/v2/ocis-pkg/shared"
"time"
)
// Config combines all available configuration parts.
type Config struct {
Commons *shared.Commons `yaml:"-"` // don't use this directly as configuration for a service
GRPC GRPC `yaml:"grpc"`
Service Service `yaml:"-"`
Debug Debug `yaml:"debug"`
TokenManager *TokenManager `yaml:"token_manager"`
Events Events `yaml:"events"`
Reva *shared.Reva `yaml:"reva"`
GRPCClientTLS *shared.GRPCClientTLS `yaml:"grpc_client_tls"`
MachineAuthAPIKey string `yaml:"machine_auth_api_key" env:"OCIS_MACHINE_AUTH_API_KEY;POLICIES_MACHINE_AUTH_API_KEY" desc:"Machine auth API key used to validate internal requests necessary for the access to resources from other services."`
Context context.Context `yaml:"-"`
Log *Log `yaml:"log"`
Engine Engine `yaml:"engine"`
Postprocessing Postprocessing `yaml:"postprocessing"`
}
// Service defines the available service configuration.
type Service struct {
Name string `yaml:"-"`
}
// GRPC defines the available grpc configuration.
type GRPC struct {
Addr string `ocisConfig:"addr" env:"POLICIES_GRPC_ADDR" desc:"The bind address of the GRPC service."`
Namespace string `ocisConfig:"-" yaml:"-"`
TLS *shared.GRPCServiceTLS `yaml:"tls"`
}
// TokenManager is the config for using the reva token manager
type TokenManager struct {
JWTSecret string `yaml:"jwt_secret" env:"OCIS_JWT_SECRET;POLICIES_JWT_SECRET" desc:"The secret to mint and validate jwt tokens."`
}
// Engine configures the policy engine.
type Engine struct {
Timeout time.Duration `yaml:"timeout" env:"POLICIES_ENGINE_TIMEOUT" desc:"Sets the timeout the rego expression evaluation can take. The timeout can be set as number followed by a unit identifier like ms, s, etc. Rules default to deny if the timeout was reached."`
Policies []string `yaml:"policies"`
}
// Postprocessing defines the config options for the postprocessing policy handling.
type Postprocessing struct {
Query string `yaml:"query" env:"POLICIES_POSTPROCESSING_QUERY" desc:"Defines the 'Complete Rules' variable defined in the rego rule set this step uses for its evaluation. Defaults to deny if the variable was not found."`
}
// Events combines the configuration options for the event bus.
type Events struct {
Endpoint string `yaml:"endpoint" env:"POLICIES_EVENTS_ENDPOINT" desc:"The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture."`
Cluster string `yaml:"cluster" env:"POLICIES_EVENTS_CLUSTER" desc:"The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system."`
TLSInsecure bool `yaml:"tls_insecure" env:"OCIS_INSECURE;POLICIES_EVENTS_TLS_INSECURE" desc:"Whether the server should skip the client certificate verification during the TLS handshake."`
TLSRootCACertificate string `yaml:"tls_root_ca_certificate" env:"POLICIES_EVENTS_TLS_ROOT_CA_CERTIFICATE" desc:"The root CA certificate used to validate the server's TLS certificate. If provided POLICIES_EVENTS_TLS_INSECURE will be seen as false."`
EnableTLS bool `yaml:"enable_tls" env:"OCIS_EVENTS_ENABLE_TLS;POLICIES_EVENTS_ENABLE_TLS" desc:"Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services."`
}
// Log defines the available log configuration.
type Log struct {
Level string `mapstructure:"level" env:"OCIS_LOG_LEVEL;POLICIES_LOG_LEVEL" desc:"The log level. Valid values are: \"panic\", \"fatal\", \"error\", \"warn\", \"info\", \"debug\", \"trace\"."`
Pretty bool `mapstructure:"pretty" env:"OCIS_LOG_PRETTY;POLICIES_LOG_PRETTY" desc:"Activates pretty log output."`
Color bool `mapstructure:"color" env:"OCIS_LOG_COLOR;POLICIES_LOG_COLOR" desc:"Activates colorized log output."`
File string `mapstructure:"file" env:"OCIS_LOG_FILE;POLICIES_LOG_FILE" desc:"The path to the log file. Activates logging to this file if set."`
}
// Debug defines the available debug configuration.
type Debug struct {
Addr string `yaml:"addr" env:"POLICIES_DEBUG_ADDR" desc:"Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed."`
Token string `yaml:"token" env:"POLICIES_DEBUG_TOKEN" desc:"Token to secure the metrics endpoint."`
Pprof bool `yaml:"pprof" env:"POLICIES_DEBUG_PPROF" desc:"Enables pprof, which can be used for profiling."`
Zpages bool `yaml:"zpages" env:"POLICIES_DEBUG_ZPAGES" desc:"Enables zpages, which can be used for collecting and viewing in-memory traces."`
}
@@ -0,0 +1,87 @@
package defaults
import (
"time"
"github.com/owncloud/ocis/v2/ocis-pkg/shared"
"github.com/owncloud/ocis/v2/ocis-pkg/structs"
"github.com/owncloud/ocis/v2/services/policies/pkg/config"
)
// FullDefaultConfig returns a fully initialized default configuration which is needed for doc generation.
func FullDefaultConfig() *config.Config {
cfg := DefaultConfig()
EnsureDefaults(cfg)
Sanitize(cfg)
return cfg
}
// DefaultConfig returns the default config
func DefaultConfig() *config.Config {
return &config.Config{
Service: config.Service{
Name: "policies",
},
Debug: config.Debug{
Addr: "127.0.0.1:9129",
Token: "",
Pprof: false,
Zpages: false,
},
GRPC: config.GRPC{
Addr: "127.0.0.1:9125",
Namespace: "com.owncloud.api",
},
Reva: shared.DefaultRevaConfig(),
Events: config.Events{
Endpoint: "127.0.0.1:9233",
Cluster: "ocis-cluster",
EnableTLS: false,
},
Engine: config.Engine{
Timeout: 10 * time.Second,
},
}
}
func EnsureDefaults(cfg *config.Config) {
if cfg.TokenManager == nil && cfg.Commons != nil && cfg.Commons.TokenManager != nil {
cfg.TokenManager = &config.TokenManager{
JWTSecret: cfg.Commons.TokenManager.JWTSecret,
}
} else if cfg.TokenManager == nil {
cfg.TokenManager = &config.TokenManager{}
}
if cfg.MachineAuthAPIKey == "" && cfg.Commons != nil && cfg.Commons.MachineAuthAPIKey != "" {
cfg.MachineAuthAPIKey = cfg.Commons.MachineAuthAPIKey
}
if cfg.Reva == nil && cfg.Commons != nil && cfg.Commons.Reva != nil {
cfg.Reva = &shared.Reva{
Address: cfg.Commons.Reva.Address,
TLS: cfg.Commons.Reva.TLS,
}
} else if cfg.Reva == nil {
cfg.Reva = &shared.Reva{}
}
if cfg.Log == nil && cfg.Commons != nil && cfg.Commons.Log != nil {
cfg.Log = &config.Log{
Level: cfg.Commons.Log.Level,
Pretty: cfg.Commons.Log.Pretty,
Color: cfg.Commons.Log.Color,
File: cfg.Commons.Log.File,
}
} else if cfg.Log == nil {
cfg.Log = &config.Log{}
}
if cfg.GRPCClientTLS == nil && cfg.Commons != nil {
cfg.GRPCClientTLS = structs.CopyOrZeroValue(cfg.Commons.GRPCClientTLS)
}
if cfg.GRPC.TLS == nil && cfg.Commons != nil {
cfg.GRPC.TLS = structs.CopyOrZeroValue(cfg.Commons.GRPCServiceTLS)
}
}
func Sanitize(_ *config.Config) {}
@@ -0,0 +1,42 @@
package parser
import (
"errors"
ociscfg "github.com/owncloud/ocis/v2/ocis-pkg/config"
"github.com/owncloud/ocis/v2/ocis-pkg/shared"
"github.com/owncloud/ocis/v2/services/policies/pkg/config"
"github.com/owncloud/ocis/v2/services/policies/pkg/config/defaults"
"github.com/owncloud/ocis/v2/ocis-pkg/config/envdecode"
)
// ParseConfig loads configuration from known paths.
func ParseConfig(cfg *config.Config) error {
_, err := ociscfg.BindSourcesToStructs(cfg.Service.Name, cfg)
if err != nil {
return err
}
defaults.EnsureDefaults(cfg)
// load all env variables relevant to the config in the current context.
if err := envdecode.Decode(cfg); err != nil {
// no environment variable set for this config is an expected "error"
if !errors.Is(err, envdecode.ErrNoTargetFieldsAreSet) {
return err
}
}
defaults.Sanitize(cfg)
return Validate(cfg)
}
func Validate(cfg *config.Config) error {
if cfg.TokenManager.JWTSecret == "" {
return shared.MissingJWTTokenError(cfg.Service.Name)
}
return nil
}
+74
View File
@@ -0,0 +1,74 @@
package engine
import (
"context"
"encoding/json"
user "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
"github.com/owncloud/ocis/v2/protogen/gen/ocis/messages/policies/v0"
"google.golang.org/protobuf/encoding/protojson"
)
// Engine defines the granted handlers.
type Engine interface {
Evaluate(ctx context.Context, query string, env Environment) (bool, error)
}
type (
// Stage defines the used auth stage
Stage string
)
var (
// StagePP defines the post-processing stage
StagePP Stage = "pp"
// StageHTTP defines the http stage
StageHTTP Stage = "http"
)
// Resource contains resource information and is used as part of the evaluated environment.
type Resource struct {
ID provider.ResourceId `json:"resource_id"`
Name string `json:"name"`
URL string `json:"url"`
Size uint64 `json:"size"`
}
// Request contains request information and is used as part of the evaluated environment.
type Request struct {
Method string `json:"method"`
Path string `json:"path"`
}
// Environment contains every data that is needed to decide if the request should pass or not
type Environment struct {
Stage Stage `json:"stage"`
User user.User `json:"user"`
Request Request `json:"request"`
Resource Resource `json:"resource"`
}
// NewEnvironmentFromPB converts a PBEnvironment to Environment.
func NewEnvironmentFromPB(pEnv *v0.Environment) (Environment, error) {
env := Environment{}
rData, err := protojson.Marshal(pEnv)
if err != nil {
return env, err
}
if err := json.Unmarshal(rData, &env); err != nil {
return env, err
}
switch pEnv.Stage {
case v0.Stage_STAGE_HTTP:
env.Stage = StageHTTP
case v0.Stage_STAGE_PP:
env.Stage = StagePP
}
return env, nil
}
@@ -0,0 +1,13 @@
package engine_test
import (
"testing"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
func TestEngine(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Engine Suite")
}
@@ -0,0 +1,25 @@
package engine_test
import (
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
pMessage "github.com/owncloud/ocis/v2/protogen/gen/ocis/messages/policies/v0"
"github.com/owncloud/ocis/v2/services/policies/pkg/engine"
)
var _ = Describe("Engine", func() {
DescribeTable("NewEnvironmentFromPB",
func(incomingStage pMessage.Stage, outgoinStage engine.Stage) {
pEnv := &pMessage.Environment{
Stage: incomingStage,
}
env, err := engine.NewEnvironmentFromPB(pEnv)
Expect(err).ToNot(HaveOccurred())
Expect(env.Stage).To(Equal(outgoinStage))
},
Entry("http stage", pMessage.Stage_STAGE_HTTP, engine.StageHTTP),
Entry("pp stage", pMessage.Stage_STAGE_PP, engine.StagePP),
)
})
+124
View File
@@ -0,0 +1,124 @@
package engine
import (
"bytes"
"context"
"fmt"
"net/http"
"strings"
"time"
"github.com/cs3org/reva/v2/pkg/rhttp"
"github.com/gabriel-vasile/mimetype"
"github.com/open-policy-agent/opa/ast"
"github.com/open-policy-agent/opa/rego"
"github.com/open-policy-agent/opa/types"
"github.com/owncloud/ocis/v2/services/policies/pkg/config"
)
// OPA wraps open policy agent makes it possible to ask if an action is granted.
type OPA struct {
policies []string
timeout time.Duration
}
// NewOPA returns a ready to use opa engine.
func NewOPA(timeout time.Duration, conf config.Engine) (OPA, error) {
return OPA{
policies: conf.Policies,
timeout: timeout,
},
nil
}
// Evaluate evaluates the opa policies and returns the result.
func (o OPA) Evaluate(ctx context.Context, qs string, env Environment) (bool, error) {
ctx, cancel := context.WithTimeout(ctx, o.timeout)
defer cancel()
q, err := rego.New(
rego.Query(qs),
rego.Load(o.policies, nil),
GetMimetype,
GetResource,
).PrepareForEval(ctx)
if err != nil {
return false, err
}
result, err := q.Eval(ctx, rego.EvalInput(env))
if err != nil {
return false, err
}
return result.Allowed(), nil
}
var GetResource = rego.Function1(
&rego.Function{
Name: "ocis_get_resource",
Decl: types.NewFunction(types.Args(types.S), types.A),
Memoize: true,
Nondeterministic: true,
},
func(_ rego.BuiltinContext, a *ast.Term) (*ast.Term, error) {
var url string
if err := ast.As(a.Value, &url); err != nil {
return nil, err
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return nil, err
}
client := rhttp.GetHTTPClient(rhttp.Insecure(true))
res, err := client.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("unexpected status code from Download %v", res.StatusCode)
}
buf := new(bytes.Buffer)
if _, err := buf.ReadFrom(res.Body); err != nil {
return nil, err
}
v, err := ast.InterfaceToValue(buf.Bytes())
if err != nil {
return nil, err
}
return ast.NewTerm(v), nil
},
)
var GetMimetype = rego.Function1(
&rego.Function{
Name: "ocis_get_mimetype",
Decl: types.NewFunction(types.Args(types.A), types.S),
Memoize: true,
Nondeterministic: true,
},
func(_ rego.BuiltinContext, a *ast.Term) (*ast.Term, error) {
var body []byte
if err := ast.As(a.Value, &body); err != nil {
return nil, err
}
mimeInfo := mimetype.Detect(body).String()
detectedMimetype := strings.Split(mimeInfo, ";")[0]
v, err := ast.InterfaceToValue(detectedMimetype)
if err != nil {
return nil, err
}
return ast.NewTerm(v), nil
},
)
+46
View File
@@ -0,0 +1,46 @@
package engine_test
import (
"context"
"encoding/base64"
"net/http"
"net/http/httptest"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/open-policy-agent/opa/rego"
"github.com/owncloud/ocis/v2/services/policies/pkg/engine"
)
var _ = Describe("Opa", func() {
Describe("Custom OPA function", func() {
Describe("GetResource", func() {
It("loads reva resources", func() {
ts := []byte("Lorem Ipsum is simply dummy text of the printing and typesetting")
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write(ts)
}))
defer srv.Close()
r := rego.New(rego.Query(`ocis_get_resource("`+srv.URL+`")`), engine.GetResource)
rs, err := r.Eval(context.Background())
Expect(err).ToNot(HaveOccurred())
data, err := base64.StdEncoding.DecodeString(rs[0].Expressions[0].String())
Expect(err).ToNot(HaveOccurred())
Expect(data).To(Equal(ts))
})
})
Describe("GetMimetype", func() {
It("is defined and returns a mimetype", func() {
r := rego.New(rego.Query(`ocis_get_mimetype("")`), engine.GetMimetype)
rs, err := r.Eval(context.Background())
Expect(err).ToNot(HaveOccurred())
Expect(rs[0].Expressions[0].String()).To(Equal("text/plain"))
})
})
})
})
@@ -0,0 +1,87 @@
package eventSVC
import (
"context"
"github.com/cs3org/reva/v2/pkg/events"
"github.com/owncloud/ocis/v2/ocis-pkg/log"
"github.com/owncloud/ocis/v2/services/policies/pkg/engine"
)
// Service defines the service handlers.
type Service struct {
query string
log log.Logger
stream events.Stream
engine engine.Engine
}
// New returns a service implementation for Service.
func New(stream events.Stream, logger log.Logger, engine engine.Engine, query string) (Service, error) {
svc := Service{
log: logger,
query: query,
engine: engine,
stream: stream,
}
return svc, nil
}
// Run to fulfil Runner interface
func (s Service) Run() error {
ch, err := events.Consume(s.stream, "policies", events.StartPostprocessingStep{})
if err != nil {
return err
}
for e := range ch {
switch ev := e.Event.(type) {
case events.StartPostprocessingStep:
if ev.StepToStart != "policies" {
continue
}
outcome := events.PPOutcomeContinue
if s.query != "" {
env := engine.Environment{
Stage: engine.StagePP,
Resource: engine.Resource{
Name: ev.Filename,
URL: ev.URL,
Size: ev.Filesize,
},
}
if ev.ExecutingUser != nil {
env.User = *ev.ExecutingUser
}
if ev.ResourceID != nil {
env.Resource.ID = *ev.ResourceID
}
result, err := s.engine.Evaluate(context.TODO(), s.query, env)
if err != nil {
s.log.Error().Err(err).Msg("unable evaluate policy")
}
if !result {
outcome = events.PPOutcomeDelete
}
}
if err := events.Publish(s.stream, events.PostprocessingStepFinished{
Outcome: outcome,
UploadID: ev.UploadID,
ExecutingUser: ev.ExecutingUser,
Filename: ev.Filename,
FinishedStep: ev.StepToStart,
}); err != nil {
return err
}
}
}
return nil
}
@@ -0,0 +1,35 @@
package grpcSVC
import (
"context"
"github.com/owncloud/ocis/v2/protogen/gen/ocis/services/policies/v0"
"github.com/owncloud/ocis/v2/services/policies/pkg/engine"
)
// Service defines the service handlers.
type Service struct {
engine engine.Engine
}
// New returns a service implementation for Service.
func New(engine engine.Engine) (Service, error) {
svc := Service{
engine: engine,
}
return svc, nil
}
// Evaluate exposes the engine policy evaluation.
func (s Service) Evaluate(ctx context.Context, request *v0.EvaluateRequest, response *v0.EvaluateResponse) error {
env, err := engine.NewEnvironmentFromPB(request.Environment)
if err != nil {
return err
}
result, err := s.engine.Evaluate(ctx, request.Query, env)
response.Result = result
return err
}
+1 -1
View File
@@ -23,7 +23,7 @@ type Config struct {
// Postprocessing defines the config options for the postprocessing service.
type Postprocessing struct {
Events Events `yaml:"events"`
Steps []string `yaml:"steps" env:"POSTPROCESSING_STEPS" desc:"A comma separated list of postprocessing steps, processed in order of their appearance. Currently supported values by the system are: 'virusscan' and 'delay'. Custom steps are allowed. See the documentation for instructions."`
Steps []string `yaml:"steps" env:"POSTPROCESSING_STEPS" desc:"A comma separated list of postprocessing steps, processed in order of their appearance. Currently supported values by the system are: 'virusscan', 'policies' and 'delay'. Custom steps are allowed. See the documentation for instructions."`
Virusscan bool `yaml:"virusscan" env:"POSTPROCESSING_VIRUSSCAN" desc:"After uploading a file but before making it available for download, virus scanning the file can be enabled. Needs as prerequisite the antivirus service to be enabled and configured." deprecationVersion:"master" removalVersion:"master" deprecationInfo:"POSTPROCESSING_VIRUSSCAN is not longer necessary and is replaced by POSTPROCESSING_STEPS which also holds information about the order of steps" deprecationReplacement:"POSTPROCESSING_STEPS"`
Delayprocessing time.Duration `yaml:"delayprocessing" env:"POSTPROCESSING_DELAY" desc:"After uploading a file but before making it available for download, a delay step can be added. Intended for developing purposes only. The duration can be set as number followed by a unit identifier like s, m or h. If a duration is set but the keyword 'delay' is not explicitely added to 'POSTPROCESSING_STEPS', the delay step will be processed as last step. In such a case, a log entry will be written on service startup to remind the admin about that situation."`
}
@@ -23,6 +23,7 @@ func NewPostprocessingService(stream events.Stream, logger log.Logger, c config.
events.StartPostprocessingStep{},
events.VirusscanFinished{},
events.UploadReady{},
events.PostprocessingStepFinished{},
)
if err != nil {
return nil, err
+1 -5
View File
@@ -199,7 +199,6 @@ func loadMiddlewares(ctx context.Context, logger log.Logger, cfg *config.Config)
Logger: logger,
RevaGatewayClient: revaClient,
})
authenticators = append(authenticators, middleware.SignedURLAuthenticator{
Logger: logger,
PreSignedURLConfig: cfg.PreSignedURL,
@@ -219,9 +218,7 @@ func loadMiddlewares(ctx context.Context, logger log.Logger, cfg *config.Config)
cfg.OIDC.RewriteWellKnown,
oidcHTTPClient,
),
router.Middleware(cfg.PolicySelector, cfg.Policies, logger),
middleware.Authentication(
authenticators,
middleware.CredentialsByUserAgent(cfg.AuthMiddleware.CredentialsByUserAgent),
@@ -237,13 +234,12 @@ func loadMiddlewares(ctx context.Context, logger log.Logger, cfg *config.Config)
middleware.UserCS3Claim(cfg.UserCS3Claim),
middleware.AutoprovisionAccounts(cfg.AutoprovisionAccounts),
),
middleware.SelectorCookie(
middleware.Logger(logger),
middleware.UserProvider(userProvider),
middleware.PolicySelectorConfig(*cfg.PolicySelector),
),
middleware.Policies(logger, cfg.PoliciesMiddleware.Query),
// finally, trigger home creation when a user logs in
middleware.CreateHome(
middleware.Logger(logger),
+21 -15
View File
@@ -21,21 +21,22 @@ type Config struct {
Reva *shared.Reva `yaml:"reva"`
GRPCClientTLS *shared.GRPCClientTLS `yaml:"grpc_client_tls"`
RoleQuotas map[string]uint64 `yaml:"role_quotas"`
Policies []Policy `yaml:"policies"`
OIDC OIDC `yaml:"oidc"`
TokenManager *TokenManager `mask:"struct" yaml:"token_manager"`
PolicySelector *PolicySelector `yaml:"policy_selector"`
PreSignedURL PreSignedURL `yaml:"pre_signed_url"`
AccountBackend string `yaml:"account_backend" env:"PROXY_ACCOUNT_BACKEND_TYPE" desc:"Account backend the PROXY service should use. Currently only 'cs3' is possible here."`
UserOIDCClaim string `yaml:"user_oidc_claim" env:"PROXY_USER_OIDC_CLAIM" desc:"The name of an OpenID Connect claim that is used for resolving users with the account backend. The value of the claim must hold a per user unique, stable and non re-assignable identifier. The availability of claims depends on your Identity Provider. There are common claims available for most Identity providers like 'email' or 'preferred_user' but you can also add your own claim."`
UserCS3Claim string `yaml:"user_cs3_claim" env:"PROXY_USER_CS3_CLAIM" desc:"The name of a CS3 user attribute (claim) that should be mapped to the 'user_oidc_claim'. Supported values are 'username', 'mail' and 'userid'."`
MachineAuthAPIKey string `mask:"password" yaml:"machine_auth_api_key" env:"OCIS_MACHINE_AUTH_API_KEY;PROXY_MACHINE_AUTH_API_KEY" desc:"Machine auth API key used to validate internal requests necessary to access resources from other services."`
AutoprovisionAccounts bool `yaml:"auto_provision_accounts" env:"PROXY_AUTOPROVISION_ACCOUNTS" desc:"Set this to 'true' to automatically provision users that do not yet exist in the users service on-demand upon first sign-in. To use this a write-enabled libregraph user backend needs to be setup an running."`
EnableBasicAuth bool `yaml:"enable_basic_auth" env:"PROXY_ENABLE_BASIC_AUTH" desc:"Set this to true to enable 'basic authentication' (username/password)."`
InsecureBackends bool `yaml:"insecure_backends" env:"PROXY_INSECURE_BACKENDS" desc:"Disable TLS certificate validation for all HTTP backend connections."`
BackendHTTPSCACert string `yaml:"backend_https_cacert" env:"PROXY_HTTPS_CACERT" desc:"Path/File for the root CA certificate used to validate the servers TLS certificate for https enabled backend services."`
AuthMiddleware AuthMiddleware `yaml:"auth_middleware"`
RoleQuotas map[string]uint64 `yaml:"role_quotas"`
Policies []Policy `yaml:"policies"`
OIDC OIDC `yaml:"oidc"`
TokenManager *TokenManager `mask:"struct" yaml:"token_manager"`
PolicySelector *PolicySelector `yaml:"policy_selector"`
PreSignedURL PreSignedURL `yaml:"pre_signed_url"`
AccountBackend string `yaml:"account_backend" env:"PROXY_ACCOUNT_BACKEND_TYPE" desc:"Account backend the PROXY service should use. Currently only 'cs3' is possible here."`
UserOIDCClaim string `yaml:"user_oidc_claim" env:"PROXY_USER_OIDC_CLAIM" desc:"The name of an OpenID Connect claim that is used for resolving users with the account backend. The value of the claim must hold a per user unique, stable and non re-assignable identifier. The availability of claims depends on your Identity Provider. There are common claims available for most Identity providers like 'email' or 'preferred_user' but you can also add your own claim."`
UserCS3Claim string `yaml:"user_cs3_claim" env:"PROXY_USER_CS3_CLAIM" desc:"The name of a CS3 user attribute (claim) that should be mapped to the 'user_oidc_claim'. Supported values are 'username', 'mail' and 'userid'."`
MachineAuthAPIKey string `mask:"password" yaml:"machine_auth_api_key" env:"OCIS_MACHINE_AUTH_API_KEY;PROXY_MACHINE_AUTH_API_KEY" desc:"Machine auth API key used to validate internal requests necessary to access resources from other services."`
AutoprovisionAccounts bool `yaml:"auto_provision_accounts" env:"PROXY_AUTOPROVISION_ACCOUNTS" desc:"Set this to 'true' to automatically provision users that do not yet exist in the users service on-demand upon first sign-in. To use this a write-enabled libregraph user backend needs to be setup an running."`
EnableBasicAuth bool `yaml:"enable_basic_auth" env:"PROXY_ENABLE_BASIC_AUTH" desc:"Set this to true to enable 'basic authentication' (username/password)."`
InsecureBackends bool `yaml:"insecure_backends" env:"PROXY_INSECURE_BACKENDS" desc:"Disable TLS certificate validation for all HTTP backend connections."`
BackendHTTPSCACert string `yaml:"backend_https_cacert" env:"PROXY_HTTPS_CACERT" desc:"Path/File for the root CA certificate used to validate the servers TLS certificate for https enabled backend services."`
AuthMiddleware AuthMiddleware `yaml:"auth_middleware"`
PoliciesMiddleware PoliciesMiddleware `yaml:"policies_middleware"`
Context context.Context `yaml:"-" json:"-"`
}
@@ -84,6 +85,11 @@ type AuthMiddleware struct {
CredentialsByUserAgent map[string]string `yaml:"credentials_by_user_agent"`
}
// PoliciesMiddleware configures the proxy policies middleware.
type PoliciesMiddleware struct {
Query string `yaml:"query" env:"PROXY_POLICIES_QUERY" desc:"Defines the 'Complete Rules' variable defined in the rego rule set this step uses for its evaluation. Rules default to deny if the variable was not found."`
}
const (
AccessTokenVerificationNone = "none"
AccessTokenVerificationJWT = "jwt"
+62
View File
@@ -0,0 +1,62 @@
package middleware
import (
"net/http"
revactx "github.com/cs3org/reva/v2/pkg/ctx"
"github.com/owncloud/ocis/v2/ocis-pkg/log"
"github.com/owncloud/ocis/v2/ocis-pkg/service/grpc"
pMessage "github.com/owncloud/ocis/v2/protogen/gen/ocis/messages/policies/v0"
pService "github.com/owncloud/ocis/v2/protogen/gen/ocis/services/policies/v0"
)
// Policies verifies if a request is granted or not.
func Policies(logger log.Logger, qs string) func(next http.Handler) http.Handler {
pClient := pService.NewPoliciesProviderService("com.owncloud.api.policies", grpc.DefaultClient())
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if qs == "" {
next.ServeHTTP(w, r)
return
}
req := &pService.EvaluateRequest{
Query: qs,
Environment: &pMessage.Environment{
Request: &pMessage.Request{
Method: r.Method,
Path: r.URL.Path,
},
Stage: pMessage.Stage_STAGE_HTTP,
},
}
if user, ok := revactx.ContextGetUser(r.Context()); ok {
req.Environment.User = &pMessage.User{
Id: &pMessage.User_ID{
OpaqueId: user.GetId().GetOpaqueId(),
},
Username: user.GetUsername(),
Mail: user.GetMail(),
DisplayName: user.GetDisplayName(),
Groups: user.GetGroups(),
}
}
rsp, err := pClient.Evaluate(r.Context(), req)
if err != nil {
logger.Err(err).Msg("error evaluating request")
w.WriteHeader(http.StatusInternalServerError)
return
}
if !rsp.Result {
w.WriteHeader(http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
})
}
}