merge master

2021-12-10 14:38:46 +01:00
parent 27863fdb43 43a14f50e2
commit 97377e5b21
783 changed files with 59360 additions and 45942 deletions
@@ -19,7 +19,7 @@ docs-init:
 	@cd hugo && git remote add origin https://github.com/owncloud/owncloud.github.io
 	@cd hugo && git fetch --depth=1
 	@cd hugo && git checkout origin/source -f
-	@make -C hugo theme
+	@$(MAKE) -C hugo theme

 .PHONY: docs-serve
 docs-serve: docs-init docs-generate docs-copy ## serve docs with hugo
@@ -0,0 +1,12 @@
+---
+title: "Architecture"
+date: 2020-03-03T10:34:00+01:00
+weight: 25
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/architecture
+geekdocFilePath: _index.md
+---
+
+In the architecture part of the documentation we collect useful developer documentation on different aspects of the architecture. We are using mermaid.js to collaborate on the necessardy diagrams.
+
+*Pictures tell more than a thousand words.*
@@ -0,0 +1,202 @@
+---
+title: "Efficient Stat Polling"
+date: 2020-03-03T10:31:00+01:00
+weight: 30
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/architecture
+geekdocFilePath: efficient-stat-polling.md
+---
+
+The fallback sync mechanism uses the ETag to determine which part of a sync tree needs to be checked by recursively descending into folders whose ETag has changed. The ETag can be calculated using a `stat()` call in the filesystem and we are going to explore how many `stat()` calls are necessary and how the number might be reduced.
+
+## ETag propagation
+
+What does ETag propagation mean? Whenever a file changes its content or metadata the ETag or "entity tag" changes. In the early days of ownCloud it was decided to extend this behavior to folders as well, which is outside of any WebDAV RFC specification. Nevertheless, here we are, using the ETag to reflect changes, not only on WebDAV resources but also WebDAV collections. The server will propagate the ETag change up to the root of the tree.
+
+{{<mermaid class="text-center">}}
+graph TD
+  linkStyle default interpolate basis
+  
+  subgraph final ETag propagation
+    ert3(( etag:N )) --- el3(( etag:O )) & er3(( etag:N ))
+    er3 --- erl3(( etag:O )) & err3(( etag:N ))
+  end
+
+  subgraph first ETag propagation
+    ert2(( etag:O )) --- el2(( etag:O )) & er2(( etag:N ))
+    er2 --- erl2(( etag:O )) & err2(( etag:N ))
+  end
+
+  subgraph initial file change 
+    ert(( etag:O )) --- el(( etag:O )) & er(( etag:O ))
+    er --- erl(( etag:O )) & err(( etag:N ))
+  end
+{{</mermaid>}}
+
+The old `etag:O` is replaced by propagating the new `etag:N` up to the root, where the client will pick it up and explore the tree by comparing the old ETags known to him with the state of the current ETags on the server. This form of sync is called *state based sync*.
+
+## Single user sync
+To let the client detect changes in the drive (a tree of files and folders) of a user, we rely on the ETag of every node in the tree. The discovery phase starts at the root of the tree and checks if the ETag has changed since the last discovery:
+- if it is still the same nothing has changed inside the tree
+- if it changed the client will compare the ETag of all immediate children and recursively descend into every node that changed
+
+This works, because the server side will propagate ETag changes in the tree up to the root.
+
+{{<mermaid class="text-center">}}
+graph TD
+  linkStyle default interpolate basis
+
+  ec( client ) -->|"stat()"|ert
+
+  subgraph  
+    ert(( )) --- el(( )) & er(( ))
+    er --- erl(( )) & err(( ))  
+  end
+{{</mermaid>}}
+
+## Multiple users
+On an ocis server there is not one user but many. Each of them may have one or more clients running. In the worst case all of them polling the ETag of his home root node every 30 seconds.
+
+Keep in mind that etags are only propagated inside each distinct tree. No sharing is considered yet.
+
+{{<mermaid class="text-center">}}
+graph TD
+  linkStyle default interpolate basis
+
+  ec( client ) -->|"stat()"|ert
+
+  subgraph  
+    ert(( )) --- el(( )) & er(( ))
+    er --- erl(( )) & err(( ))
+  end
+  
+  mc( client ) -->|"stat()"|mrt
+
+  subgraph  
+    mrt(( )) --- ml(( )) & mr(( ))
+    mr --- mrl(( )) & mrr(( ))
+  end
+
+  fc( client ) -->|"stat()"|frt
+
+  subgraph  
+    frt(( )) --- fl(( )) & fr(( ))
+    fr --- frl(( )) & frr(( ))
+  end
+{{</mermaid>}}
+
+## Sharing
+*Storage providers* are responsible for persisting shares as close to the storage as possible.
+
+One implementation may persist shares using ACLs, another might use custom extended attributes. The chosen implementation is storage specific and always a tradeoff between various requirements. Yet, the goal is to treat the storage provider as the single source of truth for all metadata. 
+
+If users can bypass the storage provider using eg. `ssh` additional mechanisms needs to make sure no inconsistencies arise:
+- the ETag must still be propagated in a tree, eg using inotify, a policy engine or workflows triggered by other means
+- deleted files should land in the trash (eg. `rm` could be wrapped to move files to trash)
+- overwriting files should create a new version ... other than a fuse fs I see no way of providing this for normal posix filesystems. Other storage backends that use the s3 protocol might provide versions natively.
+
+The storage provider is also responsible for keeps track of references eg. using a shadow tree that users normally cannot see or representing them as symbolic links in the filesystem (Beware of symbolic link cycles. The clients are currently unaware of them and would flood the filesystem).
+
+To prevent write amplification ETags must not propagate across references. When a file that was shared by einstein changes the ETag must not be propagated into any share recipients tree.
+
+{{<mermaid class="text-center">}}
+graph TD
+  linkStyle default interpolate basis
+
+
+  ec( einsteins client ) -->|"stat()"|ert
+
+  subgraph  
+    ml --- mlr(( ))
+    mrt(( )) --- ml(( )) & mr(( ))
+    mr --- mrl(( )) & mrr(( ))
+  end
+
+  mlr -. reference .-> er
+
+  subgraph  
+    ert(( )) --- el(( )) & er(( ))
+    er --- erl(( )) & err(( ))
+  end
+  
+  mc( maries client ) -->|"stat()"|mrt
+
+{{</mermaid>}}
+
+But how can maries client detect the change?
+
+We are trading writes for reads: the client needs to stat the own tree & all shares or entry points into other storage trees.
+
+It would require client changes that depend on the server side actually having an endpoint that can efficiently list all entry points into storages a user has access to including their current etag.
+
+But having to list n storages might become a bottleneck anyway, so we are going to have the gateway calculate a virtual root ETag for all entry points a user has access to and cache that.
+
+## Server Side Stat Polling
+Every client polls the virtual root ETag (every 30 sec). The gateway will cache the virutal root ETag of every storage for 30 sec as well. That way every storage provider is only stated once every 30 sec (can be throttled dynamically to adapt to storage io load).
+
+
+{{<mermaid class="text-center">}}
+graph TD
+  linkStyle default interpolate basis
+
+  ec( client ) -->|"stat()"|evr
+
+  subgraph gateway caching virtual etags
+    evr(( ))
+    mvr(( ))
+    fvr(( ))
+  end
+
+  evr --- ert
+  mvr --- mrt
+  fvr --- frt
+
+  subgraph  
+    ert(( )) --- el(( )) & er(( ))
+    er --- erl(( )) & err(( ))
+  end
+  
+  mc( client ) -->|"stat()"|mvr
+
+  subgraph  
+    mrt(( )) --- ml(( )) & mr(( ))
+    ml --- mlm(( ))
+    mr --- mrl(( )) & mrr(( ))
+  end
+
+  mlm -.- er
+  mvr -.- er
+
+  fc( client ) -->|"stat()"|fvr
+
+  subgraph  
+    frt(( )) --- fl(( )) & fr(( ))
+    fr --- frl(( )) & frr(( ))
+  end
+
+{{</mermaid>}}
+
+Since the active clients will poll the etag for all active users the gateway will have their ETag cached. This is where sharing comes into play: The gateway also needs to stat the ETag of all other entry points ... or mount points. That may increases the number of stat like requests to storage providers by an order of magnitude.
+
+### Ram considerations
+
+For a single machine using a local posix storage the linux kernel already caches the inodes that contain the metadata that is necessary to calculate the ETag (even extended attributes are supported). With 4k inodes 256 nodes take 1Mb of RAM, 1k inodes take 4Mb and 1M inodes take 4Gb to completely cache the file metadata. For distributed filesystems a dedicated cache might make sense to prevent hammering it with stat like requests to calculate ETags.
+
+### Bandwith considerations
+
+The bandwith for a single machine might be another bottleneck. Consider a propfind request with roughly 500 bytes and a response with roughly 800 bytes in size:
+- At 100Mbit (~10Mb/s) you can receive 20 000 PROPFIND requests
+- At 1000Mbit (~100Mb/s) you can receive 200 000 PROPFIND requests
+- At 10Gbit (~1Gb/s) you can receive 2 000 000 PROPFIND requests
+
+This can be scaled by adding more gateways and sharding users because these components are stateless.
+
+## Share mount point polling cache
+What can we do to reduce the number of stat calls to storage providers. Well, the gateway queries the share manager for all mounted shares of a user (or all entry points, not only the users own root/home). The share references contain the storage provider that contains the share. If every user has it's own storage provider id the gateway could check in its own cache if the storage root etag has changed. It will be up to date because another client likely already polled for its etag.
+This would reduce the number of necessary stat requests to active storages.
+
+### Active share node cache invalidation
+We can extend the lifetime of share ETag cache entries and only invalidate them when the root of the storage that contains them changes its ETag. That would reduce the number of stat requests to the number of active users.
+
+### Push notifications
+We can further enhance this by sending push notifications when the root of a storage changes. Which is becoming increasingly necessary for mobile devices anyway.
@@ -0,0 +1,25 @@
+---
+title: Rclone
+date: 2021-11-17T00:00:00+00:00
+weight: 20
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/clients/rclone
+geekdocFilePath: _index.md
+geekdocCollapseSection: true
+---
+
+## About Rclone
+
+{{< hint ok >}}
+Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
+
+Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and --dry-run protection. It is used at the command line, in scripts or via its API.
+
+Users call rclone "The Swiss army knife of cloud storage", and "Technology indistinguishable from magic".
+{{< /hint >}}
+
+Source: [Rclone project website](https://rclone.org/)
+
+## Table of Contents
+
+{{< toc-tree >}}
@@ -0,0 +1,46 @@
+---
+title: WebDAV with Basic Authentication
+date: 2021-11-17T00:00:00+00:00
+weight: 20
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/clients/rclone
+geekdocFilePath: webdav-sync-basic-auth.md
+geekdocCollapseSection: true
+---
+
+
+## WebDAV with Basic Authentication
+
+{{< hint danger >}}
+Basic Authentication is disabled by default in oCIS because of security considerations. In order to make the following Rclone commands work the oCIS administrator needs to enable Basic Authentication eg. by setting the the environment variable `PROXY_ENABLE_BASIC_AUTH` to `true`. 
+
+Please consider to use [Rclone with OpenID Connect]({{< ref "webdav-sync-oidc.md" >}}) instead.
+{{< /hint >}}
+
+For the usage of a WebDAV remote with Rclone see also the [Rclone documentation](https://rclone.org/webdav/)
+
+## Configure the WebDAV remote
+
+First of all we need to set up our credentials and the WebDAV remote for Rclone. In this example we do this by setting environment variables. You might also set up a named remote or use command line options to achieve the same.
+
+``` bash
+export RCLONE_WEBDAV_VENDOR=owncloud
+export RCLONE_WEBDAV_URL=https://ocis.owncloud.test/remote.php/webdav/
+export RCLONE_WEBDAV_USER=einstein
+export RCLONE_WEBDAV_PASS=$(rclone obscure relativity)
+```
+
+{{< hint info >}}
+Please note that `RCLONE_WEBDAV_PASS` is not set to the actual password, but to the value returned by `rclone obscure <password>`.
+{{< /hint >}}
+
+We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oCIS home folder.
+
+
+### Sync to the WebDAV remote
+
+``` bash
+rclone sync :local:/tmp :webdav:/test
+```
+
+If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`. 
@@ -0,0 +1,66 @@
+---
+title: WebDAV with OpenID Connect
+date: 2021-11-17T00:00:00+00:00
+weight: 20
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/clients/rclone
+geekdocFilePath: webdav-sync-oidc.md
+geekdocCollapseSection: true
+---
+
+
+## WebDAV with OpenID Connect
+
+Rclone itself is not able to open and maintain an OpenID Connect session. But it is able to still use OpenID Connect for authentication by leveraging a so called OIDC-agent.
+
+### Setting up the OIDC-agent
+
+You need to install the [OIDC-agent](https://github.com/indigo-dc/oidc-agent) from your OS' package repository (eg. [Debian](https://github.com/indigo-dc/oidc-agent#debian-packages) or [MacOS](https://github.com/indigo-dc/oidc-agent#debian-packages)).
+
+
+### Configuring the the OIDC-agent
+
+Run the following command to add a OpenID Connect profile to your OIDC-agent. It will open the login page of OpenID Connect identity provider where you need to log in if you don't have an active session.
+
+``` bash
+oidc-gen \
+ --client-id=oidc-agent \
+ --client-secret="" \
+ --pub \
+ --issuer https://ocis.owncloud.test \
+ --redirect-uri=http://localhost:12345 \
+ --scope max \
+ einstein-ocis-owncloud-test
+```
+
+If you have dynamic client registration enabled on your OpenID Connect identity provider, you can skip the `--client-id`,  `--client-secret` and `--pub` options.
+
+If your're using a dedicated OpenID Connect client for the OIDC-agent, we recommend a public one with the following two redirect URIs: `http://127.0.0.1:*` and `http://localhost:*`. Alternatively you also may use the already existing OIDC client of the ownCloud Desktop Client (`--client-id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69` and `--client-secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh`, no `--pub` set)
+
+Please also note that the OIDC-agent will listen on your localhost interface on port 12345 for the time of the intial authentication. If that port is already occupied on your machine, you can easily change that by setting the `--redirect-uri` parameter to a different value.
+
+After a successful login or an already existing session you will be redirected to success page of the OIDC-agent.
+You will now be asked for a password for your account configuration, so that your OIDC session is secured and cannot be used by other people with access to your computer.
+
+
+
+## Configure the WebDAV remote
+
+First of all we need to set up our credentials and the WebDAV remote for Rclone. In this example we do this by setting environment variables. You might also set up a named remote or use command line options to achieve the same.
+
+``` bash
+export RCLONE_WEBDAV_VENDOR=owncloud
+export RCLONE_WEBDAV_URL=https://ocis.owncloud.test/remote.php/webdav/
+export RCLONE_WEBDAV_BEARER_TOKEN_COMMAND="oidc-token einstein-ocis-owncloud-test"
+```
+
+
+### Sync to the WebDAV remote
+
+We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oCIS home folder.
+
+``` bash
+rclone sync :local:/tmp :webdav:/test
+```
+
+If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`. 
@@ -10,14 +10,10 @@ geekdocFilePath: releasing.md

 ## Requirements

-You need a working installation of [the Go programming language](https://golang.org/) installed to build the assets for a working release.
-
+You need a working installation of [the Go programming language](https://golang.org/), [the Node runtime](https://nodejs.org/) and [the Yarn package manager](https://yarnpkg.com/) installed to build the assets for a working release.
 ## Releasing

-After adding changes to the accounts package within oCIS and testing them locally, you want to update the compiled assets to the oCIS binary. 
-
-To achieve this, you have to run a Go command and add the results to your PR. The preferred way to do this is to run `make generate` in the root 
-of the repository and then commit the resulting changes to your branch/PR.
+The accounts service doesn't have a dedicated release process. Simply commit your changes, make sure linting and unit tests pass locally and open a pull request.

 ### Package Hierarchy

@@ -130,6 +130,7 @@ curl -k 'https://localhost:9200/graph/v1.0/me/drives' -u einstein:relativity -v
    }
  }
 ]
+```

 As we can see the response already contains a space-aware dav endpoint, which we can use to upload files to the space:

@@ -10,14 +10,11 @@ geekdocFilePath: releasing.md

 ## Requirements

-You need a working installation of [the Go programming language](https://golang.org/) installed to build the assets for a working release.
+You need a working installation of [the Go programming language](https://golang.org/), [the Node runtime](https://nodejs.org/) and [the Yarn package manager](https://yarnpkg.com/) installed to build the assets for a working release.

 ## Releasing

-After adding changes to the settings package within oCIS and testing them locally, you want to update the compiled assets to the oCIS binary. 
-
-To achieve this, you have to run a Go command and add the results to your PR. The preferred way to do this is to run `make generate` in the root 
-of the repository and then commit the resulting changes to your branch/PR.
+The settings service doesn't have a dedicated release process. Simply commit your changes, make sure linting and unit tests pass locally and open a pull request.

 ### Package Hierarchy

@@ -16,4 +16,19 @@ The storage extension wraps [reva](https://github.com/cs3org/reva/) and adds an

 *Clients* will use the *Spaces Registry* to poll or get notified about changes in all *Spaces* a user has access to. Every *Space* has a dedicated `/dav/spaces/<spaceid>` WebDAV endpoint that is served by a *Spaces Provider* which uses a specific reva storage driver to wrap an underlying *Storage System*.

+{{< svg src="extensions/storage/static/overview.drawio.svg" >}}
+
+The dashed lines in the diagram indicate requests that are made to authenticate requests or lookup the storage provider:
+1. After authenticating a request, the proxy may either use the CS3 `userprovider` or the accounts service to fetch the user information that will be minted into the `x-access-token`.
+2. The gateway will verify the JWT signature of the `x-access-token` or try to authenticate the request itself, e.g. using a public link token.
+
+{{< hint warning >}}
+The bottom part is lighter because we will deprecate it in favor of using only the CS3 user and group providers after moving some account functionality into reva and glauth. The metadata storage is not registered in the reva gateway to separate metadata necessary for running the service from data that is being served directly.
+{{< /hint >}}
+
+## Endpoints and references
+
+In order to reason about the request flow, two aspects in the architecture need to be understood well:
+1. What kind of [*namespaces*]({{< ref "./namespaces.md" >}}) are presented at the different WebDAV and CS3 endpoints?
+2. What kind of [*resource*]({{< ref "./terminology.md#resources" >}}) [*references*]({{< ref "./terminology.md#references" >}}) are exposed or required: path or id based?
 {{< svg src="extensions/storage/static/storage.drawio.svg" >}}
@@ -0,0 +1,458 @@
+---
+title: "Apps"
+date: 2018-05-02T00:00:00+00:00
+weight: 10
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/extensions/storage
+geekdocFilePath: apps.md
+---
+
+oCIS is all about files. But most of the time you want to do something with files that is beyond the basic upload, download and share behavior. Therefore, oCIS has a concept for apps, that can handle specific file types, so called mime types.
+
+## App provider capability
+
+The capabilities endpoint (eg. `https://localhost:9200/ocs/v1.php/cloud/capabilities?format=json`) gives you following capabilities which are relevant for the app provider:
+
+```json
+{
+  "ocs": {
+    "data": {
+      "capabilities": {
+        "files": {
+          "app_providers": [
+            {
+              "enabled": true,
+              "version": "1.0.0",
+              "apps_url": "/app/list",
+              "open_url": "/app/open"
+            }
+          ]
+        }
+      }
+    }
+  }
+}
+```
+
+{{< hint info >}}
+Please note that there might be two or more app providers with different versions. This is not be expected to happen on a regular basis. It was designed for a possible migration period for clients when the app provider needs a breaking change.
+{{< /hint >}}
+
+## App registry
+
+The app registry is the single point where all apps register themselves and their respective supported mime types.
+
+### Mime type configuration / creation allow list
+
+The apps will register their supported mime types automatically, so that users can open supported files with them.
+
+Administrators can set default applications on a per mime type basis and also allow the creation of new files for certain mime types. This per mime type configuration also features a description, file extension option and an icon.
+
+In order to modify the mime type config you need to set `STORAGE_APP_REGISTRY_MIMETYPES_JSON=.../mimetypes.json` to a valid JSON file with content like this:
+
+```json
+[
+  {
+    "mime_type": "applition/vnd.oasis.opendocument.text",
+    "extension": "odt",
+    "name": "OpenDocument",
+    "description": "OpenDocument text document",
+    "icon": "https://some-website.test/opendocument-text-icon.png",
+    "default_app": "Collabora",
+    "allow_creation": true
+  },
+  {
+    "mime_type": "application/vnd.oasis.opendocument.spreadsheet",
+    "extension": "ods",
+    "name": "OpenSpreadsheet",
+    "description": "OpenDocument spreadsheet document",
+    "icon": "",
+    "default_app": "Collabora",
+    "allow_creation": false
+  }
+]
+```
+
+Fields:
+
+- `mime_type` is the mime type you want to configure
+- `extension` is the file extension to be used for new files
+- `name` is the name of the file / mime type
+- `description` is a human readable description of the file / mime type
+- `icon` URL to an icon which should be used for that mime type
+- `default_app` name of the default app which opens this mime type when the user doesn't specify one
+- `allow_creation` is wether a user should be able to create new file from that mime type (`true` or `false`)
+
+### Listing available apps / mime types
+
+Clients, for example ownCloud Web, need to offer users the available apps to open files and mime types for new file creation. This information can be obtained from this endpoint.
+
+**Endpoint**: specified in the capabilities in `apps_url`, currently `/app/list`
+
+**Method**: HTTP GET
+
+**Authentication**: None
+
+**Request example**:
+
+```bash
+curl 'https://ocis.test/app/list'
+```
+
+**Response example**:
+
+HTTP status code: 200
+
+```json
+{
+  "mime-types": [
+    {
+      "mime_type": "application/pdf",
+      "ext": "pdf",
+      "app_providers": [
+        {
+          "name": "OnlyOffice",
+          "icon": "https://www.pikpng.com/pngl/m/343-3435764_onlyoffice-desktop-editors-onlyoffice-logo-clipart.png"
+        }
+      ],
+      "name": "PDF",
+      "description": "PDF document"
+    },
+    {
+      "mime_type": "application/vnd.oasis.opendocument.text",
+      "ext": "odt",
+      "app_providers": [
+        {
+          "name": "Collabora",
+          "icon": "https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png"
+        },
+        {
+          "name": "OnlyOffice",
+          "icon": "https://www.pikpng.com/pngl/m/343-3435764_onlyoffice-desktop-editors-onlyoffice-logo-clipart.png"
+        }
+      ],
+      "name": "OpenDocument",
+      "icon": "https://some-website.test/opendocument-text-icon.png",
+      "description": "OpenDocument text document",
+      "allow_creation": true,
+      "default_application": "Collabora"
+    },
+    {
+      "mime_type": "text/markdown",
+      "ext": "md",
+      "app_providers": [
+        {
+          "name": "CodiMD",
+          "icon": "https://avatars.githubusercontent.com/u/67865462?v=4"
+        }
+      ],
+      "name": "Markdown file",
+      "description": "Markdown file",
+      "allow_creation": true,
+      "default_application": "CodiMD"
+    },
+    {
+      "mime_type": "application/vnd.ms-word.document.macroenabled.12",
+      "app_providers": [
+        {
+          "name": "Collabora",
+          "icon": "https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png"
+        },
+        {
+          "name": "OnlyOffice",
+          "icon": "https://www.pikpng.com/pngl/m/343-3435764_onlyoffice-desktop-editors-onlyoffice-logo-clipart.png"
+        }
+      ]
+    },
+    {
+      "mime_type": "application/vnd.ms-powerpoint.template.macroenabled.12",
+      "app_providers": [
+        {
+          "name": "Collabora",
+          "icon": "https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png"
+        }
+      ]
+    }
+  ]
+}
+```
+
+### Open a file with the app provider
+
+**Endpoint**: specified in the capabilities in `open_url`, currently `/app/open`
+
+**Method**: HTTP POST
+
+**Authentication** (one of them):
+
+- `Authorization` header with OIDC Bearer token for authenticated users or basic auth credentials (if enabled in oCIS)
+- `Public-Token` header with public link token for public links
+- `X-Access-Token` header with a REVA token for authenticated users
+
+**Query parameters**:
+
+- `file_id` (mandatory): id of the file to be opened
+- `app_name` (optional)
+  - default (not given): default app for mime type
+  - possible values depend on the app providers for a mimetype from the `/app/open` endpoint
+- `view_mode` (optional)
+  - default (not given): highest possible view mode, depending on the file permissions
+  - possible values:
+    - `write`: user can edit and download in the opening app
+    - `read`: user can view and download from the opening app
+    - `view`: user can view in the opening app (download is not possible)
+
+**Request examples**:
+
+```bash
+curl -X POST 'https://ocis.test/app/open?file_id=ZmlsZTppZAo='
+
+curl -X POST 'https://ocis.test/app/open?file_id=ZmlsZTppZAo=&app_name=Collabora'
+
+curl -X POST 'https://ocis.test/app/open?file_id=ZmlsZTppZAo=&view_mode=read'
+
+curl -X POST 'https://ocis.test/app/open?file_id=ZmlsZTppZAo=&app_name=Collabora&view_mode=write'
+```
+
+**Response examples**:
+
+All apps are expected to be opened in an iframe and the response will give some parameters for that action.
+
+There are apps, which need to be opened in the iframe with a form post. The form post must include all form parameters included in the response. For these apps the response will look like this:
+
+HTTP status code: 200
+
+```json
+{
+  "app_url": "https://.....",
+  "method": "POST",
+  "form_parameters": {
+    "access_token": "eyJ0...",
+    "access_token_ttl": "1634300912000",
+    "arbitrary_param": "lorem-ipsum"
+  }
+}
+```
+
+There are apps, which need to be opened in the iframe with a GET request. The GET request must have set all headers included in the response. For these apps the response will look like this:
+
+HTTP status code: 200
+
+```json
+{
+  "app_url": "https://...",
+  "method": "GET",
+  "headers": {
+    "access_token": "eyJ0e...",
+    "access_token_ttl": "1634300912000",
+    "arbitrary_header": "lorem-ipsum"
+  }
+}
+```
+
+**Example responses (error case)**:
+
+- missing `file_id`
+
+  HTTP status code: 400
+
+  ```json
+  {
+    "code": "INVALID_PARAMETER",
+    "message": "missing file ID"
+  }
+  ```
+
+- wrong `view_mode`
+
+  HTTP status code: 400
+
+  ```json
+  {
+    "code": "INVALID_PARAMETER",
+    "message": "invalid view mode"
+  }
+  ```
+
+- unknown `app_name`
+
+  HTTP status code: 404
+
+  ```json
+  {
+    "code": "RESOURCE_NOT_FOUND",
+    "message": "error: not found: app 'Collabor' not found"
+  }
+  ```
+
+- wrong / invalid file id
+
+  HTTP status code: 400
+
+  ```json
+  {
+    "code": "INVALID_PARAMETER",
+    "message": "invalid file ID"
+  }
+  ```
+
+- file id does not point to a file
+
+  HTTP status code: 400
+
+  ```json
+  {
+    "code": "INVALID_PARAMETER",
+    "message": "the given file id does not point to a file"
+  }
+  ```
+
+- file does not exist / unauthorized to open the file
+
+  HTTP status code: 404
+
+  ```json
+  {
+    "code": "RESOURCE_NOT_FOUND",
+    "message": "file does not exist"
+  }
+  ```
+
+### Creating a file with the app provider
+
+**Endpoint**: specified in the capabilities in `new_file_url`, currently `/app/new`
+
+**Method**: HTTP POST
+
+**Authentication** (one of them):
+
+- `Authorization` header with OIDC Bearer token for authenticated users or basic auth credentials (if enabled in oCIS)
+- `Public-Token` header with public link token for public links
+- `X-Access-Token` header with a REVA token for authenticated users
+
+**Query parameters**:
+
+- `parent_container_id` (mandatory): ID of the folder in which the file will be created
+- `filename` (mandatory): name of the new file
+- `template` (optional): not yet implemented
+
+**Request examples**:
+
+```bash
+curl -X POST 'https://ocis.test/app/new?parent_container_id=c2lkOmNpZAo=&filename=test.odt'
+```
+
+**Response example**:
+
+You will receive a file id of the freshly created file, which you can use to open the file in an editor.
+
+```json
+{
+  "file_id": "ZmlsZTppZAo="
+}
+```
+
+**Example responses (error case)**:
+
+- missing `parent_container_id`
+
+  HTTP status code: 400
+
+  ```json
+  {
+    "code": "INVALID_PARAMETER",
+    "message": "missing parent container ID"
+  }
+  ```
+
+- missing `filename`
+
+  HTTP status code: 400
+
+  ```json
+  {
+    "code": "INVALID_PARAMETER",
+    "message": "missing filename"
+  }
+  ```
+
+- parent container not found
+
+  HTTP status code: 404
+
+  ```json
+  {
+    "code": "RESOURCE_NOT_FOUND",
+    "message": "the parent container is not accessible or does not exist"
+  }
+  ```
+
+- `parent_container_id` does not point to a container
+
+  HTTP status code: 400
+
+  ```json
+  {
+    "code": "INVALID_PARAMETER",
+    "message": "the parent container id does not point to a container"
+  }
+  ```
+
+- `filename` is invalid (eg. includes a path segment)
+
+  HTTP status code: 400
+
+  ```json
+  {
+    "code": "INVALID_PARAMETER",
+    "message": "the filename must not contain a path segment"
+  }
+  ```
+
+- file already exists
+
+  HTTP status code: 403
+
+  ```json
+  {
+    "code": "RESOURCE_ALREADY_EXISTS",
+    "message": "the file already exists"
+  }
+  ```
+
+
+## App drivers
+
+App drivers represent apps, if the app is not able to register itself. Currently there is only the CS3org WOPI server app driver.
+
+### CS3org WOPI server app driver
+
+The CS3org WOPI server app driver is included in oCIS by default. It needs at least one WOPI compliant app (eg. Collabora, OnlyOffice or Microsoft Online Online Server) or a CS3org WOPI bridge supported app (CodiMD or Etherpad) and the CS3org WOPI server.
+
+Here is a closer look at the configuration of the actual app provider in a docker-compose example (see also [full example](https://github.com/owncloud/ocis/blob/master/deployments/examples/ocis_wopi/docker-compose.yml)):
+
+```yaml
+services:
+  ocis:
+    image: owncloud/ocis:latest
+    ...
+    environment:
+      ...
+      STORAGE_GATEWAY_GRPC_ADDR: 0.0.0.0:9142 # make the REVA gateway accessible to the app drivers
+
+  ocis-appdriver-collabora:
+    image: owncloud/ocis:latest
+    command: storage-app-provider server # start only the app driver
+    environment:
+      STORAGE_GATEWAY_ENDPOINT: ocis:9142 # oCIS gateway endpoint
+      APP_PROVIDER_GRPC_ADDR: 0.0.0.0:9164
+      APP_PROVIDER_EXTERNAL_ADDR: ocis-appdriver-collabora:9164 # how oCIS can reach this app driver
+      OCIS_JWT_SECRET: ocis-jwt-secret
+      APP_PROVIDER_DRIVER: wopi
+      APP_PROVIDER_WOPI_DRIVER_APP_NAME: Collabora # will be used as name for this app
+      APP_PROVIDER_WOPI_DRIVER_APP_ICON_URI: https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png # will be used as icon for this app
+      APP_PROVIDER_WOPI_DRIVER_APP_URL: https://collabora.owncloud.test # endpoint of collabora
+      APP_PROVIDER_WOPI_DRIVER_INSECURE: false
+      APP_PROVIDER_WOPI_DRIVER_IOP_SECRET: wopi-iop-secret
+      APP_PROVIDER_WOPI_DRIVER_WOPI_URL: https://wopiserver.owncloud.test # endpoint of the CS3org WOPI server
+```
@@ -1,12 +1,181 @@
 ---
 title: "Spaces"
-date: 2018-05-02T00:00:00+00:00
-weight: 3
+date: 2020-04-27T18:46:00+01:00
+weight: 38
 geekdocRepo: https://github.com/owncloud/ocis
 geekdocEditPath: edit/master/docs/extensions/storage
 geekdocFilePath: spaces.md
 ---

+{{< toc >}}
+
+## Editing a Storage Space
+
+The OData specification allows for a mirage of ways of addressing an entity. We will support addressing a Drive entity by its unique identifier, which is the one the graph-api returns when listing spaces, and its format is:
+
+```json
+{
+  "id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c"
+}
+```
+
+This is an extract of an element of the list spaces response. An entire object has the following shape:
+
+```json
+{
+    "driveType": "project",
+    "id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c",
+    "lastModifiedDateTime": "2021-10-07T11:06:43.245418+02:00",
+    "name": "marketing",
+    "owner": {
+        "user": {
+            "id": "ddc2004c-0977-11eb-9d3f-a793888cd0f8"
+        }
+    },
+    "quota": {
+        "total": 65536
+    },
+    "root": {
+        "id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c",
+        "webDavUrl": "https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c"
+    }
+}
+```
+
+### Updating a space property
+
+Having introduced the above, one can refer to a Drive with the following URL format:
+
+```console
+'https://localhost:9200/graph/v1.0/Drive(1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570")
+```
+
+Updating an entity attribute:
+
+```console
+curl -X PATCH 'https://localhost:9200/graph/v1.0/Drive("1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570)' -d '{"name":"42"}' -v
+```
+
+The previous URL resource path segment (`Drive(1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570)`) is parsed and handed over to the storage registry in order to apply the patch changes in the body, in this case update the space name attribute to `42`. Since space names are not unique we only support addressing them by their unique identifiers, any other query would render too ambiguous and explode in complexity.
+
+
+### Updating a space description
+
+Since every space is the root of a webdav directory, following some conventions we can make use of this to set a default storage description and image. In order to do so, every space is created with a hidden `.space` folder at its root, which can be used to store such data.
+
+```curl
+curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!07c26b3a-9944-4f2b-ab33-b0b326fc7570/.space/description.md -d "Add a description to your spaces" -u admin:admin
+```
+
+Verify the description was updated:
+
+```curl
+❯ curl -k https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!07c26b3a-9944-4f2b-ab33-b0b326fc7570/.space/description.md -u admin:admin
+Add a description to your spaces
+```
+
+This feature makes use of the internal storage layout and is completely abstracted from the end user.
+
+### Quotas
+
+Spaces capacity (quota) is independent of the Storage quota. As a Space admin you can set the quota for all users of a space, and as such, there are no limitations and is up to the admin to make a correct use of this.
+
+It is possible to have a space quota greater than the storage quota. A Space may also have "infinite" quota, meaning a single space without quota can occupy the entirety of a disk.
+
+#### Quota Enforcement
+
+Creating a Space with a quota of 10 bytes:
+
+`curl -k -X POST 'https://localhost:9200/graph/v1.0/drives' -u admin:admin -d '{"name":"marketing", "quota": {"total": 10}}' -v`
+
+```console
+/var/tmp/ocis/storage/users
+├── blobs
+├── nodes
+│   ├── 627981c2-2a71-4adf-b680-177e245afdda
+│   ├── 9541e7c3-8fda-4b49-b697-e7e51457cf5a
+│   ├── b5692345-108d-4b80-9747-3a7e9739ad57
+│   └── root
+│       ├── 118351d7-67a4-4cdf-b495-6093d1e572ed -> ../627981c2-2a71-4adf-b680-177e245afdda
+│       └── ddc2004c-0977-11eb-9d3f-a793888cd0f8 -> ../b5692345-108d-4b80-9747-3a7e9739ad57
+├── spaces
+│   ├── personal
+│   │   └── b5692345-108d-4b80-9747-3a7e9739ad57 -> ../../nodes/b5692345-108d-4b80-9747-3a7e9739ad57
+│   ├── project
+│   │   └── 627981c2-2a71-4adf-b680-177e245afdda -> ../../nodes/627981c2-2a71-4adf-b680-177e245afdda
+│   └── share
+├── trash
+└── uploads
+```
+
+Verify the new space has 10 bytes, and none of it is used:
+
+```json
+{
+    "driveType": "project",
+    "id": "1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda",
+    "lastModifiedDateTime": "2021-10-15T11:16:26.029188+02:00",
+    "name": "marketing",
+    "owner": {
+        "user": {
+            "id": "ddc2004c-0977-11eb-9d3f-a793888cd0f8"
+        }
+    },
+    "quota": {
+        "remaining": 10,
+        "total": 10,
+        "used": 0
+    },
+    "root": {
+        "id": "1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda",
+        "webDavUrl": "https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda"
+    }
+}
+```
+
+Upload a 6 bytes file:
+
+`curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!627981c2-2a71-4adf-b680-177e245afdda/6bytes.txt -d "012345" -u admin:admin -v`
+
+Query the quota again:
+
+```json
+...
+"quota": {
+    "remaining": 4,
+    "total": 10,
+    "used": 6
+},
+...
+```
+
+Now attempt to upload 5 bytes to the space:
+
+`curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!627981c2-2a71-4adf-b680-177e245afdda/5bytes.txt -d "01234" -u admin:admin -v`
+
+The request will fail with `507 Insufficient Storage`:
+
+```
+ HTTP/1.1 507 Insufficient Storage
+< Access-Control-Allow-Origin: *
+< Content-Length: 0
+< Content-Security-Policy: default-src 'none';
+< Date: Fri, 15 Oct 2021 09:24:46 GMT
+< Vary: Origin
+< X-Content-Type-Options: nosniff
+< X-Download-Options: noopen
+< X-Frame-Options: SAMEORIGIN
+< X-Permitted-Cross-Domain-Policies: none
+< X-Robots-Tag: none
+< X-Xss-Protection: 1; mode=block
+<
+* Connection #0 to host localhost left intact
+* Closing connection 0
+```
+
+##### Considerations
+
+- If a Space quota is updated to unlimited, the upper limit is the entire available space on disk
 {{< hint warning >}}

 The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this.
@@ -14,7 +183,7 @@ The current implementation in oCIS might not yet fully reflect this concept. Fee
 {{< /hint >}}

 ## Storage Spaces
-A storage *space* is a logical concept. It organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*), 
+A storage *space* is a logical concept. It organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*),
 a *quota*, *permissions* and is identified by a `storage space id`.

 {{< svg src="extensions/storage/static/storagespace.drawio.svg" >}}
@@ -29,7 +198,7 @@ Finally, a logical `storage space id` is not tied to a specific [*spaces provide
 ## Notes

 We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either
- iterating over the root of the storage and treating every folder following the `<user_layout>` as a `home` *storage space*, 
+- iterating over the root of the storage and treating every folder following the `<user_layout>` as a `home` *storage space*,
 - iterating over the root of the storage and treating every folder following a new `<project_layout>` as a `project` *storage space*, or
 - iterating over the root of the storage and treating every folder following a generic `<layout>` as a *storage space* for a configurable space type, or
 - we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like
@@ -98,7 +98,7 @@ The OCS service makes a stat request to the storage provider to get a [ResourceI

 {{< hint >}}
 The user and public share provider implementations identify the file using the [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The [`ResourceInfo`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo) is passed so the share provider can also store who the owner of the resource is. The *path* is not part of the other API calls, e.g. when listing shares.
-The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.*
+The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently logged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.*
 {{< /hint >}}


@@ -130,7 +130,7 @@ To provide the other storage aspects we plan to implement a FUSE overlay filesys
 This is the current default storage driver. While it implements the file tree (using redis, including id based lookup), ETag propagation, trash, versions and sharing (including expiry) using the data directory layout of ownCloud 10 it has [known limitations](https://github.com/owncloud/core/issues/28095) that cannot be fixed without changing the actual layout on disk.

 To setup it up properly in a distributed fashion, the storage-home and the storage-oc need to share the same underlying FS. Their "data" counterparts also need access to the same shared FS.
-For a simple docker-compose setup, you can create a volume which will be used by the "storage-storage-home", "storage-storage-home-data", "storage-storage-oc" and "storage-storage-oc-data" containers. Using the `owncloud/ocis` docker image, the volume would need to be hooked in the `/var/tmp/ocis` folder inside the containers.
+For a simple docker-compose setup, you can create a volume which will be used by the "storage-storage-home", "storage-storage-home-data", "storage-storage-oc" and "storage-storage-oc-data" containers. Using the `owncloud/ocis` docker image, the volume would need to be hooked in the `/var/lib/ocis` folder inside the containers.

 - tree provided by a POSIX filesystem
  - file layout is mapped to the old ownCloud 10 layout
@@ -26,10 +26,136 @@ Absolute references have either the *path* or the *resource id* set:
 Combined references have both, *path* and *resource id* set:
 - the *resource id* identifies the root [*resource*]({{< ref "#resources" >}})
 - the *path* is relative to that root. It MUST start with `.`
+## References
+
+A *reference* is a logical concept that identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) consists of either
+- a *path* based reference, used to identify a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It must start with a `/`.
+- a [CS3 *id* based reference](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId), uniquely identifying a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It consists of a `storage provider id` and an `opaque id`. The `storage provider id` must NOT start with a `/`.
+
+{{< hint info >}}
+The `/` is important because currently the static [*storage registry*]({{< ref "#storage-space-registries" >}}) uses a map to look up which [*storage provider*]({{< ref "#storage-providers" >}}) is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map.
+{{< /hint >}}
+
+
+{{< hint warning >}}
+### Alternative: reference triple ####
+A *reference* is a logical concept. It identifies a [*resource*]({{< ref "#resources" >}}) and consists of
+a `storage_space`, a `<root_id>` and a `<path>`
+```
+<storage_space>!<root_id>:<path>
+```
+While all components are optional, only three cases are used:
+| format | example | description |
+|-|-|-|
+| `!:<absolute_path>` | `!:/absolute/path/to/file.ext` | absolute path |
+| `<storage_space>!:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space |
+| `<storage_space>!<root>:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths |
+
+`<storage_space>` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`.
+
+A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path
+
+| Name | Description | Who resolves it? |
+|------|-------------|-|
+| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider |
+| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource |
+
+Now, the same file is accessed as a share
+| Name | Description |
+|------|-------------|
+| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder |
+| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty |
+
+
+The `:`, `!` and `$` are chosen from the set of [RFC3986 sub delimiters](https://tools.ietf.org/html/rfc3986#section-2.2) on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set:
+| reference | interpretation |
+|-|-|
+| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted |
+| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted |
+| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted |
+| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present |
+| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` |
+
+{{< /hint >}}
+
+## Storage Drivers
+
+A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}):
+
+It maps the *path* and *id* based CS3 *references* to an appropriate [*storage system*]({{< ref "#storage-systems" >}}) specific reference, e.g.:
+- eos file ids
+- posix inodes or paths
+- deconstructed filesystem nodes
+
+{{< hint warning >}}
+**Proposed Change**
+iOS clients can only queue single requests to be executed in the background. The queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause workflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant.
+
+Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider.
+
+Technically, this means that every storage driver needs to have a map of a `uuid` to in internal resource identifier. This internal resource identifier can be
+- an eos fileid, because eos can look up files by id
+- an inode if the filesystem and the storage driver support looking up by inode
+- a path if the storage driver has no way of looking up files by id.
+  - In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date.
+  - to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `<parentuuid>:<childname>`, allowing to trade writes for reads
+
+{{< /hint >}}
+## Storage Providers

 ## Technical concepts

 ### Storage Systems
+{{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}}
+
+{{< hint warning >}}
+**Proposed Change**
+A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}})
+by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}).
+
+{{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}}
+
+By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space.
+{{< /hint >}}
+
+## Storage Space Registries
+
+A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}):
+It is used by the *gateway*
+to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}})
+that should handle a [*reference*]({{< ref "#references" >}}).
+
+{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}}
+
+{{< hint warning >}}
+**Proposed Change**
+A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*:
+It is used by the *gateway*
+to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}})
+that is currently serving a [*storage space*]({{< ref "#storage-space" >}}).
+
+{{< svg src="extensions/storage/static/storageregistry-spaces.drawio.svg" >}}
+
+By making *storage registries* aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can query them for a listing of all [*storage spaces*]({{< ref "#storage-spaces" >}}) a user has access to. Including his home, received shares, project folders or group drives. See [a WIP PR for spaces in the oCIS repo (#1827)](https://github.com/owncloud/ocis/pull/1827) for more info.
+{{< /hint >}}
+
+## Storage Spaces
+A *storage space* is a logical concept:
+It is a tree of [*resources*]({{< ref "#resources" >}})*resources*
+with a single *owner* (*user* or *group*),
+a *quota* and *permissions*, identified by a `storage space id`.
+
+{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}}
+
+Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ...
+
+Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs.
+
+## Shares
+*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavyweight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.*
+
+
+## Storage Systems
 Every *storage system* has different native capabilities like id and path based lookups, recursive change time propagation, permissions, trash, versions, archival and more.
 A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "./storagedrivers.md" >}}).
 There might be multiple [*storage drivers*]({{< ref "./storagedrivers.md" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements.
@@ -50,10 +50,10 @@ If the below defaults don't match your environment change them accordingly:
 ```
 export STORAGE_LDAP_HOSTNAME=localhost
 export STORAGE_LDAP_PORT=9126
-export STORAGE_LDAP_BASE_DN='dc=example,dc=org'
+export STORAGE_LDAP_BASE_DN='dc=ocis,dc=test'
 export STORAGE_LDAP_USERFILTER='(&(objectclass=posixAccount)(cn=%s))'
 export STORAGE_LDAP_GROUPFILTER='(&(objectclass=posixGroup)(cn=%s))'
-export STORAGE_LDAP_BIND_DN='cn=reva,ou=sysusers,dc=example,dc=org'
+export STORAGE_LDAP_BIND_DN='cn=reva,ou=sysusers,dc=ocis,dc=test'
 export STORAGE_LDAP_BIND_PASSWORD=reva
 export STORAGE_LDAP_USER_SCHEMA_UID=uid
 export STORAGE_LDAP_USER_SCHEMA_MAIL=mail
@@ -31,9 +31,7 @@ and take note of its release tag name.
 1. Create a branch `update-web-$version` in the [ocis repository](https://github.com/owncloud/ocis)
 2. Change into web package folder via `cd web`
 3. Inside `web/`, update the `Makefile` so that the WEB_ASSETS_VERSION variable references the currently released version of https://github.com/owncloud/web
-4. Inside `web/`, replace the current assets with newly released ones by running `make pull-assets`
-5. Inside `web/`, run `make generate`. The output should look something like this: `web: embed.go - YYY/MM/DD ... to write [./embed.go] from config file ...`
-6. Move to the changelog (`cd ../changelog/`) and add a changelog file to the `unreleased/` folder (You can copy an old web release changelog item as a template)
-7. Move to the repo root (`cd ..`)and update the WEB_COMMITID in the `/.drone.env` file to the commit id from the released version (unless the existing commit id is already newer)
-8. **Optional:** Test the changes locally by running `cd ocis && go run cmd/ocis/main.go server`, visiting [https://localhost:9200](https://localhost:9200) and confirming everything renders correctly
-9. Commit your changes, push them and [create a PR](https://github.com/owncloud/ocis/pulls)
+4. Move to the changelog (`cd ../changelog/`) and add a changelog file to the `unreleased/` folder (You can copy an old web release changelog item as a template)
+5. Move to the repo root (`cd ..`)and update the WEB_COMMITID in the `/.drone.env` file to the commit id from the released version (unless the existing commit id is already newer)
+6. **Optional:** Test the changes locally by running `cd ocis && go run cmd/ocis/main.go server`, visiting [https://localhost:9200](https://localhost:9200) and confirming everything renders correctly
+7. Commit your changes, push them and [create a PR](https://github.com/owncloud/ocis/pulls)
@@ -33,18 +33,19 @@ Einstein copies the URL in the browser (or an email with the same URL is sent au

 When Marie enters that URL she will be presented with a login form on the `https://cloud.zurich.test` instance, because the share was created on that domain. If `https://cloud.zurich.test` trusts her OpenID Connect identity provider `https://idp.paris.test` she can log in. This time, the *storage space registry* discovery will come up with `https://cloud.paris.test` though. Since that registry is different than the registry tied to `https://cloud.zurich.test` oCIS web can look up the *storage space* `716199a6-00c0-4fec-93d2-7e00150b1c84` and register the WebDAV URL `https://cloud.zurich.test/dav/spaces/716199a6-00c0-4fec-93d2-7e00150b1c84/a/rel/path` in Maries *storage space registry* at `https://cloud.paris.test`. When she accepts that share her clients will be able to sync the new *storage space* at `https://cloud.zurich.test`.

-Or in other words: _total world federation!_
-
 ### oCIS microservice runtime
+
 The oCIS runtime allows us to dynamically manage services running in a single process. We use [suture](https://github.com/thejerf/suture) to create a supervisor tree that starts each service in a dedicated goroutine. By default oCIS will start all built-in oCIS extensions in a single process. Individual services can be moved to other nodes to scale-out and meet specific performance requirements. A [go-micro](https://github.com/asim/go-micro/blob/master/registry/registry.go) based registry allows services in multiple nodes to form a distributed microservice architecture.

 ### oCIS extensions
+
 Every oCIS extension uses [ocis-pkg](https://github.com/owncloud/ocis/tree/master/ocis-pkg), which implements the [go-micro](https://go-micro.dev/) interfaces for [servers](https://github.com/asim/go-micro/blob/v3.5.0/server/server.go#L17-L37) to register and [clients](https://github.com/asim/go-micro/blob/v3.5.0/client/client.go#L11-L23) to lookup nodes with a service [registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/registry.go).
-We are following the [12 Factor](https://12factor.net/) methodology with oCIS. The uniformity of services also allows us to use the same command, logging and configuration mechanism. Configurations are forwarded from the 
+We are following the [12 Factor](https://12factor.net/) methodology with oCIS. The uniformity of services also allows us to use the same command, logging and configuration mechanism. Configurations are forwarded from the
 oCIS runtime to the individual extensions.


 ### go-micro
+
 While the [go-micro](https://go-micro.dev/) framework provides abstractions as well as implementations for the different components in a microservice architecture, it uses a more developer focused runtime philosophy: It is used to download services from a repo, compile them on the fly and start them as individual processes. For oCIS we decided to use a more admin friendly runtime: You can download a single binary and start the contained oCIS extensions with a single `bin/ocis server`. This also makes packaging easier.

 We use [ocis-pkg](https://github.com/owncloud/ocis/tree/master/ocis-pkg) to configure the default implementations for the go-micro [grpc server](https://github.com/asim/go-micro/tree/v3.5.0/plugins/server/grpc), [client](https://github.com/asim/go-micro/tree/v3.5.0/plugins/client/grpc) and [mdns registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/mdns_registry.go), swapping them out as needed, eg. to use the [kubernetes registry plugin](https://github.com/asim/go-micro/tree/v3.5.0/plugins/registry/kubernetes).
@@ -59,6 +60,7 @@ Interacting with oCIS involves a multitude af APIs. The server and all clients r
 We run a huge [test suite](https://github.com/owncloud/core/tree/master/tests), which originated in ownCloud 10 and continues to grow. A detailed description can be found in the developer docs for [testing]({{< ref "development/testing" >}}).

 ### Architecture Overview
+
 Running `bin/ocis server` will start the below services, all of which can be scaled and deployed on a single node or in a cloud native environment, as needed.

 {{< svg src="ocis/static/architecture-overview.drawio.svg" >}}
@@ -69,7 +69,7 @@ Chosen option: "Move accounts functionality to GLAuth and name it accounts", by

 ### Negative Consequences

-* If users want to store users in their IDM and at the same time guests in a seperate user management we need to implement GLAuth backends that support more than one LDAP server.
+* If users want to store users in their IDM and at the same time guests in a separate user management we need to implement GLAuth backends that support more than one LDAP server.

 ## Pros and Cons of the Options

@@ -60,7 +60,7 @@ The migration happens while the service is offline. File metadata, blobs and sha
 - Good, because oCIS can be tested in a staging system without writing to the production system.
 - Good, because file layout on disk can be changed to support new storage driver capabilities.
 - Bad, because the export and import might require significant amounts of storage.
- Bad, because a rollback to the state before the migration might cause data loss of the changes that happend in between.
+- Bad, because a rollback to the state before the migration might cause data loss of the changes that happened in between.
 - Bad, because the cold migration can mean significant downtime.

 ### Hot Migration
@@ -37,7 +37,7 @@ Chosen option: "Dynamic service registration". There were some drawbacks regardi
 * Having dynamic service registration delegates the entire lifecycle of finding a process to the service registry.
 * Removing a-priori knowledge of hostname + port for services.
 * Marrying go-micro's registry and a newly defined registry abstraction on Reva.
-* We will embrace go-micro interfaces by defining a third merger interface in order to marry go-micro registry and rega revistry.
+* We will embrace go-micro interfaces by defining a third merger interface in order to marry go-micro registry and reva registry.
 * The ability to fetch a service node relying only on its name (i.e: com.owncloud.proxy) and not on a tuple hostname + port that we rely on being preconfigured during runtime.
 * Conceptually speaking, a better framework to tie all the services together. Referring to services by names is less overall confusing than having to add a service name + where it is running. A registry is agnostic to "where is it running" because it, by definition, keeps track of this specific question, so when speaking about design or functionality, it will ease communication.

@@ -23,11 +23,11 @@ There should be a way to impose certain limitations in areas of the code that re
 ## Considered Options

 1. Build the evaluation engine in-house.
-2. Use third party libraries such as Open Policy Agent (a CNCF aproved project written in Go)
+2. Use third party libraries such as Open Policy Agent (a CNCF approved project written in Go)

 ## Decision Outcome

-Chosen option: option 2; Use third party libraries such as Open Policy Agent (a CNCF aproved project written in Go)
+Chosen option: option 2; Use third party libraries such as Open Policy Agent (a CNCF approved project written in Go)

 ### Positive Consequences

@@ -97,17 +97,18 @@ This ADR is limited to the scope of "how will a web client deal with the browser

 ## Decision Outcome

-Chosen option: "[option 1]", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)].
+Chosen option: "Mixed global URLs", because it meets the requirement to contain a path and a stable identifier.

 ### Positive Consequences <!-- optional -->

-* [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
-* …
+* The path makes it "human readable"
+* The URL can be bookmarked
+* The bookmarked URLs remain stable even if the path changes
+* All URLs can be shortened to hide any metadata like path, resource name and query parameters

 ### Negative Consequences <!-- optional -->

-* [e.g., compromising quality attribute, follow-up decisions required, …]
-* …
+* the web UI needs to look up the space alias in a registry to build an API request for the `/dav/space` endpoint

 ## Pros and Cons of the Options

@@ -177,7 +178,7 @@ There is a customized ownCloud instance that uses path only based URLs:
 {{< hint >}}
 * `/#` is used by the current vue router.
 * `/s` denotes that this is a space url.
-* `<space_id>` and `<resource_id>` both consist of `<storage_id>:<node_id>`, but the `space_id` can be replaced with a shorter id or an alias. See furthor down below.
+* `<space_id>` and `<resource_id>` both consist of `<storage_id>:<node_id>`, but the `space_id` can be replaced with a shorter id or an alias. See further down below.
 * `<relative/path>` takes precedence over the `<resource_id>`, both are optional
 {{< /hint >}}

@@ -245,11 +246,11 @@ When every space has a namespaced alias and a relative path we can build a globa
 | `https://demo.owncloud.com/files/personal/einstein/relative/path/to/resource?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | sub folder `/relative/path/to/resource` |
 | `https://demo.owncloud.com/files/shares/einstein/somesharename?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | shared URL for `/relative/path/to/resource` |
 | `https://demo.owncloud.com/files/personal/einstein/marie is stupid/and richard as well/resource?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | sub folder `marie is stupid/and richard as well/resource` ... something einstein might not want to reveal |
-| `https://demo.owncloud.com/files/shares/einstein/resource (2)?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | named link URL for `/marie is stupid/and richard as well/resource`, does not disclose the actual hierarchy, has an appended counter to avaid a collision |
+| `https://demo.owncloud.com/files/shares/einstein/resource (2)?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | named link URL for `/marie is stupid/and richard as well/resource`, does not disclose the actual hierarchy, has an appended counter to avoid a collision |
 | `https://demo.owncloud.com/files/shares/einstein/mybestfriends?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | named link URL for `/marie is stupid/and richard as well/resource`, does not disclose the actual hierarchy, has a custom alias for the share |
 | `https://demo.owncloud.com/files/public/kcZVYaXr7oZ66bg/relative/path/to/resource` | sub folder `/relative/path/to/resource` in public link with token `kcZVYaXr7oZ66bg` |
 | `https://demo.owncloud.com/files/public/kcZVYaXr7oZ66bg/relative/path/to/resource` | sub folder `/relative/path/to/resource` in public link with token `kcZVYaXr7oZ66bg` |
-| `https://demo.owncloud.com/s/kcZVYaXr7oZ66bg/` | shortened link to a resource. This is needed to be able to copy a link to a resource whithout leaking any metadata. |
+| `https://demo.owncloud.com/s/kcZVYaXr7oZ66bg/` | shortened link to a resource. This is needed to be able to copy a link to a resource without leaking any metadata. |


 `</namespaced/alias></relative/path/to/resource>` is the global path in the CS3 api. The CS3 Storage Registry is responsible by managing the mount points.
@@ -0,0 +1,96 @@
+---
+title: "13. Locking"
+weight: 13
+date: 2021-08-17T12:56:53+01:00
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/ocis/adr
+geekdocFilePath: 0013-locking.md
+---
+
+- Status: accepted
+- Deciders: @hodyroff, @pmaier1, @jojowein, @dragotin, @micbar, @tbsbdr, @wkloucek
+- Date: 2021-11-03
+
+## Context and Problem Statement
+
+At the time of this writing no locking mechanisms exists in oCIS / REVA for both directories and files. The CS3org WOPI server implements a file based locking in order to lock files. This ADR discusses if this approach is ok for the general availability of oCIS or if changes are needed.
+
+## Decision Drivers
+
+- Is the current situation acceptable for the GA
+- Is locking needed or can we have oCIS / REVA without locking
+
+## Considered Options
+
+1. File based locking
+2. No locking
+3. CS3 API locking
+
+## Decision Outcome
+
+For the GA we chose option 2. Therefore we need to remove or disable the file based locking functionality of the CS3org WOPI server. The decision was taken because the current file based locking does not work on file-only shares. The current locking also does not guarantee exclusive access to a file since other parts of oCIS like the WebDAV API or other REVA services don't respect the locks.
+
+After the GA we need to implement option 3.
+
+## Pros and Cons of the Options
+
+### File based locking
+
+The CS3org WOPI server creates a `.sys.wopilock.<filename>.` and `.~lock.<filename>#` file when opening a file in write mode
+
+**File based locking is good**, because:
+
+- it is already implemented in the current CS3org WOPI server
+
+**File based locking is bad**, because:
+
+- lock files should be checked by all partys manipulating files (eg. the WebDAV api)
+- lock files can be deleted by everyone
+- you can not lock files in a file-only share (you need a folder share to create a lock file besides the original file)
+
+If we have file based locks, we can also sync them with eg. the Desktop Client.
+
+**Syncing lock files is good**: because
+
+- native office applications can notice lock files by the WOPI server and vice versa (Libre Office also creates `.lock.<filename>#` files)
+
+**Syncing lock files is bad**, because:
+
+- if lockfile is not deleted, no one can edit the file
+- creating lock files in a folder shared with 2000000 users creates a lot of noise and pressure on the server (etag propagation, therefore oC Desktop sync client has an ignore rule for `.~lock.*` files)
+
+### No locking
+
+We remove or disable the file based locking of the CS3org WOPI server.
+
+**No locking is good**, because:
+
+- you don't need to release locks
+- overwriting a file just creates a new version of it
+
+**No locking is bad**, because:
+
+- merging changes from different versions is a pain, since there is no way to calculate differences for most of the files (eg. docx or xlsx files)
+- no locking breaks the WOPI specs, as the CS3 WOPI server won't be capable to honor the WOPI Lock related operations
+
+### CS3 API locking
+
+- Add CS3 API for resource (files, directories) locking, unlocking and checking locks
+  - locking always with timeout
+  - lock creation is a "create-if-not-exists" operation
+  - locks need to have arbitrary metadata (eg. the CS3 WOPI server is stateless by storing information on / in the locks)
+- Implement WebDAV locking using the CS3 API
+- Implement Locking in storage drivers
+- Change CS3 WOPI server to use CS3 API locking mechanism
+- Optional: manual lock / unlock in ownCloud Web (who is allowed to unlock locks of another user?)
+
+**CS3 API locking is good**, because:
+
+- you can lock files on the actual storage (if the storage supports that -> storage driver dependent)
+- you can lock files in ownCloud 10 when using the ownCloudSQL storage driver in the migration deployment (but oC10 Collabora / OnlyOffice also need to implement locking, to fully leverage that)
+- clients can get the lock information via the api without ignoring / hiding lock file changes
+- clients can use the lock information to lock the file in their context (eg. via some file explorer integration)
+
+**CS3 API locking is bad**, because:
+
+- it needs to be defined and implemented, currently not planned for the GA
@@ -0,0 +1,173 @@
+---
+title: "Configuration"
+date: "2021-11-09T00:03:16+0100"
+weight: 2
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/ocis/templates
+geekdocFilePath: config.md
+---
+
+{{< toc >}}
+
+## Configuration Framework
+
+In order to simplify deployments and development the configuration model from oCIS aims to be simple yet flexible.
+
+## Overview of the approach
+
+{{< svg src="ocis/static/ocis-config-redesign.drawio.svg" >}}
+
+## In-depth configuration
+
+Since we include a set of predefined extensions within the single binary, configuring an extension can be done in a variety of ways. Since we work with complex types, having as many cli per config value scales poorly, so we limited the options to config files and environment variables, leaving cli flags for common values, such as logging (`--log-level`, `--log-pretty`, `--log-file` or `--log-color`).
+
+The hierarchy is clear enough, leaving us with:
+
+_(each element above overwrites its precedent)_
+
+1. env variables
+2. extension config
+3. ocis config
+
+This is manifested in the previous diagram. We can then speak about "configuration file arithmetics", where resulting config transformations happen through a series of steps. An administrator must be aware of these sources, since mis-managing them can be a source of confusion, having undesired transformations on config files believed not to be applied.
+
+## Flows
+
+Let's explore the various flows with examples and workflows.
+
+### Examples
+
+Let's explore with examples this approach.
+
+#### Expected loading locations:
+
+- `$HOME/.ocis/config/`
+- `/etc/ocis/`
+- `.config/`
+
+followed by the extension name. When configuring the proxy, a valid full path that will get loaded is `$HOME/.ocis/config/proxy.yaml`.
+
+####  Only config files
+
+The following config files are present in the default loading locations:
+
+_ocis.yaml_
+```yaml
+proxy:
+  http:
+    addr: localhost:1111
+  log:
+    pretty: false
+    color: false
+    level: info
+accounts:
+  http:
+    addr: localhost:2222
+  log:
+    level: debug
+    color: false
+    pretty: false
+log:
+  pretty: true
+  color: true
+  level: info
+```
+
+_proxy.yaml_
+```yaml
+http:
+  addr: localhost:3333
+```
+
+_accounts.yaml_
+```yaml
+http:
+  addr: localhost:4444
+```
+
+Note that the extension files will overwrite values from the main `ocis.yaml`, causing `ocis server` to run with the following configuration:
+
+```yaml
+proxy:
+  http:
+    addr: localhost:3333
+accounts:
+  http:
+    addr: localhost:4444
+log:
+  pretty: true
+  color: true
+  level: info
+```
+
+#### Using ENV variables
+
+The logging configuration if defined in the main ocis.yaml is inherited by all extensions. It can be, however, overwritten by a single extension file if desired. The same example can be used to demonstrate environment values overwrites. With the same set of config files now we have the following command `PROXY_HTTP_ADDR=localhost:5555 ocis server`, now the resulting config looks like:
+
+```yaml
+proxy:
+  http:
+    addr: localhost:5555
+accounts:
+  http:
+    addr: localhost:4444
+log:
+  pretty: true
+  color: true
+  level: info
+```
+
+### Workflows
+
+Since one can run an extension using the runtime (supervised) or not (unsupervised), we ensure correct behavior in both modes, expecting the same outputs.
+
+#### Supervised
+
+You are using the supervised mode whenever you issue the `ocis server` command. We start the runtime on port `9250` (by default) that listens for commands regarding the lifecycle of the supervised extensions. When an extension runs supervised and is killed, the only way to provide / overwrite configuration values will be through an extension config file. This is due to the parent process has already started, and it already has its own environment.
+
+#### Unsupervised
+
+All the points from the priority section hold true. An unsupervised extension can be started with the format: `ocis [extension]` i.e: `ocis proxy`. First, `ocis.yaml` is parsed, then `proxy.yaml` followed by environment variables.
+
+## Shared Values
+
+When running in supervised mode (`ocis server`) it is beneficial to have common values for logging, so that the log output is correctly formatted, or everything is piped to the same file without duplicating config keys and values all over the place. This is possible using the global `log` config key:
+
+_ocis.yaml_
+```yaml
+log:
+  level: error
+  color: true
+  pretty: true
+  file: /var/tmp/ocis_output.log
+```
+
+There is, however, the option for extensions to overwrite this global values by declaring their own logging directives:
+
+_ocis.yaml_
+```yaml
+log:
+  level: info
+  color: false
+  pretty: false
+```
+
+One can go as far as to make the case of an extension overwriting its shared logging config that received from the main `ocis.yaml` file. Because things can get out of hands pretty fast we recommend not mixing logging configuration values and either use the same global logging values for all extensions.
+
+{{< hint warning >}}
+When overwriting a globally shared logging values, one *MUST* specify all values.
+{{< /hint >}}
+
+### Log config keys
+
+```yaml
+log:
+  level: [ error | warning | info | debug ]
+  color: [ true | false ]
+  pretty: [ true | false ]
+  file: [ path/to/log/file ] # MUST not be used with pretty = true
+```
+
+## Default config values (in yaml)
+
+TBD. Needs to be generated and merged with the env mappings.
@@ -20,6 +20,7 @@ oCIS deployments are super simple, yet there are many configurations possible fo
 - [oCIS setup with Traefik for SSL termination]({{< ref "ocis_traefik" >}})
 - [oCIS setup with Keycloak as identity provider]({{< ref "ocis_keycloak" >}})
 - [oCIS setup with WOPI server to open office documents in your browser]({{< ref "ocis_wopi" >}})
+- [Parallel deployment of oC10 and oCIS]({{< ref "oc10_ocis_parallel" >}})
 - [oCIS with S3 storage backend (MinIO)]({{< ref "ocis_s3" >}})
 - [oCIS with the Hello extension example]({{< ref "ocis_hello" >}})

@@ -41,10 +42,16 @@ You can change it by setting the `OCIS_JWT_SECRET` environment variable for oCIS
 Another is used secret for singing JWT tokens for uploads and downloads, which also needs to be changed by the user.
 You can change it by setting the `STORAGE_TRANSFER_SECRET` environment variable for oCIS to a random string.

+One more secret is used for machine auth, so that external applications can authenticate with an API key.
+You can change it by setting the `OCIS_MACHINE_AUTH_API_KEY` environment variable for oCIS to a random string.
+
 ### Delete demo users

 {{< hint info >}}
 Before deleting the demo users mentioned below, you must create a new account for yourself and assign it to the administrator role.
+
+To skip the generation of demo users in the first place, run the inital setup step with an additional environment variable.
+`ACCOUNTS_DEMO_USERS_AND_GROUPS=false ./bin/ocis server` generates only the admin, and one user for IDP and Reva respectively.
 {{< /hint >}}

 oCIS ships with a few demo users besides the system users:
@@ -29,9 +29,10 @@ For the following examples you need to have the oCIS binary in your current work

 ### Using automatically generated certificates

-In order to run oCIS with automatically generated and self signed certificates please execute following command. You need to replace `your-host` with an IP or hostname.
+In order to run oCIS with automatically generated and self signed certificates please execute following command. You need to replace `your-host` with an IP or hostname. Since you have only self signed certificates you need to have `OCIS_INSECURE` set to `true`.

 ```bash
+OCIS_INSECURE=true \
 PROXY_HTTP_ADDR=0.0.0.0:9200 \
 OCIS_URL=https://your-host:9200 \
 ./ocis server
@@ -42,6 +43,7 @@ OCIS_URL=https://your-host:9200 \
 If you have your own certificates already in place, you may want to make oCIS use them:

 ```bash
+OCIS_INSECURE=false \
 PROXY_HTTP_ADDR=0.0.0.0:9200 \
 OCIS_URL=https://your-host:9200 \
 PROXY_TRANSPORT_TLS_KEY=./certs/your-host.key \
@@ -49,7 +51,9 @@ PROXY_TRANSPORT_TLS_CERT=./certs/your-host.crt \
 ./ocis server
 ```

-For more configuration options check the configuration section in [oCIS]({{< ref "../configuration" >}}) and the oCIS extensions.
+If you generated these certificates on your own, you might need to set `OCIS_INSECURE` to `true`.
+
+For more configuration options check the configuration section in [oCIS]({{< ref "../config" >}}) and the oCIS extensions.

 ## Start the oCIS fullstack server with Docker Compose

@@ -15,7 +15,7 @@ This document is a work in progress of the current setup.

 ## Current status

-Using ocis and the ownCloud 10 openidconnect and graphapi plugins it is possible today to introduce openid connect based authentication to existing instances. That is a prerequisite for migrating to ocis.
+Using ocis and the ownCloud 10 [graphapi app](https://github.com/owncloud/graphapi/) it is possible today to use an existing owncloud 10 instance as a userbackend and storage backend for ocis.

 ## How to do it

@@ -36,11 +36,8 @@ occ a:e graphapi

 No configuration necessary. You can test with `curl`:
 ```console
-$ curl https://cloud.example.com/index.php/apps/graphapi/v1.0/users -u admin | jq
+$ curl https://cloud.ocis.test/index.php/apps/graphapi/v1.0/users -u admin -s | jq
 Enter host password for user 'admin':
-  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                 Dload  Upload   Total   Spent    Left  Speed
-100   694  100   694    0     0   4283      0 --:--:-- --:--:-- --:--:--  4283
 {
  "value": [
    {
@@ -56,65 +53,218 @@ Enter host password for user 'admin':
    ...

  ],
-  "@odata.nextLink": "https://oc.butonic.de/apps/graphapi/v1.0/users?$top=10&$skip=10"
+  "@odata.nextLink": "https://cloud.ocis.test/apps/graphapi/v1.0/users?$top=10&$skip=10"
 }
 ```

-> Note: The MS graph api actually asks for `Bearer` auth, but in order to check users passwords during an LDAP bind we are exploiting ownClouds authentication implementation that will grant access when `Basic` auth is used. An LDAP Bind you may ask? Read on!
+{{< hint >}}
+The MS graph api actually asks for `Bearer` auth, but in order to check users passwords during an LDAP bind we are exploiting ownClouds authentication implementation that will grant access when `Basic` auth is used. An LDAP Bind you may ask? Read on!
+{{< /hint >}}

-### Start ocis-glauth
+### Grab ocis!

-We are going to use the above ownCloud 10 and graphapi app to turn it into the datastore for an LDAP proxy.
-
-#### Grab it!
-
-In an `ocis` folder
 ```
-$ git clone git@github.com:owncloud/ocis-glauth.git
-$ cd ocis-glauth
-$ make
+$ git clone git@github.com:owncloud/ocis.git
+$ cd ocis
+$ make -C ocis build
 ```
-This should give you a `bin/ocis-glauth` binary. Try listing the help with `bin/ocis-glauth --help`.
+This should give you an `ocis/bin/ocis` binary. Try listing the help with `ocis/bin/ocis --help`.

+{{< hint >}}
+You can check out a custom branch and build a custom binary which can then be used for the below steps.
+{{< /hint >}}
+
+### Start ocis glauth
+
+We are going to use the built binary and ownCloud 10 graphapi app to turn ownCloud 10 into the datastore for an LDAP proxy.
+
+#### configure it
+
+While ocis can be configured using environment variables, eg. for a docker compose setup we are going to use a more traditional config file here.
+Create a config file for ocis in either `/etc/ocis`, `$HOME/.ocis` or `./.config`. You can use `.json`, `.yaml` or `.toml`. I will use toml here, because ... reasons.
+
+```toml
+[glauth.backend]
+datastore = "owncloud"                                   # switch to the owncloud datastore
+servers = ["https://cloud.ocis.test/apps/graphapi/v1.0"] # the graph api endpoint to connect to
+basedn = "dc=ocis,dc=test"                               # base dn to construct the LDAP dn. The user `admin` will become `cn=admin,dc=ocis,dc=test`
+```
+
+{{< hint >}}
+There is a bug in the config merging for environment variables, cli flags and config files causing log settings not to be picked up from the config file when specifying `--extensions`. That is why I will
+* configure most of the config in a file, 
+* adjust logging using `OCIS_LOG_*` environment variables and
+* specify which extension to run using `ocis/bin/ocis server --extensions "comma, separated, list, of, extensions"`. 
+{{< /hint >}}

 #### Run it!

-You need to point `ocis-glauth` to your owncloud domain:
+For now, we only start the glauth extension:
 ```console
-$ bin/ocis-glauth --log-level debug server --backend-datastore owncloud --backend-server https://cloud.example.com --backend-basedn dc=example,dc=com
+$ OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server --extensions "glauth"
 ```

-`--log-level debug` is only used to generate more verbose output
-`--backend-datastore owncloud` switches to tho owncloud datastore
-`--backend-server https://cloud.example.com` is the url to an ownCloud instance with an enabled graphapi app
-`--backend-basedn dc=example,dc=com` is used to construct the LDAP dn. The user `admin` will become `cn=admin,dc=example,dc=com`.
-
 #### Check it is up and running

 You should now be able to list accounts from your ownCloud 10 oc_accounts table using:
 ```console
-$ ldapsearch -x -H ldap://localhost:9125 -b dc=example,dc=com -D "cn=admin,dc=example,dc=com" -W '(objectclass=posixaccount)'
+$ ldapsearch -x -H ldap://127.0.0.1:9125 -b dc=ocis,dc=test -D "cn=admin,dc=ocis,dc=test" -W '(objectclass=posixaccount)'
 ```

 Groups should work as well:
 ```console
-$ ldapsearch -x -H ldap://localhost:9125 -b dc=example,dc=com -D "cn=admin,dc=example,dc=com" -W '(objectclass=posixgroup)'
+$ ldapsearch -x -H ldap://127.0.0.1:9125 -b dc=ocis,dc=test -D "cn=admin,dc=ocis,dc=test" -W '(objectclass=posixgroup)'
 ```

-> Note: This is currently a readonly implementation and minimal to the usecase of authenticating users with idp.
+{{< hint >}}
+This is currently a readonly implementation and minimal to the usecase of authenticating users with an IDP.
+{{< /hint >}}
+
+### Start ocis storage-gateway, storage-authbasic and storage-userprovider
+
+We are going to set up reva to authenticate users against our glauth LDAP proxy. This allows us to log in and use the reva cli. The ocis storage-gateway starts the reva gateway which will authenticate basic auth requests using the storage-authbasic service. Furthermore, users have to be available in the storage-userprovider to retrieve displayname, email address and other user metadata.
+
+To configure LDAP to use our glauth we add this section to the config file:
+
+```toml
+[storage.reva.ldap]
+idp = "https://ocis.ocis.test"
+basedn = "dc=ocis,dc=test" 
+binddn = "cn=admin,dc=ocis,dc=test" # an admin user in your oc10
+bindpassword = "secret"
+userschema = { uid = "uid", displayname = "givenname" } # TODO make glauth return an ownclouduuid and displayname attribute
+```
+
+Now we can start all necessary services.
+
+```console
+$ OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server --extensions "glauth, storage-gateway, storage-authbasic, storage-userprovider"
+```
+
+
+{{< hint warning >}}
+Here I ran out of time. I tried to verify this step with the reva cli:
+`cmd/reva/reva -insecure -host localhost:9142`
+`login basic`
+but it tries to create the user home, which cannot be disabled in a config file: https://github.com/owncloud/ocis/issues/2416#issuecomment-901197053
+
+starting `STORAGE_GATEWAY_DISABLE_HOME_CREATION_ON_LOGIN=true OCIS_LOG_LEVEL=debug OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server --extensions "storage-gateway, storage-authbasic, storage-userprovider"` let me login:
+
+```console
+✗ cmd/reva/reva -insecure -host localhost:9142
+reva-cli v1.11.0-27-g95b1f2ee (rev-95b1f2ee)
+Please use `exit` or `Ctrl-D` to exit this program.
+>> login basic
+username: jfd
+password: OK
+>> whoami
+id:<idp:"https://ocis.ocis.test" opaque_id:"jfd" type:USER_TYPE_PRIMARY > username:"jfd" mail:"jfd@butonic.de" display_name:"J\303\266rn" uid_number:99 gid_number:99 
+>> exit
+```
+
+I hope https://github.com/owncloud/ocis/pull/2024 fixes the parsing order of things.
+
+everything below this is outdated
+
+... gotta run
+{{< /hint >}}
+
+
+### Start ocis storage-userprovider
+
+```console
+ocis/bin/ocis storage-userprovider --ldap-port 19126 --ldap-user-schema-uid uid --ldap-user-schema-displayName givenName --addr :19144
+```
+
+TODO clone `git clone git@github.com:cs3org/cs3apis.git`
+
+query users using [grpcurl](https://github.com/fullstorydev/grpcurl)
+```console
+grpcurl -import-path ./cs3apis/ -proto ./cs3apis/cs3/identity/user/v1beta1/user_api.proto -plaintext localhost:19144 cs3.identity.user.v1beta1.UserAPI/FindUsers
+ERROR:
+  Code: Unauthenticated
+  Message: auth: core access token not found
+```
+
+
+### Start ocis idp
+
+#### Set environment variables
+
+The built in [libregraph/lico](https://github.com/libregraph/lico) needs environment variables to configure the LDAP server:
+```console
+export OCIS_URL=https://ocis.ocis.test
+export IDP_LDAP_URI=ldap://127.0.0.1:9125
+export IDP_LDAP_BASE_DN="dc=ocis,dc=test"
+export IDP_LDAP_BIND_DN="cn=admin,dc=ocis,dc=test"
+export IDP_LDAP_BIND_PASSWORD="its-a-secret"
+export IDP_LDAP_SCOPE=sub
+export IDP_LDAP_LOGIN_ATTRIBUTE=uid
+export IDP_LDAP_NAME_ATTRIBUTE=givenName
+```
+Don't forget to use an existing user with admin permissions (only admins are allowed to list all users via the graph api) and the correct password.
+
+{{< hint warning >}}
+* TODO: change the default values in glauth & ocis to use an `ownclouduuid` attribute.
+* TODO: split `OCIS_URL` and `IDP_ISS` env vars and use `OCIS_URL` to generate the clients in the `identifier-registration.yaml`.
+{{< /hint >}}
+
+### Configure clients
+
+When the `identifier-registration.yaml` does not exist it will be generated based on the `OCIS_URL` environment variable.
+
+#### Run it!
+
+You can now bring up `ocis/bin/ocis idp` with:
+```console
+$ ocis/bin/ocis idp server --iss http://127.0.0.1:9130 --signing-kid gen1-2020-02-27
+```
+
+`ocis/bin/ocis idp` needs to know
+- `--iss http://127.0.0.1:9130` the issuer, which must be a reachable http endpoint. For testing an ip works. For openid connect HTTPS is NOT optional. This URL is exposed in the `http://127.0.0.1:9130/.well-known/openid-configuration` endpoint and clients need to be able to connect to it, securely. We will change this when introducing the proxy.
+- `--signing-kid gen1-2020-02-27` a signature key id, otherwise the jwks key has no name, which might cause problems with clients. a random key is ok, but it should change when the actual signing key changes.
+
+{{< hint warning >}}
+* TODO: the port in the `--iss` needs to be changed when hiding the idp behind the proxy
+* TODO: the signing keys and encryption keys should be precerated so they are reused between restarts. Otherwise all client sessions will become invalid when restarting the IdP.
+{{< /hint >}}
+
+
+#### Check it is up and running
+
+1. Try getting the configuration:
+```console
+$ curl http://127.0.0.1:9130/.well-known/openid-configuration
+```
+
+2. Check if the login works at http://127.0.0.1:9130/signin/v1/identifier
+
+{{< hint >}}
+If you later get a `Unable to find a key for (algorithm, kid):PS256, )` Error make sure you did set a `--signing-kid` when starting `ocis/bin/ocis idp` by checking it is present in http://127.0.0.1:9130/konnect/v1/jwks.json
+{{< /hint >}}
+
+### Start ocis proxy
+
+
+{{< hint >}}
+Everything below this hint is outdated. Next steps are roughly:
+* directly after glauth start the `ocis storage-userporvider`?
+  - how to verify that works?
+  - https://github.com/fullstorydev/grpcurl
+* start proxy
+  - the ocis ipd url can be changed to https
+  - when do we hide oc10 behind ocis? -> advanced bridge at the end? for now run it without touching the existing oc10 instance
+* start web
+  - verify the login works, but how?
+    - TODO the login works, but then the capabilities requests will fail ... unless we make the proxy answer them by talking to oc10?
+
+Other ideas:
+* the owncloud backend in glauth also works with the user provisioning api ... no changes to a running production instance? db access could be done with a read only account as well...
+{{< /hint >}}
+

 ### Start ocis-web

-#### Get it!
-
-In an `ocis` folder
-```
-$ git clone git@github.com:owncloud/ocis.git
-$ cd web
-$ make
-```
-This should give you a `bin/web` binary. Try listing the help with `bin/web --help`.
-
 #### Run it!

 Point `ocis-web` to your owncloud domain and tell it where to find the openid connect issuing authority:
@@ -128,80 +278,6 @@ $ bin/web server --web-config-server https://cloud.example.com --oidc-authority
 - `--oidc-metadata-url https://192.168.1.100:9130/.well-known/openid-configuration` the openid connect configuration endpoint, typically the issuer host with `.well-known/openid-configuration`, but there are cases when another endpoint is used, eg. ping identity provides multiple endpoints to separate domains
 - `--oidc-client-id ocis` the client id we will register later with `ocis-idp` in the `identifier-registration.yaml`

-### Start ocis-idp
-
-#### Get it!
-
-In an `ocis` folder
-```
-$ git clone git@github.com:owncloud/ocis-idp.git
-$ cd ocis-idp
-$ make
-```
-This should give you a `bin/ocis-idp` binary. Try listing the help with `bin/ocis-idp --help`.
-
-#### Set environment variables
-
-Konnectd needs environment variables to configure the LDAP server:
-```console
-export LDAP_URI=ldap://192.168.1.100:9125
-export LDAP_BINDDN="cn=admin,dc=example,dc=com"
-export LDAP_BINDPW="its-a-secret"
-export LDAP_BASEDN="dc=example,dc=com"
-export LDAP_SCOPE=sub
-export LDAP_LOGIN_ATTRIBUTE=uid
-export LDAP_EMAIL_ATTRIBUTE=mail
-export LDAP_NAME_ATTRIBUTE=givenName
-export LDAP_UUID_ATTRIBUTE=uid
-export LDAP_UUID_ATTRIBUTE_TYPE=text
-export LDAP_FILTER="(objectClass=posixaccount)"
-```
-Don't forget to use an existing user and the correct password.
-
-### Configure clients
-
-Now we need to configure a client we can later use to configure the ownCloud 10 openidconnect app. In the `assets/identifier-registration.yaml` have:
-```yaml
---
-
-# OpenID Connect client registry.
-clients:
-  - id: ocis
-    name: ownCloud Infinite Scale
-    application_type: web
-    redirect_uris:
-      - https://cloud.example.com/apps/openidconnect/redirect
-      - http://localhost:9100/oidc-callback.html
-      - http://localhost:9100
-      - http://localhost:9100/
-```
-Replace `cloud.example.com` in the redirect URI with your ownCloud 10 host and port.
-Replace `localhost:9100` in the redirect URIs with your `ocis-web` host and port.
-
-#### Run it!
-
-You can now bring up `ocis-idp` with:
-```console
-$ bin/ocis-idp server --iss https://192.168.1.100:9130 --identifier-registration-conf assets/identifier-registration.yaml --signing-kid gen1-2020-02-27
-```
-
-`ocis-idp` needs to know
- `--iss https://192.168.1.100:9130` the issuer, which must be a reachable https endpoint. For testing an ip works. HTTPS is NOT optional. This url is exposed in the `https://192.168.1.100:9130/.well-known/openid-configuration` endpoint and clients need to be able to connect to it
- `--identifier-registration-conf assets/identifier-registration.yaml` the identifier-registration.yaml you created
- `--signing-kid gen1-2020-02-27` a signature key id, otherwise the jwks key has no name, which might cause problems with clients. a random key is ok, but it should change when the actual signing key changes.
-
-
-#### Check it is up and running
-
-1. Try getting the configuration:
-```console
-$ curl https://192.168.1.100:9130/.well-known/openid-configuration
-```
-
-2. Check if the login works at https://192.168.1.100:9130/signin/v1/identifier
-
-> Note: If you later get a `Unable to find a key for (algorithm, kid):PS256, )` Error make sure you did set a `--signing-kid` when starting `ocis-idp` by checking it is present in https://192.168.1.100:9130/konnect/v1/jwks.json
-
 ### Patch owncloud

 While the UserSession in ownCloud 10 is currently used to test all available IAuthModule implementations, it immediately logs out the user when an exception occurs. However, existing owncloud 10 instances use the oauth2 app to create Bearer tokens for mobile and desktop clients.
@@ -73,6 +73,24 @@ Credentials:
 - oCIS: [ocis.ocis-keycloak.released.owncloud.works](https://ocis.ocis-keycloak.released.owncloud.works)
 - Keycloak: [keycloak.ocis-keycloak.released.owncloud.works](https://keycloak.ocis-keycloak.released.owncloud.works)

+# Parallel deployment of oC10 and oCIS
+
+Credentials:
+
+- oC10 / oCIS: see [default demo users]({{< ref "../getting-started#login-to-owncloud-web" >}})
+- Keycloak:
+  - username: admin
+  - password: admin
+- LDAP management:
+  - username: cn=admin,dc=owncloud,dc=com
+  - password: admin
+
+## Latest
+
+- oC10 / oCIS: [cloud.oc10-ocis-parallel.latest.owncloud.works](https://cloud.oc10-ocis-parallel.latest.owncloud.works)
+- LDAP management: [ldap.oc10-ocis-parallel.latest.owncloud.works](https://ldap.oc10-ocis-parallel.latest.owncloud.works)
+- Keycloak: [keycloak.oc10-ocis-parallel.latest.owncloud.works](https://keycloak.oc10-ocis-parallel.latest.owncloud.works)
+
 # oCIS with Hello extension

 Credentials:
@@ -97,7 +115,7 @@ Credentials:
 - oCIS: [ocis.ocis-s3.latest.owncloud.works](https://ocis.ocis-s3.latest.owncloud.works)
 - MinIO: [minio.ocis-s3.latest.owncloud.works](https://minio.ocis-s3.latest.owncloud.works)

-# oCIS with CS3 users
+# oCIS with LDAP for users and groups

 Credentials:

@@ -108,10 +126,10 @@ Credentials:

 ## Latest

- oCIS: [ocis.ocis-cs3-users.latest.owncloud.works](https://ocis.ocis-cs3-users.latest.owncloud.works)
- LDAP admin: [ldap.ocis-cs3-users.latest.owncloud.works](https://ldap.ocis-cs3-users.latest.owncloud.works)
+- oCIS: [ocis.ocis-ldap.latest.owncloud.works](https://ocis.ocis-ldap.latest.owncloud.works)
+- LDAP admin: [ldap.ocis-ldap.latest.owncloud.works](https://ldap.ocis-ldap.latest.owncloud.works)

 ## Released

- oCIS: [ocis.ocis-cs3-users.released.owncloud.works](https://ocis.ocis-cs3-users.released.owncloud.works)
- LDAP admin: [ldap.ocis-cs3-users.released.owncloud.works](https://ldap.ocis-cs3-users.released.owncloud.works)
+- oCIS: [ocis.ocis-ldap.released.owncloud.works](https://ocis.ocis-ldap.released.owncloud.works)
+- LDAP admin: [ldap.ocis-ldap.released.owncloud.works](https://ldap.ocis-ldap.released.owncloud.works)
@@ -0,0 +1,256 @@
+---
+title: "Kubernetes"
+date: 2021-09-23T11:04:00+01:00
+weight: 25
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/ocis/deployment
+geekdocFilePath: kubernetes.md
+---
+
+{{< toc >}}
+
+## What is Kubernetes
+
+Formally described as:
+
+> Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.
+
+_[source](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)_
+
+Without getting too deep in definitions, and for the purpose of compactness, Kubernetes can be summarized as a way of managing containers that run applications to ensure that there is no downtime and a optimal usage of resources. It provides with a framework in which to run distributed systems.
+
+Kubernetes provides you with:
+- **Service discovery and load balancing**: Kubernetes can expose a container using the DNS name or using their own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.
+- **Storage orchestration**: Kubernetes allows you to automatically mount a storage system of your choice, such as local storages, public cloud providers, and more.
+- **Automated rollouts and rollbacks**: You can describe the desired state for your deployed containers using Kubernetes, and it can change the actual state to the desired state at a controlled rate. For example, you can automate Kubernetes to create new containers for your deployment, remove existing containers and adopt all their resources to the new container.
+- **Automatic bin packing**: You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks. You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit containers onto your nodes to make the best use of your resources.
+- **Self-healing**: Kubernetes restarts containers that fail, replaces containers, kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve.
+- **Secret and configuration management**: Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration without rebuilding your container images, and without exposing secrets in your stack configuration.
+
+_[extracted from k8s docs](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/#why-you-need-kubernetes-and-what-can-it-do)_
+
+If that is still too abstract, [here is an ELI5 writeup](https://dev.to/miguelmota/comment/filh).
+
+### References and further reads
+
+- [Marcel Wunderlich's](https://github.com/Deaddy) [4 series articles](http://deaddy.net/introduction-to-kubernetes-pt-1.html) on Kubernetes clarifying its declarative nature, deep diving into ingress networking, storage and monitoring.
+
+### How does oCIS fit in the Kubernetes model
+
+oCIS was designed with running on Kubernetes in mind. We set up to adopt the [Twelve-Factor App](https://12factor.net/) principles regarding configuration, with almost every aspect of oCIS being modifiable via environment variables. This comes in handy when you especially have a look at how a helm chart's (we will introduce this concept shortly) [list of values](https://github.com/refs/ocis-charts/blob/d8735e3222d2050504303851d3461909c86fcc89/ocis/values.yaml) looks like.
+
+## What is Minikube
+
+[Minikube](https://minikube.sigs.k8s.io/docs/) lets you run a Kubernetes cluster locally. It is the most approachable way to test a deployment. It requires no extra configuration on any cloud platform, as everything runs on your local machine. For the purpose of these docs, this is the first approach we chose to run oCIS and will develop on how to set it up.
+
+## What is `kubectl`
+
+[kubectl](https://kubernetes.io/docs/tasks/tools/) is the command-line tool for Kubernetes. It allows users to run commands against a k8s cluster the user has access to. It supports for having multiple contexts for as many clusters as you have access to. In these docs we will setup 2 contexts, a minikube and a GCP context.
+
+## What are Helm Charts, and why they are useful for oCIS
+
+[Helm](https://helm.sh/) is the equivalent of a package manager for Kubernetes. It can be described as a layer on top of how you would write pods, deployments or any other k8s resource declaration.
+
+### Installing Helm
+
+[Follow the official installation guide](https://helm.sh/docs/intro/install/).
+
+## Setting up Minikube
+
+For a guide on how to set minikube up follow the [official minikube start guide](https://minikube.sigs.k8s.io/docs/start/) for your specific OS.
+
+### Start minikube
+
+First off, verify your installation is correct:
+
+```console
+~/code/refs/ocis-charts
+❯ minikube status
+minikube
+type: Control Plane
+host: Stopped
+kubelet: Stopped
+apiserver: Stopped
+kubeconfig: Stopped
+```
+
+After that, start the cluster:
+
+```console
+~/code/refs/ocis-charts
+❯ minikube start
+😄  minikube v1.23.0 on Darwin 11.4
+✨  Using the docker driver based on existing profile
+👍  Starting control plane node minikube in cluster minikube
+🚜  Pulling base image ...
+🔄  Restarting existing docker container for "minikube" ...
+🐳  Preparing Kubernetes v1.22.1 on Docker 20.10.8 ...
+🔎  Verifying Kubernetes components...
+    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
+🌟  Enabled addons: storage-provisioner, default-storageclass
+🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
+```
+
+_On these docs, we are using the Docker driver on Mac._
+
+## Run a chart
+
+The easiest way to run the entire package is by using the available charts on https://github.com/refs/ocis-charts. It is not the purpose of this guide to explain the inner working of Kubernetes or its resources, as Helm builds an abstraction oon top of it, letting you interact with a refined interface that roughly translates as "helm install" and "helm uninstall".
+
+In order to host charts one can create a [charts repository](https://helm.sh/docs/topics/chart_repository/), but this is outside the scope of this documentation. Having said that, we will assume you have access to a cli and git.
+
+### Requirements
+
+1. minikube up and running.
+2. `kubectl` installed. By [default you should be able to access the minikube's cluster](https://minikube.sigs.k8s.io/docs/handbook/kubectl/). If you chose not to install `kubectl`, minikube wraps `kubectl` as `minikube kubectl`.
+3. helm cli installed.
+4. git installed.
+
+### Setup
+
+1. clone the charts: `git clone https://github.com/refs/ocis-charts.git /var/tmp/ocis-charts`
+2. cd into the charts root: `cd /var/tmp/ocis-charts/ocis`
+3. install the package: `helm install ocis .`
+4. verify the application is running in the cluster: `kubectl get pods`
+
+```console
+❯ kubectl get pods
+NAME                          READY   STATUS    RESTARTS         AGE
+glauth-5fb678b9cb-zs5qh       1/1     Running   3 (10m ago)      3h33m
+ocis-proxy-848f988687-g7fmb   1/1     Running   2 (10m ago)      130m
+ocs-6bb8896dd6-t4bkx          1/1     Running   3 (10m ago)      3h33m
+settings-6bf77f978d-27rdf     1/1     Running   3 (10m ago)      3h33m
+storages-6b45f9c4-2j696       10/10   Running   23 (4m43s ago)   112m
+store-cf79db94d-hvb7z         1/1     Running   3 (10m ago)      3h33m
+web-8685fdd574-tmkfb          1/1     Running   2 (10m ago)      157m
+webdav-f8d4dd7c6-vv4n7        1/1     Running   3 (10m ago)      3h33m
+```
+
+5. expose the proxy as a service to the host
+
+```console
+~/code/refs/ocis-charts
+❯ minikube service proxy-service --url
+🏃  Starting tunnel for service proxy-service.
+|-----------|---------------|-------------|------------------------|
+| NAMESPACE |     NAME      | TARGET PORT |          URL           |
+|-----------|---------------|-------------|------------------------|
+| default   | proxy-service |             | http://127.0.0.1:63633 |
+|-----------|---------------|-------------|------------------------|
+http://127.0.0.1:63633
+❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.
+```
+
+6. attempt a `PROPFIND` WebDAV request to the storage: `curl -v -k -u einstein:relativity -H "depth: 0" -X PROPFIND https://127.0.0.1:63633/remote.php/dav/files/ | xmllint --format -`
+
+If all is correctly setup, you should expect a response back:
+
+```xml
+<?xml version="1.0" encoding="utf-8"?>
+<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns">
+  <d:response>
+    <d:href>/remote.php/dav/files/einstein/</d:href>
+    <d:propstat>
+      <d:prop>
+        <oc:id>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjZlMWIyMjdmLWZmYTQtNDU4Ny1iNjQ5LWE1YjBlYzFkMTNmYw==</oc:id>
+        <oc:fileid>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjZlMWIyMjdmLWZmYTQtNDU4Ny1iNjQ5LWE1YjBlYzFkMTNmYw==</oc:fileid>
+        <d:getetag>"92cc7f069c8496ee2ce33ad4f29de763"</d:getetag>
+        <oc:permissions>WCKDNVR</oc:permissions>
+        <d:resourcetype>
+          <d:collection/>
+        </d:resourcetype>
+        <d:getcontenttype>httpd/unix-directory</d:getcontenttype>
+        <oc:size>4096</oc:size>
+        <d:getlastmodified>Tue, 14 Sep 2021 12:45:29 +0000</d:getlastmodified>
+        <oc:favorite>0</oc:favorite>
+      </d:prop>
+      <d:status>HTTP/1.1 200 OK</d:status>
+    </d:propstat>
+  </d:response>
+</d:multistatus>
+```
+
+## Setting up an external identity provider
+
+The previous setup works because the proxy is configured to run using basic auth, but if we want to actually use the WebUI we will need an external identity provider. From here on the setup is composed of:
+
+- keycloak
+  - traefik
+  - postgresql
+
+Running on i.e: `https://keycloak.owncloud.works`. Because of this we have to adjust some of `values.yaml` key / values to:
+
+```diff
+diff --git a/ocis/values.yaml b/ocis/values.yaml
+index fbc229c..5b36fbd 100644
+--- a/ocis/values.yaml
+++ b/ocis/values.yaml
+@@ -1,9 +1,9 @@
+ # when in local tunnel mode, ingressDomain is the proxy address.
+ # sadly when in combination with --set, anchors are lost.
+-ingressDomain: &ingressDomain "https://stale-wasp-86.loca.lt"
+ingressDomain: &ingressDomain "https://keycloak.owncloud.works"
+
+ # base ocis image
+-image: owncloud/ocis:1.0.0-rc8-linux-amd64
+image: owncloud/ocis:1.11.0-linux-amd64
+
+ # set of ocis services to create deployments objects.
+ services:
+@@ -22,6 +22,8 @@ services:
+       value: "debug"
+     - name: "PROXY_REVA_GATEWAY_ADDR"
+       value: "storages-service:9142"
+    - name: "PROXY_OIDC_ISSUER"
+      value: "https://keycloak.ocis-keycloak.released.owncloud.works/auth/realms/oCIS"
+     - name: "PROXY_ENABLE_BASIC_AUTH"
+       value: "'true'" # see https://stackoverflow.com/a/44692213/2295410
+     volumeMounts:
+@@ -81,34 +85,6 @@ services:
+     labels:
+       app: "glauth"
+     args: ["glauth"]
+   settings:
+     metadata:
+       name: "settings"
+@@ -135,11 +111,11 @@ services:
+     args: ["web"]
+     env:
+     - name: "WEB_UI_CONFIG_SERVER"
+-      value: *ingressDomain
+      value: "https://127.0.0.1:51559/"
+     - name: "WEB_OIDC_METADATA_URL"
+-      value: *ingressDomain
+      value: "https://keycloak.owncloud.works/auth/realms/oCIS/.well-known/openid-configuration"
+     - name: "WEB_OIDC_AUTHORITY"
+-      value: *ingressDomain
+      value: "https://keycloak.owncloud.works/auth/realms/oCIS/.well-known/openid-configuration"
+     ports:
+       values:
+       - name: "http"
+@@ -231,4 +207,4 @@ kubeServices:
+       - protocol: TCP
+         port: 9100
+         targetPort: 9100
+```
+
+NOTE: The IDP has to be properly configure with an oCIS realm and a `web` client configured. There are example config file that have to be adjusted depending on your environment on our [docker-compose examples](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_keycloak/config/keycloak).
+
+You might still need to adjust the IDP:
+
+- Valid Redirect URIs (under clients > web)
+- Web Origins (under clients > web)
+
+## What is GCP
+
+> Google Cloud Platform (GCP), offered by Google, is a suite of cloud computing services that runs on the same infrastructure that Google uses internally for its end-user products
+
+One of such offered services are [Google Kubernetes Engines (GKE)](https://cloud.google.com/kubernetes-engine).
+
+### Can Helm charts run on GCP?
+
+Yes. The next logical step would be to deploy this charts on GKE. There is a pretty thorough guide [at shippable.com](http://docs.shippable.com/deploy/tutorial/deploy-to-gcp-gke-helm/) that, for the purposes of our docs, we are only interested on step 5, as we already explain the previous concepts, and provide with the Charts.
+
+## TODOs
+
+- While log-in works and creating folders work, uploading fails, most likely a configuration issue that has to be solved.
@@ -31,6 +31,8 @@ For more information and how to deploy it, see [monitoring & tracing client](htt

 ## Monitoring & tracing server

+A live version of the monitoring and tracing server for our demo instances is available here: [Grafana](https://grafana.infra.owncloud.works), [Prometheus](https://prometheus.infra.owncloud.works) and [Jaeger Query](https://jaeger.infra.owncloud.works).
+
 The monitoring & tracing server is considered as shared infrastructure and is normally used for different services. This means that oCIS is not the only software whose metrics and traces are available on the monitoring server. It is also possible that data of multiple oCIS instances are available on the monitoring server.

 Metrics are scraped, stored and can be queried with Prometheus. For the visualization of these metrics Grafana is used. Because Prometheus is scraping the metrics from the oCIS server (pull model instead of a push model), the Prometheus server must have access to the exposed endpoint of the Telegraf Prometheus output plugin.
@@ -0,0 +1,170 @@
+---
+title: "Parallel deployment of oC10 and oCIS"
+date: 2020-10-12T14:04:00+01:00
+weight: 24
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/ocis/deployment
+geekdocFilePath: oc10_ocis_parallel.md
+---
+
+{{< toc >}}
+
+## Overview
+
+- This setup reflects [stage 6 of the oC10 to oCIS migration plan]({{< ref "migration#stage-6-parallel-deployment" >}})
+- Traefik generating self signed certificates for local setup or obtaining valid SSL certificates for a server setup
+- OpenLDAP server with demo users
+- LDAP admin interface to edit users
+- Keycloak as OpenID Connect provider in federation with the LDAP server
+- ownCloud 10 with MariaDB and Redis
+  - ownCloud 10 is configured to synchronize users from the LDAP server
+  - ownCloud 10 is used to use OpenID Connect for authentication with Keycloak
+- oCIS running behind Traefik as reverse proxy
+  - oCIS is using the ownCloud storage driver on the same files and same database as ownCloud 10
+  - oCIS is using Keycloak as OpenID Connect provider
+  - oCIS is using the LDAP server as user backend
+- All requests to both oCIS and oC10 are routed through the oCIS proxy and will be routed based on an OIDC claim to one of them. Therefore admins can change on a user basis in the LDAP which backend is used.
+
+[Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/oc10_ocis_parallel)
+
+## Server Deployment
+
+### Requirements
+
+- Linux server with docker and docker-compose installed
+- four domains set up and pointing to your server
+  - cloud.\* for serving oCIS
+  - keycloak.\* for serving Keycloak
+  - ldap .\* for serving the LDAP managment UI
+  - traefik.\* for serving the Traefik dashboard
+
+See also [example server setup]({{< ref "preparing_server" >}})
+
+### Install this example
+
+- Clone oCIS repository
+
+  `git clone https://github.com/owncloud/ocis.git`
+
+- Go to the deployment example
+
+  `cd ocis/deployment/examples/oc10_ocis_parallel`
+
+- Open the `.env` file in a text editor
+  The file by default looks like this:
+
+  ```bash
+      # If you're on a internet facing server please comment out following line.
+      # It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates.
+      INSECURE=true
+
+      ### Traefik settings ###
+      TRAEFIK_LOG_LEVEL=
+      # Serve Treafik dashboard. Defaults to "false".
+      TRAEFIK_DASHBOARD=
+      # Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test"
+      TRAEFIK_DOMAIN=
+      # Basic authentication for the dashboard. Defaults to user "admin" and password "admin"
+      TRAEFIK_BASIC_AUTH_USERS=
+      # Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server
+      TRAEFIK_ACME_MAIL=
+
+      ### shared oCIS / oC10 settings ###
+      # Domain of oCIS / oC10, where you can find the frontend. Defaults to "cloud.owncloud.test"
+      CLOUD_DOMAIN=
+
+      ### oCIS settings ###
+      # oCIS version. Defaults to "latest"
+      OCIS_DOCKER_TAG=
+      # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
+      OCIS_JWT_SECRET=
+      # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
+      STORAGE_TRANSFER_SECRET=
+      # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
+      OCIS_MACHINE_AUTH_API_KEY=
+
+      ### oCIS settings ###
+      # oC10 version. Defaults to "latest"
+      OC10_DOCKER_TAG=
+      # client secret which the openidconnect app uses to authenticate to Keycloak. Defaults to "oc10-oidc-secret"
+      OC10_OIDC_CLIENT_SECRET=
+      # app which will be shown when opening the ownCloud 10 UI. Defaults to "files" but also could be set to "web"
+      OWNCLOUD_DEFAULT_APP=
+      # if set to "false" (default) links will be opened in the classic UI, if set to "true" ownCloud Web is used
+      OWNCLOUD_WEB_REWRITE_LINKS=
+
+      ### LDAP settings ###
+      # password for the LDAP admin user "cn=admin,dc=owncloud,dc=com", defaults to "admin"
+      LDAP_ADMIN_PASSWORD=
+      # Domain of the LDAP management frontend. Defaults to "ldap.owncloud.test"
+      LDAP_MANAGER_DOMAIN=
+
+      ### Keycloak ###
+      # Domain of Keycloak, where you can find the managment and authentication frontend. Defaults to "keycloak.owncloud.test"
+      KEYCLOAK_DOMAIN=
+      # Realm which to be used with oC10 and oCIS. Defaults to "owncloud"
+      KEYCLOAK_REALM=
+      # Admin user login name. Defaults to "admin"
+      KEYCLOAK_ADMIN_USER=
+      # Admin user login password. Defaults to "admin"
+      KEYCLOAK_ADMIN_PASSWORD=
+  ```
+
+  You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
+
+  If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` eg. `TRAEFIK_DOMAIN=traefik.owncloud.test`.
+
+  The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (eg. by using [an online tool](https://htpasswdgenerator.de/) or a cli tool).
+
+  Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`.
+
+  By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OCIS_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated).
+
+  Set your domain for the oC10 and oCIS frontend in `CLOUD_DOMAIN=`, eg. `CLOUD_DOMAIN=cloud.owncloud.test`.
+
+  You also must override the default secrets in `STORAGE_TRANSFER_SECRET` and `OCIS_JWT_SECRET` in order to secure your oCIS instance. Choose some random strings eg. from the output of `openssl rand -base64 32`. For more information see [secure an oCIS instance]({{< ref "./#secure-an-ocis-instance" >}}).
+
+  By default ownCloud 10 will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OC10_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated).
+
+  You can switch the default application of ownCloud 10 by setting`OWNCLOUD_DEFAULT_APP=files` in oder to have the classic UI as frontend, which is also the default. If you prefer ownCloud Web as the default application in ownCloud 10 just set `OWNCLOUD_DEFAULT_APP=web`.
+
+  In oder to change the default link open action which defaults to the classic UI (`OWNCLOUD_WEB_REWRITE_LINKS=false`) you can set it to `OWNCLOUD_WEB_REWRITE_LINKS=true`. This will lead to links being opened in ownCloud Web.
+
+  The OpenLDAP server in this example deployment has an admin users, which is also used as bind user in order to keep theses examples simple. You can change the default password "admin" to a different one by setting it to `LDAP_ADMIN_PASSWORD=...`.
+
+  Set your domain for the LDAP manager UI in `LDAP_MANAGER_DOMAIN=`, eg. `ldap.owncloud.test`.
+
+  Set your domain for the Keycloak administration panel and authentication endpoints to `KEYCLOAK_DOMAIN=` eg. `KEYCLOAK_DOMAIN=keycloak.owncloud.test`.
+
+  Changing the used Keycloak realm can be done by setting `KEYCLOAK_REALM=`. This defaults to the ownCloud realm `KEYCLOAK_REALM=owncloud`. The ownCloud realm will be automatically imported on startup and includes our demo users.
+
+  You probably should secure your Keycloak admin account by setting `KEYCLOAK_ADMIN_USER=` and `KEYCLOAK_ADMIN_PASSWORD=` to values other than `admin`.
+
+  Now you have configured everything and can save the file.
+
+- Start the docker stack
+
+  `docker-compose up -d`
+
+- You now can visit the cloud, oC10 or oCIS depending on the user configuration. Marie defaults to oC10 and Richard and Einstein default to oCIS, but you can change the ownCloud selector at any time in the LDAP management UI.
+
+## Local setup
+
+For a more simple local ocis setup see [Getting started]({{< ref "../getting-started" >}})
+
+This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer.
+
+On Linux and macOS you can add them to your `/etc/hosts` files like this:
+
+```
+127.0.0.1 cloud.owncloud.test
+127.0.0.1 keycloak.owncloud.test
+127.0.0.1 ldap.owncloud.test
+127.0.0.1 traefik.owncloud.test
+```
+
+After that you're ready to start the application stack:
+
+`docker-compose up -d`
+
+You now can visit the cloud, oC10 or oCIS depending on the user configuration. Marie defaults to oC10 and Richard and Einstein default to oCIS, but you can change the ownCloud selector at any time in the LDAP management UI.
@@ -74,7 +74,9 @@ See also [example server setup]({{< ref "preparing_server" >}})
      # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
      OCIS_JWT_SECRET=
      # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
-      OCIS_TRANSFER_SECRET=
+      STORAGE_TRANSFER_SECRET=
+      # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
+      OCIS_MACHINE_AUTH_API_KEY=

      ### oCIS Hello settings ###
      # oCIS Hello version. Defaults to "latest"
@@ -115,7 +117,7 @@ On Linux and macOS you can add them to your `/etc/hosts` files like this:

 ```
 127.0.0.1 ocis.owncloud.test
-127.0.0.1 traefik.owncloud.testt
+127.0.0.1 traefik.owncloud.test
 ```

 After that you're ready to start the application stack:
@@ -17,9 +17,10 @@ geekdocFilePath: ocis_keycloak.md

 [Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_keycloak)

-The docker stack consists 4 containers. One of them is Traefik, a proxy which is terminating ssl and forwards the requests to oCIS in the internal docker network.
+The docker stack consists 4 containers. One of them is Traefik, a proxy which is terminating ssl and forwards the requests to oCIS in the internal docker network. It
+is also responsible for redirecting requests on the OIDC discovery endpoints (e.g. `.well-known/openid-configuration`) to the correct destination in Keycloak.

-Keykloak add two containers: Keycloak itself and a PostgreSQL as database. Keycloak will be configured as oCIS' IDP instead of the internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}})
+Keycloak add two containers: Keycloak itself and a PostgreSQL as database. Keycloak will be configured as oCIS' IDP instead of the internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}})

 The other container is oCIS itself running all extensions in one container. In this example oCIS uses the [oCIS storage driver]({{< ref "../../extensions/storage/storagedrivers" >}})

@@ -77,7 +78,9 @@ See also [example server setup]({{< ref "preparing_server" >}})
  # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
  OCIS_JWT_SECRET=
  # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
-  OCIS_TRANSFER_SECRET=
+  STORAGE_TRANSFER_SECRET=
+  # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
+  OCIS_MACHINE_AUTH_API_KEY=

  ### Keycloak ###
  # Domain of Keycloak, where you can find the management and authentication frontend. Defaults to "keycloak.owncloud.test"
@@ -0,0 +1,129 @@
+---
+title: "oCIS with LDAP"
+date: 2020-10-12T14:04:00+01:00
+weight: 24
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/ocis/deployment
+geekdocFilePath: ocis_ldap.md
+---
+
+
+{{< toc >}}
+
+## Overview
+
+- Traefik generating self signed certificates for local setup or obtaining valid SSL certificates for a server setup
+- OpenLDAP server with demo users
+- LDAP admin interface to edit users
+- oCIS running behind Traefik as reverse proxy
+  - oCIS is using the LDAP server as user backend
+
+[Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_ldap)
+
+## Server Deployment
+
+### Requirements
+
+- Linux server with docker and docker-compose installed
+- four domains set up and pointing to your server
+  - ocis.\* for serving oCIS
+  - ldap .\* for serving the LDAP managment UI
+  - traefik.\* for serving the Traefik dashboard
+
+See also [example server setup]({{< ref "preparing_server" >}})
+
+### Install this example
+
+- Clone oCIS repository
+
+  `git clone https://github.com/owncloud/ocis.git`
+
+- Go to the deployment example
+
+  `cd ocis/deployment/examples/ocis_ldap`
+
+- Open the `.env` file in a text editor
+  The file by default looks like this:
+
+  ```bash
+  # If you're on a internet facing server please comment out following line.
+  # It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates.
+  INSECURE=true
+
+  ### Traefik settings ###
+  # Serve Treafik dashboard. Defaults to "false".
+  TRAEFIK_DASHBOARD=
+  # Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test"
+  TRAEFIK_DOMAIN=
+  # Basic authentication for the dashboard. Defaults to user "admin" and password "admin"
+  TRAEFIK_BASIC_AUTH_USERS=
+  # Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server
+  TRAEFIK_ACME_MAIL=
+
+  ### oCIS settings ###
+  # oCIS version. Defaults to "latest"
+  OCIS_DOCKER_TAG=
+  # Domain of oCIS, where you can find the frontend. Defaults to "ocis.owncloud.test"
+  OCIS_DOMAIN=
+  # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
+  OCIS_JWT_SECRET=
+  # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
+  STORAGE_TRANSFER_SECRET=
+  # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
+  OCIS_MACHINE_AUTH_API_KEY=
+
+  ### LDAP server settings ###
+  # Password of LDAP user "cn=admin,dc=owncloud,dc=com". Defaults to "admin"
+  LDAP_ADMIN_PASSWORD=
+
+  ### LDAP manager settings ###
+  # Domain of LDAP manager. Defaults to "ldap.owncloud.test"
+  LDAP_MANAGER_DOMAIN=
+  ```
+
+  You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
+
+  If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` eg. `TRAEFIK_DOMAIN=traefik.owncloud.test`.
+
+  The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (eg. by using [an online tool](https://htpasswdgenerator.de/) or a cli tool).
+
+  Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`.
+
+  By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OCIS_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated).
+
+  Set your domain for the oCIS frontend in `OCIS_DOMAIN=`, eg. `OCIS_DOMAIN=cloud.owncloud.test`.
+
+  You also must override the default secrets in `STORAGE_TRANSFER_SECRET` and `OCIS_JWT_SECRET` in order to secure your oCIS instance. Choose some random strings eg. from the output of `openssl rand -base64 32`. For more information see [secure an oCIS instance]({{< ref "./#secure-an-ocis-instance" >}}).
+
+  The OpenLDAP server in this example deployment has an admin users, which is also used as bind user in order to keep theses examples simple. You can change the default password "admin" to a different one by setting it to `LDAP_ADMIN_PASSWORD=...`.
+
+  Set your domain for the LDAP manager UI in `LDAP_MANAGER_DOMAIN=`, eg. `ldap.owncloud.test`.
+
+  Now you have configured everything and can save the file.
+
+- Start the docker stack
+
+  `docker-compose up -d`
+
+- You now can visit oCIS and Traefik dashboard on your configured domains. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.
+
+## Local setup
+
+For a more simple local ocis setup see [Getting started]({{< ref "../getting-started" >}})
+
+This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer.
+
+On Linux and macOS you can add them to your `/etc/hosts` files like this:
+
+```
+127.0.0.1 cloud.owncloud.test
+127.0.0.1 keycloak.owncloud.test
+127.0.0.1 ldap.owncloud.test
+127.0.0.1 traefik.owncloud.test
+```
+
+After that you're ready to start the application stack:
+
+`docker-compose up -d`
+
+Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You now can login to oCIS with the default users, which also can be found here: [Getting started]({{< ref "../getting-started#login-to-ocis-web" >}}). You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.
@@ -76,7 +76,9 @@ See also [example server setup]({{< ref "preparing_server" >}})
      # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
      OCIS_JWT_SECRET=
      # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
-      OCIS_TRANSFER_SECRET=
+      STORAGE_TRANSFER_SECRET=
+      # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
+      OCIS_MACHINE_AUTH_API_KEY=

      ### MINIO / S3 settings ###
      # Domain of MinIO where the Web UI is accessible. Defaults to "minio.owncloud.test".
@@ -71,7 +71,9 @@ See also [example server setup]({{< ref "preparing_server" >}})
  # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
  OCIS_JWT_SECRET=
  # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
-  OCIS_TRANSFER_SECRET=
+  STORAGE_TRANSFER_SECRET=
+  # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
+  OCIS_MACHINE_AUTH_API_KEY=
  ```

  You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
@@ -9,22 +9,26 @@ geekdocFilePath: ocis_wopi.md

 {{< toc >}}

+{{< hint warning >}}
+OnlyOffice and CodiMD are not yet fully integrated and there are known issues. For the current state please have a look at [owncloud/ocis#2595](https://github.com/owncloud/ocis/issues/2595)
+{{< /hint >}}
+
 ## Overview

-* oCIS, Wopi server and Collabora running behind Traefik as reverse proxy
-* Collabora enables you to edit text documents in your browser
-* Wopi server acts as a bridge to make the oCIS storage accessible to Collabora
+* oCIS, Wopi server, Collabora, OnlyOffice and CodiMD running behind Traefik as reverse proxy
+* Collabora, OnlyOffice and CodiMD enable you to edit documents in your browser
+* Wopi server acts as a bridge to make the oCIS storage accessible to Collabora, OnlyOffice and CodiMD
 * Traefik generating self signed certificates for local setup or obtaining valid SSL certificates for a server setup

 [Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_wopi)

-The docker stack consists 5 containers. One of them is Traefik, a proxy which is terminating SSL and forwards the requests to oCIS in the internal docker network.
+The docker stack consists of 10 containers. One of them is Traefik, a proxy which is terminating SSL and forwards the requests to oCIS in the internal docker network.

-The next container is oCIS itself in a configuration like the [oCIS with Traefik example]({{< ref "ocis_traefik" >}}), except that for this example a custom proxy and web UI configuration is used to enable the oCIS Wopi extension.
+The next container is oCIS itself in a configuration like the [oCIS with Traefik example]({{< ref "ocis_traefik" >}}), except that for this example a custom mimetype configuration is used.

-The oCIS WOPI server extension is running in another container and enables you to open files in Collabora from within ownCloud Web.
+There are three oCIS app driver containers that register Collabora, OnlyOffice and CodiMD at the app registry.

-The last two containers are the WOPI server and Collabora.
+The last four containers are the WOPI server, Collabora, OnlyOffice and CodiMD.

 ## Server Deployment

@@ -34,6 +38,8 @@ The last two containers are the WOPI server and Collabora.
 * Three domains set up and pointing to your server
  - ocis.* for serving oCIS
  - collabora.* for serving Collabora
+  - onlyoffice.* for serving OnlyOffice
+  - codimd.* for serving CodiMD
  - wopiserver.* for serving the WOPI server
  - traefik.* for serving the Traefik dashboard

@@ -79,13 +85,13 @@ See also [example server setup]({{< ref "preparing_server" >}})
    # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
    OCIS_JWT_SECRET=
    # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
-    OCIS_TRANSFER_SECRET=
+    STORAGE_TRANSFER_SECRET=
+    # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
+    OCIS_MACHINE_AUTH_API_KEY=

    ### Wopi server settings ###
-    # oCIS Wopi server version. Defaults to "latest"
-    OCIS_WOPISERVER_DOCKER_TAG=
    # cs3org wopi server version. Defaults to "latest"
-    CS3ORG_WOPISERVER_DOCKER_TAG=
+    WOPISERVER_DOCKER_TAG=
    # cs3org wopi server domain. Defaults to "wopiserver.owncloud.test"
    WOPISERVER_DOMAIN=
    # JWT secret which is used for the documents to be request by the Wopi client from the cs3org Wopi server. Must be change in order to have a secure Wopi server. Defaults to "LoremIpsum567"
@@ -98,9 +104,18 @@ See also [example server setup]({{< ref "preparing_server" >}})
    COLLABORA_DOMAIN=
    # Admin user for Collabora. Defaults to blank, provide one to enable access
    COLLABORA_ADMIN_USER=
-    # Admin password for COllabora. Defaults to blank, provide one to enable access
+    # Admin password for Collabora. Defaults to blank, provide one to enable access
    COLLABORA_ADMIN_PASSWORD=

+    ### OnlyOffice settings ###
+    # Domain of OnlyOffice, where you can find the frontend. Defaults to "onlyoffice.owncloud.test"
+    ONLYOFFICE_DOMAIN=
+
+    ### CodiMD settings ###
+    # Domain of Collabora, where you can find the frontend. Defaults to "codimd.owncloud.test"
+    CODIMD_DOMAIN=
+    # Secret which is used for the communication with the WOPI server. Must be changed in order to have a secure CodiMD. Defaults to "LoremIpsum456"
+    CODIMD_SECRET=
  ```

  You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
@@ -117,18 +132,20 @@ See also [example server setup]({{< ref "preparing_server" >}})

  You also must override three default secrets in `IDP_LDAP_BIND_PASSWORD`, `STORAGE_LDAP_BIND_PASSWORD` and `OCIS_JWT_SECRET` in order to secure your oCIS instance. Choose some random strings eg. from the output of `openssl rand -base64 32`. For more information see [secure an oCIS instance]({{< ref "./#secure-an-ocis-instance" >}}).

-  By default the oCIS WOPI server extension will be started in the `latest` version. If you want to start a specific version of oCIS WOPI server set the version to `OCIS_WOPISERVER_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis-wopiserver/tags?page=1&ordering=last_updated).
+  By default the CS3Org WOPI server will also be started in the `latest` version. If you want to start a specific version of it, you can set the version to `WOPISERVER_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/cs3org/wopiserver/tags?page=1&ordering=last_updated).

-  By default the CS3Org WOPI server will also be started in the `latest` version. If you want to start a specific version of it, you can set the version to `CS3ORG_WOPISERVER_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/cs3org/wopiserver/tags?page=1&ordering=last_updated).
-
-  Set your domain for the CS3Org WOPI server in `WOPISERVER_DOMAIN=`, where Collabora can download the files.
+  Set your domain for the CS3Org WOPI server in `WOPISERVER_DOMAIN=`, where all office suites can download the files via the WOPI protocol.

  You also must override the default WOPI JWT secret and the WOPI IOP secret, in order to have a secure setup. Do this by setting `WOPI_JWT_SECRET` and `WOPI_IOP_SECRET` to a long and random string.

-  Now it's time to set up Collabora and you need to configure the Domain of Collabora in `COLLABORA_DOMAIN=`.
+  Now it's time to set up Collabora and you need to configure the domain of Collabora in `COLLABORA_DOMAIN=`.

  If you want to use the Collabora admin panel you need to set user name and passwort for in `COLLABORA_ADMIN_USER=` and `COLLABORA_ADMIN_PASSWORD=`.

+  Next up is OnlyOffice, which also needs a domain in `ONLYOFFICE_DOMAIN=`.
+
+  The last configuration options are for CodiMD, which needs a domain in `CODIMD_DOMAIN=` and a random secret in `CODIMD_SECRET=`.
+
  Now you have configured everything and can save the file.

 * Start the docker stack
@@ -147,6 +164,8 @@ On Linux and macOS you can add them to your `/etc/hosts` files like this:
 127.0.0.1 ocis.owncloud.test
 127.0.0.1 traefik.owncloud.test
 127.0.0.1 collabora.owncloud.test
+127.0.0.1 onlyoffice.owncloud.test
+127.0.0.1 codimd.owncloud.test
 127.0.0.1 wopiserver.owncloud.test
 ```

@@ -154,6 +173,6 @@ After that you're ready to start the application stack:

 `docker-compose up -d`

-Open https://collabora.owncloud.test and https://wopiserver.owncloud.test  in your browser and accept the invalid certificate warning.
+Open https://collabora.owncloud.test, https://onlyoffice.owncloud.test, https://codimd.owncloud.test and https://wopiserver.owncloud.test  in your browser and accept the invalid certificate warning.

 Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You are now able to open an office document in your browser. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.
@@ -0,0 +1,69 @@
+---
+title: "Systemd service"
+date: 2020-09-27T06:00:00+01:00
+weight: 16
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/ocis/deployment
+geekdocFilePath: systemd.md
+---
+
+{{< toc >}}
+
+## Install the oCIS binary
+Download the oCIS binary of your preferred version and for your CPU architecture and operating system from [download.owncloud.com](https://download.owncloud.com/ocis/ocis).
+
+Rename the downloaded binary to `ocis` and move it to `/usr/bin/`. As a next step, you need to mark it as executable with `chmod +x /usr/bin/ocis`.
+
+When you now run `ocis help` on your command line, you should see the available options for the oCIS command.
+
+
+## Systemd service definition
+
+Create the Systemd service definition for oCIS in the file `/etc/systemd/system/ocis.service` with following content:
+```
+[Unit]
+Description=OCIS server
+
+[Service]
+Type=simple
+User=root
+Group=root
+EnvironmentFile=/etc/ocis/ocis.env
+ExecStart=ocis server
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+```
+
+For reasons of simplicity we are using the root user and group to run oCIS which is not recommended. Please use a non-root user in production environments and modify the oCIS service definition accordingly.
+
+
+In the service definition we referenced `/etc/ocis/ocis.env` as our file containing environment variables for the oCIS process.
+In order to create the file we need first to create the folder `/etc/ocis/` and than we can add the actual `/etc/ocis/ocis.env` with following content:
+
+```
+OCIS_URL=https://some-hostname-or-ip:9200
+PROXY_HTTP_ADDR=0.0.0.0:9200
+OCIS_INSECURE=false
+
+OCIS_LOG_LEVEL=error
+
+GLAUTH_LDAPS_CERT=/etc/ocis/ldap/ldaps.crt
+GLAUTH_LDAPS_KEY=/etc/ocis/ldap/ldaps.key
+IDP_TRANSPORT_TLS_CERT=/etc/ocis/idp/server.crt
+IDP_TRANSPORT_TLS_KEY=/etc/ocis/idp/server.key
+PROXY_TRANSPORT_TLS_CERT=/etc/ocis/proxy/server.crt
+PROXY_TRANSPORT_TLS_KEY=/etc/ocis/proxy/server.key
+```
+
+Please change your `OCIS_URL` in order to reflect your actual deployment. If you are using self signed certificates you need to set `OCIS_INSECURE=true` in `/etc/ocis/ocis.env`.
+
+
+## Starting the oCIS service
+
+You can enable oCIS now by running `systemctl enable --now ocis`. It will ensure that oCIS also is restarted after a reboot of the host.
+
+If you need to restart oCIS because of configuration changes in `/etc/ocis/ocis.env`, run `systemctl restart ocis`.
+
+You can have a look at the logs of oCIS by issuing `journalctl -f -u ocis`.
@@ -35,10 +35,6 @@ You may add flags to your commit message or PR title in order to speed up pipeli

 - `[full-ci]`: deactivates the fail early mechanism and runs all available test (as default only smoke tests are run)

- `[docs-only]`: please add this flag, if you only changed documentation. This will only trigger documentation related CI steps.
-
- `[tests-only]`: please add this flag, if you only changed tests or test-related tooling. You do not need to add a changelog for tests-only changes.
-
 ### Knowledge base

 - My pipeline fails because some CI related files or commands are missing.
@@ -7,7 +7,7 @@ geekdocEditPath: edit/master/docs/ocis/development
 geekdocFilePath: extensions.md
 ---

-oCIS is all about files, sync and share - but most of the time there is more you want to do with your files, e.g. having a different view on your photo collection or editing your offices files in an online file editor. ownCloud 10 faced the same problem and solved it with `applications`, which can extend the functionality of ownCloud 10 in a wide range. Since oCIS is different in its architecture compared to ownCloud 10, we had to come up with a similiar (yet slightly different) solution. To extend the functionality of oCIS, you can write or install `extensions`. An extension is basically any running code which integrates into oCIS and provides functionality to oCIS and its users. Because extensions are just microservices providing an API, you can technically choose any programming language you like - a huge improvement to ownCloud 10, where it was nearly impossible to use a different programming language than PHP.
+oCIS is all about files, sync and share - but most of the time there is more you want to do with your files, e.g. having a different view on your photo collection or editing your offices files in an online file editor. ownCloud 10 faced the same problem and solved it with `applications`, which can extend the functionality of ownCloud 10 in a wide range. Since oCIS is different in its architecture compared to ownCloud 10, we had to come up with a similar (yet slightly different) solution. To extend the functionality of oCIS, you can write or install `extensions`. An extension is basically any running code which integrates into oCIS and provides functionality to oCIS and its users. Because extensions are just microservices providing an API, you can technically choose any programming language you like - a huge improvement to ownCloud 10, where it was nearly impossible to use a different programming language than PHP.

 We will now introduce you to the oCIS extension system and show you how you can create a custom extension yourself.

@@ -16,7 +16,7 @@ So we are trying to reflect this in the tooling. It should be kept simple and qu

 Besides standard development tools like git and a text editor, you need the following software for development:

- Go >= v1.16 ([install instructions](https://golang.org/doc/install))
+- Go >= v1.17 ([install instructions](https://golang.org/doc/install))
 - Yarn ([install instructions](https://classic.yarnpkg.com/en/docs/install))
 - docker ([install instructions](https://docs.docker.com/get-docker/))
 - docker-compose ([install instructions](https://docs.docker.com/compose/install/))
@@ -27,7 +27,7 @@ If you find tools needed besides the mentioned above, please feel free to open a

 oCIS consists of multiple micro services, also called extensions. We started by having standalone repositories for each of them, but quickly noticed that this adds a time consuming overhead for developers. So we ended up with a monorepo housing all the extensions in one repository.

-Each extension lives in a subfolder (eg. `accounts` or `settings`) within this respository as an independent Go module, following the [golang-standard project-layout](https://github.com/golang-standards/project-layout). They have common Makefile targets and can be used to change, build and run individual extensions. This allows us to version and release each extension independently.
+Each extension lives in a subfolder (eg. `accounts` or `settings`) within this repository as an independent Go module, following the [golang-standard project-layout](https://github.com/golang-standards/project-layout). They have common Makefile targets and can be used to change, build and run individual extensions. This allows us to version and release each extension independently.

 The `ocis` folder contains our [go-micro](https://github.com/asim/go-micro/) and [suture](https://github.com/thejerf/suture) based runtime. It is used to import all extensions and implements commands to manage them, similar to a small orchestrator. With the resulting oCIS binary you can start single extensions or even all extensions at the same time.

@@ -48,7 +48,7 @@ For example `make -C tests/acceptance/docker Core-API-Tests-owncloud-storage-3`r
 The single feature tests can also be run against the different storage backends. Therefore multiple make targets with the schema test-<test source>-feature-<storage backend> exists. For selecting a single feature test you have to add an additional `BEHAT_FEATURE=...` parameter when invoking the make command:

 ```
-make -C tests/acceptance/docker test-ocis-feature-ocis BEHAT_FEATURE='tests/acceptance/features/apiAccountsHashDifficulty/addUser.feature'
+make -C tests/acceptance/docker test-ocis-feature-ocis-storage BEHAT_FEATURE='tests/acceptance/features/apiAccountsHashDifficulty/addUser.feature'
 ```

 This must be pointing to a valid feature definition.
@@ -98,7 +98,7 @@ git clone https://github.com/owncloud/core.git
 To start ocis:

 ```
-PROXY_ENABLE_BASIC_AUTH=true bin/ocis server
+OCIS_INSECURE=true PROXY_ENABLE_BASIC_AUTH=true bin/ocis server
 ```

 `PROXY_ENABLE_BASIC_AUTH` will allow the acceptance tests to make requests against the provisioning api (and other endpoints) using basic auth.
@@ -50,7 +50,7 @@ sequenceDiagram
            end

            proxy->>+accounts: TODO API call to exchange sub@iss with account UUID
-            Note over proxy,accounts: does not autoprovision users. They are explicitly provsioned later.
+            Note over proxy,accounts: does not autoprovision users. They are explicitly provisioned later.

            alt account exists or has been migrated

@@ -33,11 +33,11 @@ You can find the latest official release of oCIS at [our download mirror](https:
 The latest build from the master branch can be found at [our download mirrors testing section](https://download.owncloud.com/ocis/ocis/testing/).

 To run oCIS as binary you need to download it first and then run the following commands.
-For this example, assuming version 1.11.0 of oCIS running on a Linux AMD64 host:
+For this example, assuming version 1.13.0 of oCIS running on a Linux AMD64 host:

 ```console
 # download
-curl https://download.owncloud.com/ocis/ocis/1.11.0/ocis-1.11.0-linux-amd64 --output ocis
+curl https://download.owncloud.com/ocis/ocis/1.13.0/ocis-1.13.0-linux-amd64 --output ocis

 # make binary executable
 chmod +x ocis
@@ -46,7 +46,7 @@ chmod +x ocis
 ./ocis server
 ```

-The default primary storage location is `/var/tmp/ocis`. You can change that value by configuration.
+The default primary storage location is `~/.ocis` or `/var/lib/ocis` depending on the packaging format and your operating system user. You can change that value by configuration.

 {{< hint warning >}}
 oCIS by default relies on Multicast DNS (mDNS), usually via avahi-daemon. If your system has a firewall, make sure mDNS is allowed in your active zone.
@@ -71,7 +71,7 @@ Open [https://localhost:9200](https://localhost:9200) and [login using one of th

 ### Basic Management Commands

-The oCIS single binary contains multiple extensions and the `ocis` command helps you to manage them. You already used `ocis server` to run all available extensions in the [Run oCIS]({{< ref "#run-ocis" >}}) section. We now will show you some more management commands, which you may also explore by typing `ocis --help` or going to the [docs]({{< ref "../configuration" >}}).
+The oCIS single binary contains multiple extensions and the `ocis` command helps you to manage them. You already used `ocis server` to run all available extensions in the [Run oCIS]({{< ref "#run-ocis" >}}) section. We now will show you some more management commands, which you may also explore by typing `ocis --help` or going to the [docs]({{< ref "../config" >}}).

 To start oCIS server:

@@ -9,6 +9,11 @@ geekdocFilePath: demo-users.md

 As long as oCIS is released as [technology preview]({{< ref "../release_roadmap#release_roadmap" >}}) it will come with default demo users. These enable you to do quick testing and developing.

+{{< hint info >}}
+To skip the generation of demo users, run the inital setup step with an additional environment variable.
+`ACCOUNTS_DEMO_USERS_AND_GROUPS=false ./bin/ocis server` generates only the admin, and one user for IDP and Reva respectively.
+{{< /hint >}}
+
 Following users are available in the demo set:

 | username | password      | email                | role  | groups                                                                  |
@@ -9,6 +9,14 @@ geekdocFilePath: migration.md

 The migration happens in subsequent stages while the service is online. First all users need to migrate to the new architecture, then the global namespace needs to be introduced. Finally, the data on disk can be migrated user by user by switching the storage driver.

+<div class="editpage">
+
+{{< hint warning >}}
+@jfd: It might be easier to introduce the spaces api in oc10 and then migrate to oCIS. We cannot migrate both at the same time, the architecture to oCIS (which will change fileids) and introduce a global namespace (which requires stable fileids to let clients handle moves without redownloading). Either we implement arbitrary mounting of shares in oCIS / reva or we make clients and oc10 spaces aware.
+{{< /hint >}}
+
+</div>
+
 ## Migration Stages

 ### Stage 0: pre migration
@@ -17,7 +25,7 @@ Is the pre-migration stage when having a functional ownCloud 10 instance.
 <div class="editpage">

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

@@ -37,10 +45,10 @@ _TODO allow limiting the web ui switch to an 'early adopters' group_
 </div>

 #### Validation
-Ensure switching back an forth between the classic ownCloud 10 web UI and ownCloud web works as at our https://demo.owncloud.com. 
+Ensure switching back an forth between the classic ownCloud 10 web UI and ownCloud web works as at our https://demo.owncloud.com.

 #### Rollback
-Should there be problems with ownCloud web at this point it can simply be removed from the menu and be undeployed. 
+Should there be problems with ownCloud web at this point it can simply be removed from the menu and be undeployed.

 #### Notes
 <div style="break-after: avoid"></div>
@@ -51,7 +59,7 @@ The ownCloud 10 demo instance uses OAuth to obtain a token for ownCloud web and
 _TODO make oauth2 in oc10 trust the new web ui, based on `redirect_uri` and CSRF so no explicit consent is needed?_

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

@@ -60,7 +68,7 @@ _Feel free to add your question as a PR to this document using the link at the t
 ### Stage 2: introduce OpenID Connect

 Basic auth requires us to properly store and manage user credentials. Something we would rather like to delegate to a tool specifically built for that task.
-While SAML and Shibboleth are protocols that solve that problem, they are limited to web clients. Desktop and mobile clients were an afterthought and keep running into timeouts. For these reasons, we decided to move to [OpenID Connect as our primary authentication protocol](https://owncloud.com/news/openid-connect-oidc-app/). 
+While SAML and Shibboleth are protocols that solve that problem, they are limited to web clients. Desktop and mobile clients were an afterthought and keep running into timeouts. For these reasons, we decided to move to [OpenID Connect as our primary authentication protocol](https://owncloud.com/news/openid-connect-oidc-app/).

 <div class="editpage">

@@ -69,7 +77,9 @@ _TODO @butonic add ADR for OpenID Connect and flesh out pros and cons of the abo
 </div>

 #### User impact
-When introducing OpenID Connect, the clients will detect the new authentication scheme when their current way of authenticating returns an error. Users will then have to reauthorize at the OpenID Connect IdP, which again, may be configured to skip the consent step for trusted clients.
+
+When introducing OpenID Connect, the clients will detect the new authentication scheme when their current way of authenticating returns an error. Users will then have to
+reauthorize at the OpenID Connect IdP, which again, may be configured to skip the consent step for trusted clients.

 #### Steps
 1. There are multiple products that can be used as an OpenID Connect IdP. We test with [LibreGraph Connect](https://github.com/libregraph/lico), which is also [embedded in oCIS](https://github.com/owncloud/web/). Other alternatives include [Keycloak](https://www.keycloak.org/) or [Ping](https://www.pingidentity.com/). Please refer to the corresponding setup instructions for the product you intent to use.
@@ -102,7 +112,7 @@ While OpenID Connect providers will send an `iss` and `sub` claim that relying p
 <div class="editpage">

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

@@ -114,7 +124,7 @@ Before letting oCIS handle end user requests we will first make it available in

 Start oCIS backend and make read only tests on existing data using the `owncloudsql` storage driver which will read (and write)
 - blobs from the same datadirectory layout as in ownCloud 10
- metadata from the ownCloud 10 database: 
+- metadata from the ownCloud 10 database:
 The oCIS share manager will read share information from the ownCloud database using an `owncloud` driver as well.

 <div class="editpage">
@@ -172,7 +182,7 @@ Roll out new clients that understand the spaces API and know how to convert loca
 One pair for `/webdav/home` or `/dav/files/<username>/home` and another pair for every accepted share. The shares will be accessible at `/webdav/shares/` when the server side enables the spaces API. Files can be identified using the `uuid`  and moved to the correct sync pair.

 ### Stage-4.1
-When reading the files from oCIS return the same `uuid`. It can be migrated to an extended attribute or it can be read from oc10. If users change it the client will not be able to detect a move and maybe other weird stuff happens. *What if the uuid gets lost on the server side due to a partial restore?* 
+When reading the files from oCIS return the same `uuid`. It can be migrated to an extended attribute or it can be read from oc10. If users change it the client will not be able to detect a move and maybe other weird stuff happens. *What if the uuid gets lost on the server side due to a partial restore?*

 {{< /hint >}}
 </div>
@@ -185,7 +195,7 @@ When reading the files from oCIS return the same `uuid`. It can be migrated to a
 2. Use curl to list spaces using graph drives endpoint

 ##### owncloud flavoured WebDAV endpoint
-1. Deploy Ocdav
+1. Deploy ocdav
 2. Use curl to send PROPFIND

 ##### data provider for up and download
@@ -196,13 +206,13 @@ When reading the files from oCIS return the same `uuid`. It can be migrated to a
 Deploy ...

 ##### share manager
-Deploy share manager with owncloud driver
+Deploy share manager with ownCloud driver

 ##### reva gateway
 1. Deploy gateway to authenticate requests? I guess we need that first... Or we need the to mint a token. Might be a good exercise.

 ##### automated deployment
-Finally, deploy OCIS with a config to set up everything running in a single oCIS runtime or in multiple containers.
+Finally, deploy oCIS with a config to set up everything running in a single oCIS runtime or in multiple containers.

 #### Rollback
 You can stop the oCIS process at any time.
@@ -214,7 +224,7 @@ Multiple ownCloud instances can be merged into one oCIS instance. The file ids w
 <div class="editpage">

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

@@ -252,7 +262,7 @@ With write access it becomes possible to manipulate existing files and shares.
 <div class="editpage">

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

@@ -264,10 +274,10 @@ In the previous stages oCIS was only accessible for administrators with access t
 #### User impact
 The IP address of the ownCloud host changes. There is no change for the file sync and share functionality when requests are handled by the oCIS codebase as it uses the same database and storage system as owncloud 10.

-#### Steps and verifications 
+#### Steps and verifications

 ##### Deploy oCIS proxy
-1. Deploy the `ocis proxy` 
+1. Deploy the `ocis proxy`
 2. Verify the requests are routed based on the ownCloud 10 routing policy `oc10` by default

 ##### Test user based routing
@@ -291,7 +301,7 @@ The proxy is stateless, multiple instances can be deployed as needed.
 <div class="editpage">

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

@@ -331,19 +341,41 @@ _TODO @butonic we need a canary app that allows users to decide for themselves w
 <div style="break-after: page"></div>

 #### Notes
-Running the two systems in parallel stage 
+Running the two systems in parallel stage
 Try to keep the duration of this stage short. Until now we only added services and made the system more complex. oCIS aims to reduce the maintenance cost of an ownCloud instance. You will not get there if you keep both systems alive.

 <div class="editpage">

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

 <div style="break-after: page"></div>

-### Stage-7: shut down ownCloud 10 
+### Stage-7: introduce spaces using ocis
+To encourage users to switch you can promote the workspaces feature that is built into oCIS. The ownCloud 10 storage backend can be used for existing users. New users and group or project spaces can be provided by storage providers that better suit the underlying storage system.
+
+#### Steps
+First, the admin needs to
+- deploy a storage provider with the storage driver that best fits the underlying storage system and requirements.
+- register the storage in the storage registry with a new storage id (we recommend a uuid).
+
+Then a user with the necessary create storage space role can create a storage space and assign Managers.
+
+<div class="editpage">
+
+_TODO @butonic a user with management permission needs to be presented with a list of storage spaces where he can see the amount of free space and decide on which storage provider the storage space should be created. For now a config option for the default storage provider for a specific type might be good enough._
+
+</div>
+
+#### Verification
+The new storage space should show up in the `/graph/drives` endpoint for the managers and the creator of the space.
+
+#### Notes
+Depending on the requirements and acceptable tradeoffs, a database less deployment using the ocis or s3ng storage driver is possible. There is also a [cephfs driver](https://github.com/cs3org/reva/pull/1209) on the way, that directly works on the API level instead of POSIX.
+
+### Stage-8: shut down ownCloud 10
 Disable ownCloud 10 in the proxy, all requests are now handled by oCIS, shut down oc10 web servers and redis (or keep for calendar & contacts only? rip out files from oCIS?)

 #### User impact
@@ -355,7 +387,7 @@ _TODO @butonic recommend alternatives_

 </div>

-#### Steps 
+#### Steps
 1. Shut down the apache servers that are running the ownCloud 10 PHP code.
 2. DO NOT SHUT DOWN THE DATABASE, YET!

@@ -372,19 +404,19 @@ The database needs to remain online until the storage layer and share metadata h
 <div class="editpage">

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

 <div style="break-after: page"></div>

-### Stage 8: storage migration
+### Stage 9: storage migration
 To get rid of the database we will move the metadata from the old ownCloud 10 database into dedicated storage providers. This can happen in a user by user fashion. group drives can properly be migrated to group, project or workspaces in this stage.

 #### User impact
 Noticeable performance improvements because we effectively shard the storage logic and persistence layer.

-#### Steps 
+#### Steps
 1. User by user storage migration from `owncloud` or `ownclouds3` driver to `ocis`/`s3ng`/`cephfs`... currently this means copying the metadata from one storage provider to another using the cs3 api.
 2. Change the responsible storage provider for a storage space (e.g. a user home, a group or project space are a workspace) in the storage registry.

@@ -400,7 +432,7 @@ _TODO @butonic document how to manually do that until the storage registry can d
 Start with a test user, then move to early adopters and finally migrate all users.

 #### Rollback
-To switch the storage provider again the same storage space migration can be performed again: copy medatata and blob data using the CS3 api, then change the responsible storage provider in the storage registry.
+To switch the storage provider again the same storage space migration can be performed again: copy metadata and blob data using the CS3 api, then change the responsible storage provider in the storage registry.

 #### Notes
 <div style="break-after: avoid"></div>
@@ -411,13 +443,13 @@ The storage space migration will become a seamless feature in the future that al
 <div class="editpage">

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

 <div style="break-after: page"></div>

-### Stage-9: share metadata migration
+### Stage-10: share metadata migration
 Migrate share data to _yet to determine_ share manager backend and shut down ownCloud database.

 The ownCloud 10 database still holds share information in the `oc_share` and `oc_share_external` tables. They are used to efficiently answer queries about who shared what with whom. In oCIS shares are persisted using a share manager and if desired these grants are also sent to the storage provider so it can set ACLs if possible. Only one system should be responsible for the shares, which in case of treating the storage as the primary source effectively turns the share manager into a cache.
@@ -427,7 +459,7 @@ Depending on chosen the share manager provider some sharing requests should be f
  - For non HA scenarios they can be served from memory, backed by a simple json file.
  - TODO: implement share manager with redis / nats / ... key value store backend: use the micro store interface please ...

-#### Steps 
+#### Steps
 1. Start new share manager
 2. Migrate metadata using the CS3 API (copy from old to new)
 3. Shut down old share manager
@@ -437,7 +469,7 @@ Depending on chosen the share manager provider some sharing requests should be f

 _TODO for HA implement share manager with redis / nats / ... key value store backend: use the micro store interface please ..._
 _TODO for batch migration implement share data migration cli with progress that reads all shares via the cs3 api from one provider and writes them into another provider_
-_TODO for seamless migration implement tiered/chained share provider that reads share data from the old provider and writes new shares to the new one_
+_TODO for seamless migration implement tiered/chained share provider that reads share data from the old provider and writes newc shares to the new one_
 _TODO for storage provider as source of truth persist ALL share data in the storage provider. Currently, part is stored in the share manager, part is in the storage provider. We can keep both, but the the share manager should directly persist its metadata to the storage system used by the storage provider so metadata is kept in sync_

 </div>
@@ -456,11 +488,11 @@ To switch the share manager to the database one revert routing users to the new

 <div class="editpage">

-### Stage-10
-Profit! Well, on the one hand you do not need to maintain a clustered database setup and can rely on the storage system. On the other hand you are now in micro service wonderland and will have to relearn how to identify bottlenecks and scale oCIS accordingly. The good thing is that tools like jaeger and prometheus have evolved and will help you understand what is going on. But this is a different Topic. See you on the other side!
+### Stage-11
+Profit! Well, on the one hand you do not need to maintain a clustered database setup and can rely on the storage system. On the other hand you are now in microservice wonderland and will have to relearn how to identify bottlenecks and scale oCIS accordingly. The good thing is that tools like jaeger and prometheus have evolved and will help you understand what is going on. But this is a different topic. See you on the other side!

 #### FAQ
-_Feel free to add your question as a PR to this document using the link at the top of this page!_ 
+_Feel free to add your question as a PR to this document using the link at the top of this page!_

 </div>

@@ -473,7 +505,7 @@ The fundamental difference between ownCloud 10 and oCIS is that the file metadat

 ## Data that will be migrated

-Currently, oCIS focuses on file sync and share use cases. 
+Currently, oCIS focuses on file sync and share use cases.

 ### Blob data

@@ -503,7 +535,7 @@ data
 │   │   │   └── Notes.md.v1540305560
 │   │   └── ownCloud Manual.pdf.v1396628249
 │   ├── thumbnails
-│   │   └── 123                  
+│   │   └── 123
 │   │   │   ├── 2048-1536-max.png
 │   │   │   └── 32-32.png                 // the file id, eg. of /Photos/Portugal.jpg
 │   └── uploads
@@ -523,9 +555,9 @@ The *data directory* may also contain subfolders for ownCloud 10 applications li

 When an object storage is used as the primary storage all file blobs are stored by their file id and a prefix, eg.: `urn:oid:<fileid>`.

-The three types of blobs we need to migrate are stored in 
+The three types of blobs we need to migrate are stored in
 - `files` for file blobs, the current file content,
- `files_trashbin` for trashed files (and their versions) and 
+- `files_trashbin` for trashed files (and their versions) and
 - `files_versions` for file blobs of older versions.

 <div style="break-after: page"></div>
@@ -538,7 +570,7 @@ The `filecache` table itself has more metadata:

 | Field              | Type          | Null | Key | Default | Extra          | Comment        | Migration         |
 |--------------------|---------------|------|-----|---------|----------------|----------------|----------------|
-| `fileid`           | bigint(20)    | NO   | PRI | NULL    | auto_increment |                | MUST become the oCIS `opaqueid` of a file reference. `ocis` driver stores it in extendet attributes and can use numbers as node ids on disk. for eos see note below table |
+| `fileid`           | bigint(20)    | NO   | PRI | NULL    | auto_increment |                | MUST become the oCIS `opaqueid` of a file reference. `ocis` driver stores it in extended attributes and can use numbers as node ids on disk. for eos see note below table |
 | `storage`          | int(11)       | NO   | MUL | 0       |                | *the filecache holds metadata for multiple storages* | corresponds to an oCIS *storage space* |
 | `path`             | varchar(4000) | YES  |     | NULL    |                | *the path relative to the storages root* | MUST become the `path` relative to the storage root. `files` prefix needs to be trimmed. |
 | `path_hash`        | varchar(32)   | NO   |     |         |                | *mysql once had problems indexing long paths, so we stored a hash for lookup by path. | - |
@@ -556,7 +588,7 @@ The `filecache` table itself has more metadata:
 | `checksum`         | varchar(255)  | YES  |     | NULL    |                | *same as blob checksum* | SHOULD become the checksum in the storage provider. eos calculates it itself, `ocis` driver stores it in extended attributes |


-> Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. otherwise either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid 
+> Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. otherwise either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid

 <div style="break-after: page"></div>

@@ -7,6 +7,123 @@ geekdocEditPath: edit/master/docs/ocis
 geekdocFilePath: release_notes.md
 ---

+## ownCloud Infinite Scale 1.16.0 Technology Preview
+
+Version 1.16.0 brings bug fixes, new features and progress for ongoing feature implementations like 'Spaces' and application integrations. ownCloud Web comes with a couple of usability improvements (e.g., breadcrumb context menu, right-click menu for multi-select). Infinite Scale has got a revamped config handling that makes deployments easier and more flexible. Additionally, it enables easy and fast collaboration via public links.
+
+The most prominent changes in ownCloud Infinite Scale 1.16.0 and ownCloud Web 4.6.0 comprise:
+
+- ownCloud Web now provides a context menu in the navigation breadcrumb that allows users to conduct actions for the parent folder (e.g., sharing). [web#6044](https://github.com/owncloud/web/pull/6044)
+- It is now possible to edit files with integrated applications in public links. [cs3org/reva#2310](https://github.com/cs3org/reva/pull/2310)
+- Infinite Scale now provides the API endpoints to manage Spaces (e.g., add/remove users, manage their roles). [ocis#2740](https://github.com/owncloud/ocis/issues/2740) [cs3org/reva#2250](https://github.com/cs3org/reva/pull/2250)
+- The config handling in Infinite Scale has received a huge rework to better enable different deployment and configuration models (environment variables, single config file, service-specific config files). More information can be found in the [documentation](https://owncloud.dev/ocis/config/). [#2708](https://github.com/owncloud/ocis/pull/2708)
+- The right-click context menu in ownCloud Web now works when multiple files have been selected. [web#5973](https://github.com/owncloud/web/pull/5973)
+- ownCloud Web now shows accessibility-optimized tooltips with absolute dates on relative dates. [web#6037](https://github.com/owncloud/web/pull/6037)
+- Pagination in folders with many files now works properly again. [#6056](https://github.com/owncloud/web/pull/6056)
+- The s3ng metadata storage backend works again. [#2807](https://github.com/owncloud/ocis/pull/2807)
+- Improvements have been added to support more identity providers (e.g., Authelia). [cs3org/reva#2314](https://github.com/cs3org/reva/pull/2314)
+
+You can also read the full [ownCloud Infinite Scale changelog](https://github.com/owncloud/ocis/releases/tag/v1.16.0) and [ownCloud Web changelog](https://github.com/owncloud/web/releases/tag/v4.6.0) for further details on what has changed.
+
+### Breaking changes
+{{< hint warning >}}
+We are currently in a Tech Preview state and breaking changes may occur at any time. For more information see our [release roadmap]({{< ref "./release_roadmap" >}})
+{{< /hint >}}
+
+## ownCloud Infinite Scale 1.15.0 Technology Preview
+
+Version 1.15.0 brings improvements for the app provider (external application integrations) and more progress on the 'Spaces' feature. Public links now support multi-file and folder downloads as well as all other external application integrations. ownCloud Web 4.5.0 furthermore comes with improvements for use with the ownCloud Classic backend. 
+
+The most prominent changes in ownCloud Infinite Scale 1.15.0 and ownCloud Web 4.5.0 comprise:
+
+- Multi-file and folder downloads as well as other external application (Collabora Online, ONLYOFFICE, CodiMD, etc.) integrations now work in public links. [web#5924](https://github.com/owncloud/web/pull/5924)
+- New files (created/uploaded and file versions) will now be highlighted in ownCloud Web. [web#6020](https://github.com/owncloud/web/pull/6020)
+- When using ownCloud Web with the ownCloud Classic backend, Web will now automatically display app entries in the app switcher based on the entries in the app switcher of the Classic UI (e.g., Activity, Market) so that users can easily find and use the apps. [web#5996](https://github.com/owncloud/web/pull/5996)
+- The width of the right sidebar in the Files app of ownCloud Web has been reduced to make it better usable on medium-sized screens. [web#5983](https://github.com/owncloud/web/pull/5983)
+- ownCloud Web has received performance and other improvements for external application integrations. [web#5952](https://github.com/owncloud/web/pull/5952)
+- Spaces: A new API endpoint has been introduced that allows listing all Spaces in an installation. [ocis#2692](https://github.com/owncloud/ocis/pull/2692)
+- Spaces: A permission has been added to control which users can list all Spaces. [cs3org/reva#2207](https://github.com/cs3org/reva/pull/2207)
+- The app provider (for external application integrations) has received improvements for announcing and prioritizing applications as well as for error handling. [cs3org/reva#2230](https://github.com/cs3org/reva/pull/2230) [cs3org/reva#2263](https://github.com/cs3org/reva/pull/2263) [cs3org/reva#2258](https://github.com/cs3org/reva/pull/2258)
+- The configuration defaults have been revisited and improved towards better security. [ocis#2700](https://github.com/owncloud/ocis/issues/2700)
+- IPv6 support for Infinite Scale has been added. [ocis#2698](https://github.com/owncloud/ocis/pull/2698)
+- A capability for the 'Resharing' feature will now be correctly announced. [ocis#2690](https://github.com/owncloud/ocis/pull/2690)
+- Restoring a file version now works properly. [cs3org/reva#2270](https://github.com/cs3org/reva/pull/2270)
+
+You can also read the full [ownCloud Infinite Scale changelog](https://github.com/owncloud/ocis/releases/tag/v1.15.0) and [ownCloud Web changelog](https://github.com/owncloud/web/releases/tag/v4.5.0) for further details on what has changed.
+
+### Breaking changes
+{{< hint warning >}}
+We are currently in a Tech Preview state and breaking changes may occur at any time. For more information see our [release roadmap]({{< ref "./release_roadmap" >}})
+{{< /hint >}}
+
+## ownCloud Infinite Scale 1.14.0 Technology Preview
+
+Version 1.14.0 brings more progress on the backend for the 'Spaces' and 'Quota' features. ownCloud Web 4.4.0 has received performance and usability improvements.
+
+The most prominent changes in ownCloud Infinite Scale 1.14.0 and ownCloud Web 4.4.0 comprise:
+
+- The media viewer in ownCloud Web is now accessible and themeable. [web#5900](https://github.com/owncloud/web/pull/5900)
+- The share expiration date setting has been moved to a dropdown menu to better fit the interface. [web#5806](https://github.com/owncloud/web/pull/5806)
+- The performance of ownCloud Web has been improved by removing unnecessary requests and redirects. [web#5910](https://github.com/owncloud/web/pull/5910) [web#5893](https://github.com/owncloud/web/pull/5893) [web#5917](https://github.com/owncloud/web/pull/5917) 
+- It is now possible for the sysadmin to set a default quota for new Spaces. This way, users with the respective permission can create new Spaces but administrators still keep a leverage on storage usage. [ocis#2619](https://github.com/owncloud/ocis/pull/2619)
+- The permission to change Space quota is now enforced. [ocis#2650](https://github.com/owncloud/ocis/pull/2650)
+- The maximum chunk size for upload file chunking has been set to 100 MB which will make chunking apply more frequently resulting in more stable uploads. [ocis#2584](https://github.com/owncloud/ocis/pull/2584)
+- It is now possible to set a default storage path for Infinite Scale. [ocis#2590](https://github.com/owncloud/ocis/pull/2590)
+- Infinite Scale services now by default only listen on localhost to prevent accidental exposure. [ocis#2612](https://github.com/owncloud/ocis/pull/2612)
+- A capability for the user settings endpoint has been added to improve request handling in Web between when used with ownCloud Classic and Infinite Scale, respectively. [ocis#2655](https://github.com/owncloud/ocis/pull/2655)
+- Requests in public links are now authenticated properly paving the way for Office capabilities in public links. [ocis#2536](https://github.com/owncloud/ocis/pull/2536)
+
+You can also read the full [ownCloud Infinite Scale changelog](https://github.com/owncloud/ocis/releases/tag/v1.14.0) and [ownCloud Web changelog](https://github.com/owncloud/web/releases/tag/v4.4.0) for further details on what has changed.
+
+### Breaking changes
+{{< hint warning >}}
+We are currently in a Tech Preview state and breaking changes may occur at any time. For more information see our [release roadmap]({{< ref "./release_roadmap" >}})
+{{< /hint >}}
+
+## ownCloud Infinite Scale 1.13.0 Technology Preview
+
+Version 1.13.0 brings progress on the backend for the 'Spaces' feature. ownCloud Web and Infinite Scale now provide ZIP/TAR download for multiple files/folders and can integrate external file viewer/editor applications (e.g., Collabora Online, ONLYOFFICE, CodiMD, Microsoft Office Online). 
+
+The most prominent changes in ownCloud Infinite Scale 1.13.0 and ownCloud Web 4.3.0 comprise:
+
+- Infinite Scale and Web now allow downloading multiple files or folders as archives [ocis#2509](https://github.com/owncloud/ocis/pull/2509) [cs3org/reva#2088](https://github.com/cs3org/reva/pull/2088)
+- Infinite Scale and Web can now integrate external applications like file viewers/editors via the [cs3org/wopiserver](https://github.com/cs3org/wopiserver) (e.g., Collabora Online, ONLYOFFICE, CodiMD, Microsoft Office Online). [web#5805](https://github.com/owncloud/web/pull/5805)
+- The 'Shared with me' page in ownCloud Web now clearly separates pending, declined and accepted shares. Pending shares are always displayed prominently so that users are aware and can react accordingly. [web#5814](https://github.com/owncloud/web/pull/5814)
+- Legacy URLs (e.g., from the address bar, public links) from ownCloud Classic are now properly resolved after migrating to Infinite Scale and Web [cs3org/reva#1089](https://github.com/cs3org/reva/pull/1989)
+- A capability for the Favorites feature has been added [ocis#2599](https://github.com/owncloud/ocis/pull/2599)
+
+You can also read the full [ownCloud Infinite Scale changelog](https://github.com/owncloud/ocis/releases/tag/v1.13.0) and [ownCloud Web changelog](https://github.com/owncloud/web/releases/tag/v4.3.0) for further details on what has changed.
+
+### Breaking changes
+{{< hint warning >}}
+We are currently in a Tech Preview state and breaking changes may occur at any time. For more information see our [release roadmap]({{< ref "./release_roadmap" >}})
+{{< /hint >}}
+
+## ownCloud Infinite Scale 1.12.0 Technology Preview
+
+Version 1.12.0 is a maintenance release with the foundations for the 'Spaces' feature and for viewer/editor application integrations. The Infinite Scale backend has been further hardened by fixing known issues, improving error handling and stabilizing existing features. Apart from bugfixing, ownCloud Web 4.2.0 has received a number of usability and design improvements for sharing and the file list.
+
+The most prominent changes in ownCloud Infinite Scale 1.12.0 and ownCloud Web 4.2.0 comprise:
+
+- The Infinite Scale backend now supports the first parts of the 'Spaces' feature
+  - Creating a new Space is now possible via Graph API [#2471](https://github.com/owncloud/ocis/pull/2471)
+  - A new sharing role, `Manager`, has been introduced for Spaces [cs3org/reva#2065](https://github.com/cs3org/reva/pull/2065)
+  - A capability for Spaces has been added [cs3org/reva#2015](https://github.com/cs3org/reva/pull/2015)
+- Infinite Scale now provides an app provider and an app registry as a foundation for integrations with viewer/editor applications. [#2204](https://github.com/owncloud/ocis/pull/2204)
+- ownCloud Web now has a re-designed sharing role selection. [#5632](https://github.com/owncloud/web/pull/5632)
+- ownCloud Web now shows people in sharing as a collapsed list of avatars to save space. This can be expanded to show more details and the full list. [#5758](https://github.com/owncloud/web/pull/5758)
+- ownCloud Web now shows sharing information in file/folder details. [#5735](https://github.com/owncloud/web/issues/5735)
+- The file size calculation in ownCloud Web has been changed from base-2 (e.g., KB / Kibibyte) to base-10 (e.g., kB / Kilobyte) to match better with user expectations. [#5739](https://github.com/owncloud/web/pull/5739)
+- The URL encoding/decoding in ownCloud Web has been improved. [#5714](https://github.com/owncloud/web/issues/5714)
+- ownCloud Web now provides a robots.txt file. [#5762](https://github.com/owncloud/web/pull/5762)
+
+You can also read the full [ownCloud Infinite Scale changelog](https://github.com/owncloud/ocis/releases/tag/v1.12.0) and [ownCloud Web changelog](https://github.com/owncloud/web/releases/tag/v4.2.0) for further details on what has changed.
+
+### Breaking changes
+{{< hint warning >}}
+We are currently in a Tech Preview state and breaking changes may occur at any time. For more information see our [release roadmap]({{< ref "./release_roadmap" >}})
+{{< /hint >}}
+
 ## ownCloud Infinite Scale 1.11.0 Technology Preview

 Version 1.11.0 brings new features, usability improvements and bug fixes. ownCloud Web 4.1.0 now supports drag & drop and allows users to do actions (e.g., sharing) for the folder they are currently in.
@@ -0,0 +1,77 @@
+---
+title: "cephfs"
+date: 2021-09-13T15:36:00+01:00
+weight: 30
+geekdocRepo: https://github.com/owncloud/ocis
+geekdocEditPath: edit/master/docs/ocis/storage-backends/
+geekdocFilePath: cephfs.md
+---
+
+{{< toc >}}
+
+oCIS intends to make the aspects of existing storage systems available as transparently as possible, but the static sync algorithm of the desktop client relies on some form of recursive change time propagation on the server side to detect changes. While this can be bolted on top of existing file systems with inotify, the kernel audit or a fuse based overlay filesystem, a storage system that already implements this aspect is preferable. Aside from EOS, cephfs supports a recursive change time that oCIS can use to calculate an etag for the webdav API.
+
+## Development
+
+The cephfs development happens in a [Reva branch](https://github.com/cs3org/reva/pull/1209) and is currently driven by CERN. 
+
+## Architecture
+
+In the original approach the driver was based on the [localfs](https://github.com/cs3org/reva/blob/a8c61401b662d8e09175416c0556da8ef3ba8ed6/pkg/storage/utils/localfs/localfs.go) driver, relying on a locally mounted cephfs. It would interface with it using the POSIX apis. This has been changed to directly call the Ceph API using https://github.com/ceph/go-ceph. It allows using the ceph admin APIs to create subvolumes for user homes and maintain a file id to path mapping using symlinks.
+
+## Implemented Aspects
+The recursive change time built ino cephfs is used to implement the etag propagation expected by the ownCloud clients. This allows oCIS to pick up changes that have been made by external tools, bypassing any oCIS APIs. 
+
+Like other filesystems cephfs uses inodes and like most other filesystems inodes are reused. To get stable file identifiers the current cephfs driver assigns every node a file id and maintains a custom fileid to path mapping in a system directory:
+```
+/tmp/cephfs $ tree -a
+.
+├── reva
+│   └── einstein
+│       ├── Pictures
+│       └── welcome.txt
+└── .reva_hidden
+    ├── .fileids
+    │   ├── 50BC39D364A4703A20C58ED50E4EADC3_570078 -> /tmp/cephfs/reva/einstein
+    │   ├── 571EFB3F0ACAE6762716889478E40156_570081 -> /tmp/cephfs/reva/einstein/Pictures
+    │   └── C7A1397524D0419B38D04D539EA531F8_588108 -> /tmp/cephfs/reva/einstein/welcome.txt
+    └── .uploads
+```
+
+Versions are not file but snapshot based, a [native feature of cephfs](https://docs.ceph.com/en/latest/dev/cephfs-snapshots/). The driver maps entries in the native cephfs `.snap` folder to the CS3 api recycle bin concept and makes them available in the web UI using the versions sidebar. Snapshots can be triggered by users themselves or on a schedule.
+
+Trash is not implemented, as cephfs has no native recycle bin and instead relies on the snapshot functionality that can be triggered by end users. It should be possible to automatically create a snapshot before deleting a file. This needs to be explored.
+
+Shares [are be mapped to ACLs](https://github.com/cs3org/reva/pull/1209/files#diff-5e532e61f99bffb5754263bc6ce75f84a30c6f507a58ba506b0b487a50eda1d9R168-R224) supported by cephfs. The share manager is used to persist the intent of a share and can be used to periodically verify or reset the ACLs on cephfs.
+
+## Future work
+- The spaces concept matches cephfs subvolumes. We can implement the CreateStorageSpace call with that, keep track of the list of storage spaces using symlinks, like for the id based lookup.
+- The share manager needs a persistence layer.
+- Currently we persist using a single json file.
+- As it basically provides two lists, *shared with me* and *shared with others*, we could persist them directly on cephfs!
+  - If needed for redundancy, the share manager can be run multiple times, backed by the same cephfs
+  - To save disk io the data can be cached in memory, and invalidated using stat requests.
+- A good tradeoff would be a folder for each user with a json file for each list. That way, we only have to open and read a single file when the user want's to list the shares.    
+- To allow deprovisioning a user the data should by sharded by userid. That way all share information belonging to a user can easily be removed from the system. If necessary it can also be restored easily by copying the user specific folder back in place.
+- For consistency over metadata any file blob data, backups can be done using snapshots.
+- An example where einstein has shared a file with marie would look like this on disk:
+```
+/tmp/cephfs $ tree -a
+.
+├── reva
+│   └── einstein
+│       ├── Pictures
+│       └── welcome.txt
+├── .reva_hidden
+│   ├── .fileids
+│   │   ├── 50BC39D364A4703A20C58ED50E4EADC3_570078 -> /tmp/cephfs/reva/einstein
+│   │   ├── 571EFB3F0ACAE6762716889478E40156_570081 -> /tmp/cephfs/reva/einstein/Pictures
+│   │   └── C7A1397524D0419B38D04D539EA531F8_588108 -> /tmp/cephfs/reva/einstein/welcome.txt
+│   └── .uploads
+└── .reva_share_manager
+    ├── einstein
+    │   └── sharedWithOthers.json
+    └── marie
+        └── sharedWithMe.json
+```
+- The fileids should [not be based on the path](https://github.com/cs3org/reva/pull/1209/files#diff-eba5c8b77ccdd1ac570c54ed86dfa7643b6b30e5625af191f789727874850172R125-R127) and instead use a uuid that is also persisted in the extended attributes to allow rebuilding the index from scratch if necessary.