merge master

This commit is contained in:
Michael Barz
2021-12-10 14:38:46 +01:00
783 changed files with 59360 additions and 45942 deletions
+2 -6
View File
@@ -10,14 +10,10 @@ geekdocFilePath: releasing.md
## Requirements
You need a working installation of [the Go programming language](https://golang.org/) installed to build the assets for a working release.
You need a working installation of [the Go programming language](https://golang.org/), [the Node runtime](https://nodejs.org/) and [the Yarn package manager](https://yarnpkg.com/) installed to build the assets for a working release.
## Releasing
After adding changes to the accounts package within oCIS and testing them locally, you want to update the compiled assets to the oCIS binary.
To achieve this, you have to run a Go command and add the results to your PR. The preferred way to do this is to run `make generate` in the root
of the repository and then commit the resulting changes to your branch/PR.
The accounts service doesn't have a dedicated release process. Simply commit your changes, make sure linting and unit tests pass locally and open a pull request.
### Package Hierarchy
+1
View File
@@ -130,6 +130,7 @@ curl -k 'https://localhost:9200/graph/v1.0/me/drives' -u einstein:relativity -v
}
}
]
```
As we can see the response already contains a space-aware dav endpoint, which we can use to upload files to the space:
+2 -5
View File
@@ -10,14 +10,11 @@ geekdocFilePath: releasing.md
## Requirements
You need a working installation of [the Go programming language](https://golang.org/) installed to build the assets for a working release.
You need a working installation of [the Go programming language](https://golang.org/), [the Node runtime](https://nodejs.org/) and [the Yarn package manager](https://yarnpkg.com/) installed to build the assets for a working release.
## Releasing
After adding changes to the settings package within oCIS and testing them locally, you want to update the compiled assets to the oCIS binary.
To achieve this, you have to run a Go command and add the results to your PR. The preferred way to do this is to run `make generate` in the root
of the repository and then commit the resulting changes to your branch/PR.
The settings service doesn't have a dedicated release process. Simply commit your changes, make sure linting and unit tests pass locally and open a pull request.
### Package Hierarchy
+15
View File
@@ -16,4 +16,19 @@ The storage extension wraps [reva](https://github.com/cs3org/reva/) and adds an
*Clients* will use the *Spaces Registry* to poll or get notified about changes in all *Spaces* a user has access to. Every *Space* has a dedicated `/dav/spaces/<spaceid>` WebDAV endpoint that is served by a *Spaces Provider* which uses a specific reva storage driver to wrap an underlying *Storage System*.
{{< svg src="extensions/storage/static/overview.drawio.svg" >}}
The dashed lines in the diagram indicate requests that are made to authenticate requests or lookup the storage provider:
1. After authenticating a request, the proxy may either use the CS3 `userprovider` or the accounts service to fetch the user information that will be minted into the `x-access-token`.
2. The gateway will verify the JWT signature of the `x-access-token` or try to authenticate the request itself, e.g. using a public link token.
{{< hint warning >}}
The bottom part is lighter because we will deprecate it in favor of using only the CS3 user and group providers after moving some account functionality into reva and glauth. The metadata storage is not registered in the reva gateway to separate metadata necessary for running the service from data that is being served directly.
{{< /hint >}}
## Endpoints and references
In order to reason about the request flow, two aspects in the architecture need to be understood well:
1. What kind of [*namespaces*]({{< ref "./namespaces.md" >}}) are presented at the different WebDAV and CS3 endpoints?
2. What kind of [*resource*]({{< ref "./terminology.md#resources" >}}) [*references*]({{< ref "./terminology.md#references" >}}) are exposed or required: path or id based?
{{< svg src="extensions/storage/static/storage.drawio.svg" >}}
+458
View File
@@ -0,0 +1,458 @@
---
title: "Apps"
date: 2018-05-02T00:00:00+00:00
weight: 10
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/extensions/storage
geekdocFilePath: apps.md
---
oCIS is all about files. But most of the time you want to do something with files that is beyond the basic upload, download and share behavior. Therefore, oCIS has a concept for apps, that can handle specific file types, so called mime types.
## App provider capability
The capabilities endpoint (eg. `https://localhost:9200/ocs/v1.php/cloud/capabilities?format=json`) gives you following capabilities which are relevant for the app provider:
```json
{
"ocs": {
"data": {
"capabilities": {
"files": {
"app_providers": [
{
"enabled": true,
"version": "1.0.0",
"apps_url": "/app/list",
"open_url": "/app/open"
}
]
}
}
}
}
}
```
{{< hint info >}}
Please note that there might be two or more app providers with different versions. This is not be expected to happen on a regular basis. It was designed for a possible migration period for clients when the app provider needs a breaking change.
{{< /hint >}}
## App registry
The app registry is the single point where all apps register themselves and their respective supported mime types.
### Mime type configuration / creation allow list
The apps will register their supported mime types automatically, so that users can open supported files with them.
Administrators can set default applications on a per mime type basis and also allow the creation of new files for certain mime types. This per mime type configuration also features a description, file extension option and an icon.
In order to modify the mime type config you need to set `STORAGE_APP_REGISTRY_MIMETYPES_JSON=.../mimetypes.json` to a valid JSON file with content like this:
```json
[
{
"mime_type": "applition/vnd.oasis.opendocument.text",
"extension": "odt",
"name": "OpenDocument",
"description": "OpenDocument text document",
"icon": "https://some-website.test/opendocument-text-icon.png",
"default_app": "Collabora",
"allow_creation": true
},
{
"mime_type": "application/vnd.oasis.opendocument.spreadsheet",
"extension": "ods",
"name": "OpenSpreadsheet",
"description": "OpenDocument spreadsheet document",
"icon": "",
"default_app": "Collabora",
"allow_creation": false
}
]
```
Fields:
- `mime_type` is the mime type you want to configure
- `extension` is the file extension to be used for new files
- `name` is the name of the file / mime type
- `description` is a human readable description of the file / mime type
- `icon` URL to an icon which should be used for that mime type
- `default_app` name of the default app which opens this mime type when the user doesn't specify one
- `allow_creation` is wether a user should be able to create new file from that mime type (`true` or `false`)
### Listing available apps / mime types
Clients, for example ownCloud Web, need to offer users the available apps to open files and mime types for new file creation. This information can be obtained from this endpoint.
**Endpoint**: specified in the capabilities in `apps_url`, currently `/app/list`
**Method**: HTTP GET
**Authentication**: None
**Request example**:
```bash
curl 'https://ocis.test/app/list'
```
**Response example**:
HTTP status code: 200
```json
{
"mime-types": [
{
"mime_type": "application/pdf",
"ext": "pdf",
"app_providers": [
{
"name": "OnlyOffice",
"icon": "https://www.pikpng.com/pngl/m/343-3435764_onlyoffice-desktop-editors-onlyoffice-logo-clipart.png"
}
],
"name": "PDF",
"description": "PDF document"
},
{
"mime_type": "application/vnd.oasis.opendocument.text",
"ext": "odt",
"app_providers": [
{
"name": "Collabora",
"icon": "https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png"
},
{
"name": "OnlyOffice",
"icon": "https://www.pikpng.com/pngl/m/343-3435764_onlyoffice-desktop-editors-onlyoffice-logo-clipart.png"
}
],
"name": "OpenDocument",
"icon": "https://some-website.test/opendocument-text-icon.png",
"description": "OpenDocument text document",
"allow_creation": true,
"default_application": "Collabora"
},
{
"mime_type": "text/markdown",
"ext": "md",
"app_providers": [
{
"name": "CodiMD",
"icon": "https://avatars.githubusercontent.com/u/67865462?v=4"
}
],
"name": "Markdown file",
"description": "Markdown file",
"allow_creation": true,
"default_application": "CodiMD"
},
{
"mime_type": "application/vnd.ms-word.document.macroenabled.12",
"app_providers": [
{
"name": "Collabora",
"icon": "https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png"
},
{
"name": "OnlyOffice",
"icon": "https://www.pikpng.com/pngl/m/343-3435764_onlyoffice-desktop-editors-onlyoffice-logo-clipart.png"
}
]
},
{
"mime_type": "application/vnd.ms-powerpoint.template.macroenabled.12",
"app_providers": [
{
"name": "Collabora",
"icon": "https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png"
}
]
}
]
}
```
### Open a file with the app provider
**Endpoint**: specified in the capabilities in `open_url`, currently `/app/open`
**Method**: HTTP POST
**Authentication** (one of them):
- `Authorization` header with OIDC Bearer token for authenticated users or basic auth credentials (if enabled in oCIS)
- `Public-Token` header with public link token for public links
- `X-Access-Token` header with a REVA token for authenticated users
**Query parameters**:
- `file_id` (mandatory): id of the file to be opened
- `app_name` (optional)
- default (not given): default app for mime type
- possible values depend on the app providers for a mimetype from the `/app/open` endpoint
- `view_mode` (optional)
- default (not given): highest possible view mode, depending on the file permissions
- possible values:
- `write`: user can edit and download in the opening app
- `read`: user can view and download from the opening app
- `view`: user can view in the opening app (download is not possible)
**Request examples**:
```bash
curl -X POST 'https://ocis.test/app/open?file_id=ZmlsZTppZAo='
curl -X POST 'https://ocis.test/app/open?file_id=ZmlsZTppZAo=&app_name=Collabora'
curl -X POST 'https://ocis.test/app/open?file_id=ZmlsZTppZAo=&view_mode=read'
curl -X POST 'https://ocis.test/app/open?file_id=ZmlsZTppZAo=&app_name=Collabora&view_mode=write'
```
**Response examples**:
All apps are expected to be opened in an iframe and the response will give some parameters for that action.
There are apps, which need to be opened in the iframe with a form post. The form post must include all form parameters included in the response. For these apps the response will look like this:
HTTP status code: 200
```json
{
"app_url": "https://.....",
"method": "POST",
"form_parameters": {
"access_token": "eyJ0...",
"access_token_ttl": "1634300912000",
"arbitrary_param": "lorem-ipsum"
}
}
```
There are apps, which need to be opened in the iframe with a GET request. The GET request must have set all headers included in the response. For these apps the response will look like this:
HTTP status code: 200
```json
{
"app_url": "https://...",
"method": "GET",
"headers": {
"access_token": "eyJ0e...",
"access_token_ttl": "1634300912000",
"arbitrary_header": "lorem-ipsum"
}
}
```
**Example responses (error case)**:
- missing `file_id`
HTTP status code: 400
```json
{
"code": "INVALID_PARAMETER",
"message": "missing file ID"
}
```
- wrong `view_mode`
HTTP status code: 400
```json
{
"code": "INVALID_PARAMETER",
"message": "invalid view mode"
}
```
- unknown `app_name`
HTTP status code: 404
```json
{
"code": "RESOURCE_NOT_FOUND",
"message": "error: not found: app 'Collabor' not found"
}
```
- wrong / invalid file id
HTTP status code: 400
```json
{
"code": "INVALID_PARAMETER",
"message": "invalid file ID"
}
```
- file id does not point to a file
HTTP status code: 400
```json
{
"code": "INVALID_PARAMETER",
"message": "the given file id does not point to a file"
}
```
- file does not exist / unauthorized to open the file
HTTP status code: 404
```json
{
"code": "RESOURCE_NOT_FOUND",
"message": "file does not exist"
}
```
### Creating a file with the app provider
**Endpoint**: specified in the capabilities in `new_file_url`, currently `/app/new`
**Method**: HTTP POST
**Authentication** (one of them):
- `Authorization` header with OIDC Bearer token for authenticated users or basic auth credentials (if enabled in oCIS)
- `Public-Token` header with public link token for public links
- `X-Access-Token` header with a REVA token for authenticated users
**Query parameters**:
- `parent_container_id` (mandatory): ID of the folder in which the file will be created
- `filename` (mandatory): name of the new file
- `template` (optional): not yet implemented
**Request examples**:
```bash
curl -X POST 'https://ocis.test/app/new?parent_container_id=c2lkOmNpZAo=&filename=test.odt'
```
**Response example**:
You will receive a file id of the freshly created file, which you can use to open the file in an editor.
```json
{
"file_id": "ZmlsZTppZAo="
}
```
**Example responses (error case)**:
- missing `parent_container_id`
HTTP status code: 400
```json
{
"code": "INVALID_PARAMETER",
"message": "missing parent container ID"
}
```
- missing `filename`
HTTP status code: 400
```json
{
"code": "INVALID_PARAMETER",
"message": "missing filename"
}
```
- parent container not found
HTTP status code: 404
```json
{
"code": "RESOURCE_NOT_FOUND",
"message": "the parent container is not accessible or does not exist"
}
```
- `parent_container_id` does not point to a container
HTTP status code: 400
```json
{
"code": "INVALID_PARAMETER",
"message": "the parent container id does not point to a container"
}
```
- `filename` is invalid (eg. includes a path segment)
HTTP status code: 400
```json
{
"code": "INVALID_PARAMETER",
"message": "the filename must not contain a path segment"
}
```
- file already exists
HTTP status code: 403
```json
{
"code": "RESOURCE_ALREADY_EXISTS",
"message": "the file already exists"
}
```
## App drivers
App drivers represent apps, if the app is not able to register itself. Currently there is only the CS3org WOPI server app driver.
### CS3org WOPI server app driver
The CS3org WOPI server app driver is included in oCIS by default. It needs at least one WOPI compliant app (eg. Collabora, OnlyOffice or Microsoft Online Online Server) or a CS3org WOPI bridge supported app (CodiMD or Etherpad) and the CS3org WOPI server.
Here is a closer look at the configuration of the actual app provider in a docker-compose example (see also [full example](https://github.com/owncloud/ocis/blob/master/deployments/examples/ocis_wopi/docker-compose.yml)):
```yaml
services:
ocis:
image: owncloud/ocis:latest
...
environment:
...
STORAGE_GATEWAY_GRPC_ADDR: 0.0.0.0:9142 # make the REVA gateway accessible to the app drivers
ocis-appdriver-collabora:
image: owncloud/ocis:latest
command: storage-app-provider server # start only the app driver
environment:
STORAGE_GATEWAY_ENDPOINT: ocis:9142 # oCIS gateway endpoint
APP_PROVIDER_GRPC_ADDR: 0.0.0.0:9164
APP_PROVIDER_EXTERNAL_ADDR: ocis-appdriver-collabora:9164 # how oCIS can reach this app driver
OCIS_JWT_SECRET: ocis-jwt-secret
APP_PROVIDER_DRIVER: wopi
APP_PROVIDER_WOPI_DRIVER_APP_NAME: Collabora # will be used as name for this app
APP_PROVIDER_WOPI_DRIVER_APP_ICON_URI: https://www.collaboraoffice.com/wp-content/uploads/2019/01/CP-icon.png # will be used as icon for this app
APP_PROVIDER_WOPI_DRIVER_APP_URL: https://collabora.owncloud.test # endpoint of collabora
APP_PROVIDER_WOPI_DRIVER_INSECURE: false
APP_PROVIDER_WOPI_DRIVER_IOP_SECRET: wopi-iop-secret
APP_PROVIDER_WOPI_DRIVER_WOPI_URL: https://wopiserver.owncloud.test # endpoint of the CS3org WOPI server
```
+173 -4
View File
@@ -1,12 +1,181 @@
---
title: "Spaces"
date: 2018-05-02T00:00:00+00:00
weight: 3
date: 2020-04-27T18:46:00+01:00
weight: 38
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/extensions/storage
geekdocFilePath: spaces.md
---
{{< toc >}}
## Editing a Storage Space
The OData specification allows for a mirage of ways of addressing an entity. We will support addressing a Drive entity by its unique identifier, which is the one the graph-api returns when listing spaces, and its format is:
```json
{
"id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c"
}
```
This is an extract of an element of the list spaces response. An entire object has the following shape:
```json
{
"driveType": "project",
"id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c",
"lastModifiedDateTime": "2021-10-07T11:06:43.245418+02:00",
"name": "marketing",
"owner": {
"user": {
"id": "ddc2004c-0977-11eb-9d3f-a793888cd0f8"
}
},
"quota": {
"total": 65536
},
"root": {
"id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c",
"webDavUrl": "https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c"
}
}
```
### Updating a space property
Having introduced the above, one can refer to a Drive with the following URL format:
```console
'https://localhost:9200/graph/v1.0/Drive(1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570")
```
Updating an entity attribute:
```console
curl -X PATCH 'https://localhost:9200/graph/v1.0/Drive("1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570)' -d '{"name":"42"}' -v
```
The previous URL resource path segment (`Drive(1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570)`) is parsed and handed over to the storage registry in order to apply the patch changes in the body, in this case update the space name attribute to `42`. Since space names are not unique we only support addressing them by their unique identifiers, any other query would render too ambiguous and explode in complexity.
### Updating a space description
Since every space is the root of a webdav directory, following some conventions we can make use of this to set a default storage description and image. In order to do so, every space is created with a hidden `.space` folder at its root, which can be used to store such data.
```curl
curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!07c26b3a-9944-4f2b-ab33-b0b326fc7570/.space/description.md -d "Add a description to your spaces" -u admin:admin
```
Verify the description was updated:
```curl
curl -k https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!07c26b3a-9944-4f2b-ab33-b0b326fc7570/.space/description.md -u admin:admin
Add a description to your spaces
```
This feature makes use of the internal storage layout and is completely abstracted from the end user.
### Quotas
Spaces capacity (quota) is independent of the Storage quota. As a Space admin you can set the quota for all users of a space, and as such, there are no limitations and is up to the admin to make a correct use of this.
It is possible to have a space quota greater than the storage quota. A Space may also have "infinite" quota, meaning a single space without quota can occupy the entirety of a disk.
#### Quota Enforcement
Creating a Space with a quota of 10 bytes:
`curl -k -X POST 'https://localhost:9200/graph/v1.0/drives' -u admin:admin -d '{"name":"marketing", "quota": {"total": 10}}' -v`
```console
/var/tmp/ocis/storage/users
├── blobs
├── nodes
│   ├── 627981c2-2a71-4adf-b680-177e245afdda
│   ├── 9541e7c3-8fda-4b49-b697-e7e51457cf5a
│   ├── b5692345-108d-4b80-9747-3a7e9739ad57
│   └── root
│   ├── 118351d7-67a4-4cdf-b495-6093d1e572ed -> ../627981c2-2a71-4adf-b680-177e245afdda
│   └── ddc2004c-0977-11eb-9d3f-a793888cd0f8 -> ../b5692345-108d-4b80-9747-3a7e9739ad57
├── spaces
│   ├── personal
│   │   └── b5692345-108d-4b80-9747-3a7e9739ad57 -> ../../nodes/b5692345-108d-4b80-9747-3a7e9739ad57
│   ├── project
│   │   └── 627981c2-2a71-4adf-b680-177e245afdda -> ../../nodes/627981c2-2a71-4adf-b680-177e245afdda
│   └── share
├── trash
└── uploads
```
Verify the new space has 10 bytes, and none of it is used:
```json
{
"driveType": "project",
"id": "1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda",
"lastModifiedDateTime": "2021-10-15T11:16:26.029188+02:00",
"name": "marketing",
"owner": {
"user": {
"id": "ddc2004c-0977-11eb-9d3f-a793888cd0f8"
}
},
"quota": {
"remaining": 10,
"total": 10,
"used": 0
},
"root": {
"id": "1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda",
"webDavUrl": "https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda"
}
}
```
Upload a 6 bytes file:
`curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!627981c2-2a71-4adf-b680-177e245afdda/6bytes.txt -d "012345" -u admin:admin -v`
Query the quota again:
```json
...
"quota": {
"remaining": 4,
"total": 10,
"used": 6
},
...
```
Now attempt to upload 5 bytes to the space:
`curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!627981c2-2a71-4adf-b680-177e245afdda/5bytes.txt -d "01234" -u admin:admin -v`
The request will fail with `507 Insufficient Storage`:
```
HTTP/1.1 507 Insufficient Storage
< Access-Control-Allow-Origin: *
< Content-Length: 0
< Content-Security-Policy: default-src 'none';
< Date: Fri, 15 Oct 2021 09:24:46 GMT
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Robots-Tag: none
< X-Xss-Protection: 1; mode=block
<
* Connection #0 to host localhost left intact
* Closing connection 0
```
##### Considerations
- If a Space quota is updated to unlimited, the upper limit is the entire available space on disk
{{< hint warning >}}
The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this.
@@ -14,7 +183,7 @@ The current implementation in oCIS might not yet fully reflect this concept. Fee
{{< /hint >}}
## Storage Spaces
A storage *space* is a logical concept. It organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*),
A storage *space* is a logical concept. It organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*),
a *quota*, *permissions* and is identified by a `storage space id`.
{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}}
@@ -29,7 +198,7 @@ Finally, a logical `storage space id` is not tied to a specific [*spaces provide
## Notes
We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either
- iterating over the root of the storage and treating every folder following the `<user_layout>` as a `home` *storage space*,
- iterating over the root of the storage and treating every folder following the `<user_layout>` as a `home` *storage space*,
- iterating over the root of the storage and treating every folder following a new `<project_layout>` as a `project` *storage space*, or
- iterating over the root of the storage and treating every folder following a generic `<layout>` as a *storage space* for a configurable space type, or
- we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like
+1 -1
View File
@@ -98,7 +98,7 @@ The OCS service makes a stat request to the storage provider to get a [ResourceI
{{< hint >}}
The user and public share provider implementations identify the file using the [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The [`ResourceInfo`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo) is passed so the share provider can also store who the owner of the resource is. The *path* is not part of the other API calls, e.g. when listing shares.
The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.*
The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently logged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.*
{{< /hint >}}
+1 -1
View File
@@ -130,7 +130,7 @@ To provide the other storage aspects we plan to implement a FUSE overlay filesys
This is the current default storage driver. While it implements the file tree (using redis, including id based lookup), ETag propagation, trash, versions and sharing (including expiry) using the data directory layout of ownCloud 10 it has [known limitations](https://github.com/owncloud/core/issues/28095) that cannot be fixed without changing the actual layout on disk.
To setup it up properly in a distributed fashion, the storage-home and the storage-oc need to share the same underlying FS. Their "data" counterparts also need access to the same shared FS.
For a simple docker-compose setup, you can create a volume which will be used by the "storage-storage-home", "storage-storage-home-data", "storage-storage-oc" and "storage-storage-oc-data" containers. Using the `owncloud/ocis` docker image, the volume would need to be hooked in the `/var/tmp/ocis` folder inside the containers.
For a simple docker-compose setup, you can create a volume which will be used by the "storage-storage-home", "storage-storage-home-data", "storage-storage-oc" and "storage-storage-oc-data" containers. Using the `owncloud/ocis` docker image, the volume would need to be hooked in the `/var/lib/ocis` folder inside the containers.
- tree provided by a POSIX filesystem
- file layout is mapped to the old ownCloud 10 layout
+126
View File
@@ -26,10 +26,136 @@ Absolute references have either the *path* or the *resource id* set:
Combined references have both, *path* and *resource id* set:
- the *resource id* identifies the root [*resource*]({{< ref "#resources" >}})
- the *path* is relative to that root. It MUST start with `.`
## References
A *reference* is a logical concept that identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) consists of either
- a *path* based reference, used to identify a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It must start with a `/`.
- a [CS3 *id* based reference](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId), uniquely identifying a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It consists of a `storage provider id` and an `opaque id`. The `storage provider id` must NOT start with a `/`.
{{< hint info >}}
The `/` is important because currently the static [*storage registry*]({{< ref "#storage-space-registries" >}}) uses a map to look up which [*storage provider*]({{< ref "#storage-providers" >}}) is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map.
{{< /hint >}}
{{< hint warning >}}
### Alternative: reference triple ####
A *reference* is a logical concept. It identifies a [*resource*]({{< ref "#resources" >}}) and consists of
a `storage_space`, a `<root_id>` and a `<path>`
```
<storage_space>!<root_id>:<path>
```
While all components are optional, only three cases are used:
| format | example | description |
|-|-|-|
| `!:<absolute_path>` | `!:/absolute/path/to/file.ext` | absolute path |
| `<storage_space>!:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space |
| `<storage_space>!<root>:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths |
`<storage_space>` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`.
A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path
| Name | Description | Who resolves it? |
|------|-------------|-|
| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource |
Now, the same file is accessed as a share
| Name | Description |
|------|-------------|
| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty |
The `:`, `!` and `$` are chosen from the set of [RFC3986 sub delimiters](https://tools.ietf.org/html/rfc3986#section-2.2) on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set:
| reference | interpretation |
|-|-|
| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted |
| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` |
{{< /hint >}}
## Storage Drivers
A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}):
It maps the *path* and *id* based CS3 *references* to an appropriate [*storage system*]({{< ref "#storage-systems" >}}) specific reference, e.g.:
- eos file ids
- posix inodes or paths
- deconstructed filesystem nodes
{{< hint warning >}}
**Proposed Change**
iOS clients can only queue single requests to be executed in the background. The queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause workflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant.
Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider.
Technically, this means that every storage driver needs to have a map of a `uuid` to in internal resource identifier. This internal resource identifier can be
- an eos fileid, because eos can look up files by id
- an inode if the filesystem and the storage driver support looking up by inode
- a path if the storage driver has no way of looking up files by id.
- In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date.
- to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `<parentuuid>:<childname>`, allowing to trade writes for reads
{{< /hint >}}
## Storage Providers
## Technical concepts
### Storage Systems
{{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}}
{{< hint warning >}}
**Proposed Change**
A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}})
by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}).
{{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}}
By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space.
{{< /hint >}}
## Storage Space Registries
A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}):
It is used by the *gateway*
to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}})
that should handle a [*reference*]({{< ref "#references" >}}).
{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}}
{{< hint warning >}}
**Proposed Change**
A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*:
It is used by the *gateway*
to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}})
that is currently serving a [*storage space*]({{< ref "#storage-space" >}}).
{{< svg src="extensions/storage/static/storageregistry-spaces.drawio.svg" >}}
By making *storage registries* aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can query them for a listing of all [*storage spaces*]({{< ref "#storage-spaces" >}}) a user has access to. Including his home, received shares, project folders or group drives. See [a WIP PR for spaces in the oCIS repo (#1827)](https://github.com/owncloud/ocis/pull/1827) for more info.
{{< /hint >}}
## Storage Spaces
A *storage space* is a logical concept:
It is a tree of [*resources*]({{< ref "#resources" >}})*resources*
with a single *owner* (*user* or *group*),
a *quota* and *permissions*, identified by a `storage space id`.
{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}}
Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ...
Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs.
## Shares
*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavyweight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.*
## Storage Systems
Every *storage system* has different native capabilities like id and path based lookups, recursive change time propagation, permissions, trash, versions, archival and more.
A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "./storagedrivers.md" >}}).
There might be multiple [*storage drivers*]({{< ref "./storagedrivers.md" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements.
+2 -2
View File
@@ -50,10 +50,10 @@ If the below defaults don't match your environment change them accordingly:
```
export STORAGE_LDAP_HOSTNAME=localhost
export STORAGE_LDAP_PORT=9126
export STORAGE_LDAP_BASE_DN='dc=example,dc=org'
export STORAGE_LDAP_BASE_DN='dc=ocis,dc=test'
export STORAGE_LDAP_USERFILTER='(&(objectclass=posixAccount)(cn=%s))'
export STORAGE_LDAP_GROUPFILTER='(&(objectclass=posixGroup)(cn=%s))'
export STORAGE_LDAP_BIND_DN='cn=reva,ou=sysusers,dc=example,dc=org'
export STORAGE_LDAP_BIND_DN='cn=reva,ou=sysusers,dc=ocis,dc=test'
export STORAGE_LDAP_BIND_PASSWORD=reva
export STORAGE_LDAP_USER_SCHEMA_UID=uid
export STORAGE_LDAP_USER_SCHEMA_MAIL=mail
+4 -6
View File
@@ -31,9 +31,7 @@ and take note of its release tag name.
1. Create a branch `update-web-$version` in the [ocis repository](https://github.com/owncloud/ocis)
2. Change into web package folder via `cd web`
3. Inside `web/`, update the `Makefile` so that the WEB_ASSETS_VERSION variable references the currently released version of https://github.com/owncloud/web
4. Inside `web/`, replace the current assets with newly released ones by running `make pull-assets`
5. Inside `web/`, run `make generate`. The output should look something like this: `web: embed.go - YYY/MM/DD ... to write [./embed.go] from config file ...`
6. Move to the changelog (`cd ../changelog/`) and add a changelog file to the `unreleased/` folder (You can copy an old web release changelog item as a template)
7. Move to the repo root (`cd ..`)and update the WEB_COMMITID in the `/.drone.env` file to the commit id from the released version (unless the existing commit id is already newer)
8. **Optional:** Test the changes locally by running `cd ocis && go run cmd/ocis/main.go server`, visiting [https://localhost:9200](https://localhost:9200) and confirming everything renders correctly
9. Commit your changes, push them and [create a PR](https://github.com/owncloud/ocis/pulls)
4. Move to the changelog (`cd ../changelog/`) and add a changelog file to the `unreleased/` folder (You can copy an old web release changelog item as a template)
5. Move to the repo root (`cd ..`)and update the WEB_COMMITID in the `/.drone.env` file to the commit id from the released version (unless the existing commit id is already newer)
6. **Optional:** Test the changes locally by running `cd ocis && go run cmd/ocis/main.go server`, visiting [https://localhost:9200](https://localhost:9200) and confirming everything renders correctly
7. Commit your changes, push them and [create a PR](https://github.com/owncloud/ocis/pulls)