sevenhill/QSfera
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

separate proposed changes from terminology

Browse Source 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
This commit is contained in:
Jörn Friedrich Dreyer
2021-09-13 11:10:22 +00:00
parent 47c2c3ad4f
commit 93c833348e
9 changed files with 924 additions and 576 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/extensions/storage/proposedchanges.md
+114
View File
@@ -0,0 +1,114 @@
---
title: "Proposed Changes"
date: 2018-05-02T00:00:00+00:00
weight: 18
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/extensions/storage
geekdocFilePath: proposedchanges.md
---
Some architectural changes still need to be clarified or changed. Maybe an ADR is in order for all of the below.
## Reva Gateway changes
## A dedicated shares storage provider
Currently, the *gateway* treats `/home/shares` different than any other path: it will stat all children and calculate an etag to allow clients to discover changes in accepted shares. This requires the storage provider to cooperate and provide this special `/shares` folder in the root of a users home when it is accessed as a home storage, which is a config flag that needs to be set for every storage driver.
The `enable_home` flag will cause drivers to jail path based requests into a `<userlayout>` subfolder. In effect it divides a storage provider into multiple [*storage spaces*]({{< ref "#storage-spaces" >}}): when calling `CreateHome` a subfolder following the `<userlayout>` is created and market as the root of a users home. Both, the eos and ocis storage drivers use extended attributes to mark the folder as the end of the size aggregation and tree mtime propagation mechanism. Even setting the quota is possible like that. All this literally is a [*storage space*]({{< ref "#storage-spaces" >}}).
We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either
- iterating over the root of the storage and treating every folder following the `<userlayout>` as a `home` *storage space*,
- iterating over the root of the storage and treating every folder following a new `<projectlayout>` as a `project` *storage space*, or
- iterating over the root of the storage and treating every folder following a generic `<layout>` as a *storage space* for a configurable space type, or
- we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like
```
home=/var/lib/ocis/storage/home/{{substr 0 1 .Owner.Username}}/{{.Owner.Username}}
spaces=/spaces/var/lib/ocis/storage/projects/{{.Name}}
```
This would make the `GetHome()` call return the path to the *storage provider* including the relative path to the *storage space*. No need for a *storage provider* mounted at `/home`. This is just a UI alias for `/users/<userlayout>`. Just like a normal `/home/<username>` on a linux machine.
But if we have no `/home` where do we find the shares, and how can clients discover changes in accepted shares?
The `/shares` namespace should be provided by a *shares storage provider* that lists all accepted shares for the current user... but what about copy pasting links from the browser? Well this storage is only really needed to have a path to ocm shares that actually reside on other instances. In the UI the shares would be listed by querying a *share manager*. It returns ResourceIds, which can be stated to fetch a path that is then accessible in the CS3 global namespace. Two caveats:
- This only works for resources that are actually hosted by the current instance. For those it would leak the parent path segments to a shared resource.
- For accepted OCM shares there must be a path in the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}) that has to be the same for all users, otherwise they cannot copy and share those URLs.
Work on this is done in https://github.com/cs3org/reva/pull/1846
### The gateway should be responsible for path transformations
Currently, storage providers are aware af their mount point, coupling them tightly with the gateway.
Tracked in https://github.com/cs3org/reva/issues/578
Work is done in https://github.com/cs3org/reva/pull/1866
## URL escaped string representation of a CS3 reference
For the `/dav/spaces/` endpoint we need to encode the *reference* in a url compatible way.
1. We can separate the path using a `/`: `/dav/spaces/<spaceid>/<path>`
2. The `spaceid` currently is a cs3 resourceid, consisting of `<storageid>` and `<nodeid>`. Since the nodeid might contain `/` eg. for the local driver we have to urlencode the spaceid.
To access resources by id we need to make the `/dav/meta/<resourceid>` able to list directories... Otherwise id based navigation first has to look up the path. Or we use the libregraph api for id based navigation.
A *reference* is a logical concept. It identifies a [*resource*]({{< ref "#resources" >}}) and consists of a `<resource_id>` and a `<path>`. A `<resource_id>` consists of a `<storage_id>` and a `<node_id>`. They can be concatenated using the separators `!` and `:`:
```
<storage_id>!<node_id>:<path>
```
While all components are optional, only three cases are used:
| format | example | description |
|-|-|-|
| `!:<absolute_path>` | `!:/absolute/path/to/file.ext` | absolute path |
| `<storage_space>!:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space |
| `<storage_space>!<root>:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths |
`<storage_space>` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`.
A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path
| Name | Description | Who resolves it? |
|------|-------------|-|
| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource |
Now, the same file is accessed as a share
| Name | Description |
|------|-------------|
| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty |
The `:`, `!` and `$` are chosen from the set of [RFC3986 sub delimiters](https://tools.ietf.org/html/rfc3986#section-2.2) on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set:
| reference | interpretation |
|-|-|
| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted |
| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` |
## space providers
When looking up an id based resource the reference must use a logical space id, not a CS3 resource id. Otherwise id based requests, which only have a resourceid consisting of a storage id and a node id cannot be routed to the correct storage provider if the storage has moved from one storage provider to another.
if the registry routes based on the storageid AND the nodeid it has to keep a cache of all nodeids in order to route all requests for a storage space (which consists of storage it + nodeid) to the correct storage provider. the correct resourceid for a node in a storage space would be `<storageid>$<rootnodeid>!<nodeid>`. The `<storageid>$<rootnodeid>` part allow the storage registry to route all id based requests to the correct storage provider. This becomes relevant when the storage space was moved from one storage provider to another. The storage space id remains the same, but the internal address and port change.
TODO discuss to clarify further
## Storage drivers
### allow clients to send a uuid on upload
iOS clients can only queue single requests to be executed in the background. They queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause workflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant.
Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider.
Technically, this means that every storage driver needs to have a map of a `uuid` to an internal resource identifier. This internal resource identifier can be
- an eos fileid, because eos can look up files by id
- an inode if the filesystem and the storage driver support looking up by inode
- a path if the storage driver has no way of looking up files by id.
- In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date.
- to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `<parentuuid>:<childname>`, in order to trade writes for reads
- as a fallback a sync job can read the file id from the metadata of the resources and populate the uuid to internal id map.
The TUS upload can take metadata, for PUT we might need a header.
docs/extensions/storage/static/spacesregistry.drawio.svg
+434
View File
@@ -0,0 +1,434 @@
<svg host="65bd71144e" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="632px" height="901px" viewBox="-0.5 -0.5 632 901" content="&lt;mxfile pages=&quot;4&quot;&gt;&lt;diagram id=&quot;ivlTgcF_GoFNmAPzzRq-&quot; name=&quot;overview&quot;&gt;7ZxZd5s4FIB/jR/Tw+L10Us6zZl0mhOn26NsZFALiAE5jufXz72S2LGz1LHdhiQ9RRft0v24ki507Gnw8FdMIu8jd6jfsQznoWPPOpZlmsYA/kPJVkn63aESuDFzdKRcMGf/US00tHTNHJqUIgrOfcGisnDJw5AuRUlG4phvytFW3C+XGhGX1gTzJfHr0q/MEV7arv4ov/GBMtfTRQ8t3eCApJF1SxKPOHxTENmXHXsacy7UVfAwpT52XtovKt37HXezisU0FE9KYAx1ne+Jv9bN49OrOUjm20TQoGP1fchqsojhysWrTm8y5yuxgULyWL2ZbpDYpr0U83XoUCzIgFQbjwk6j8gS725gXoDME4EPIRMuV8z3p9znMYRDHkKkSSJi/pOmwo5l9+VPdifteui0iUMSTxaFeRGfuSFc+3SF9fXJgvo3PGGCcRQvoW8oZDm5p7FgMKrXlQgBcxxsRBZhrDNccCF4gLXloWiqmO5KSEYfdo6ImY0zKAjlARXxFqLk2qGSaN3IJv0mn2n9dPp4hUlmj7SQ6NntZnnnEwAu9ByQQb74gfoB9fNhYDzQAxpXVKj7Dwn0gF6yEMaahemdu22k79xAMp7JZzRZxizSvYkJQwdurBPodRlFjoi6lU2vbHZZPV2m1UtvQq0L95XMYfeFCSlTyfpAKpyLWcJivPpUbsgMcyq2oFSNUkSrpEgDuzCuWs0K2lCY69lkzfQiKetFURlgehnDrjVY1DRCq0llLq7kD0b2iBye4MFFCr9bdt9FaphyDclUASYKuXSY0DWMOAuFnAg97F3jHfTB1JD/sHOnIDMxpOVl2aAs1AKzImySWQ3CxizLZcOfasGMCIKVHuDA/rtGiE7SaZGGO/a4dJ9Cq8nCL8dYET+hncEM/w6i1Rnwt+Un2ab4+DDeGcUfs67jZvf5Og5BreYvUPmpz7CFNYWfwqQjLEwVGm7RpRfCLHR1A6HYifyDC+NvLnzEBl7PN2yFxcN8tYwv8MjZBY0xGgg0+Sl4pFIGfMFwnFTSDQUkGMX6vTJVbKyQkuRtPR1tuntpkz97zcefvbLjJmT505XJqnwpw6hrD2ez3h7u1NhC4qU24Uyj6bFuTwfvp0YdQq+o0TW+WXXANUEvI1GdelYTCs8CjwWZVWbmIcyVfsVcsetky2QllJlHRtlc8BjtekCQ1ATLuKUuS9ISn4c3l/skdBWXPtzd3agrny1iKp+2O6n2kYRQiQS1FWuRyOVHrK2j5C2CrNeCrAXZ6UHW65ZB1u02mGj9ZpA12m3H59pNzO+ZU4DW83m2g1q4vgNUSmxlxWUAWycM0xoxvSdvkWCDlmAtwU5PMPP3J1i65VjlV32/cRepIsVAxFKkoCXXhzm2/rD9pzrKbz7Nr77ptbZdY9XoRKwayZ9Dsmo4xt+WVX8Eq/rD34RVH3mS1Pn0yP732AlYiEtNIvifvAU+NPbSpd0Cf8tb4D3TKqn8eW+B57p9S32CWpB4LHrGM7cBA7eUOHLDJ+YBNkCeim1inPwdPMHeSQapcgVNSocJBkNc6ImMSyJ54JkPYpOGPyEXrRE781k8jRM1XDWT6JdrlZLvWSvBRzhm7uUYdVw610FNoPJZdwF0P6gQW23HkDWMsT3hsfC4y0PiX3Me6Xg182Zg4G+n6bC7grss5o91EFXqVToXP56hpDoU+2k/LKBb+TpeZu4dA+0nIUjsUpFJR3vBAhi0h73d5kMsFfi+XJUmeugSbpD8efb9yoG83a8ASDVBpyr6WlQyGgweyUi1ek9GaUS+WiW0FEcCL+uakzDwK13Mxl86tY1wDE+Ringh1pn11OLxt8Wj1eLxl/BIH5j4hp2B5p4Kftel4vXsQXeUDGwLAVhnMKgX2qtS9mLOdhs5W9zILHD2FaDaNSp7VS+Faq//SEY7oHoGwHzWaaHwaCc7JDQqZ5koSt2yWj6eno/7PcFaPj6dj4Oz4mPxqLrl4xEMSpB/vupoj69QdShElU5fuw3JnJoxVf2UyFESXovIc0Hkfve1FpGvtsK2G8lWHI7jks06FNlqGZ0v2bLlcMEEtN8r0c3n+Yc9S2T0WFM2IBMJ9VfYXpGulvswV9VhGQ2dzMcMxwxfKIHpIgj6aSw9ErrtovrMiNhviXhOe46ns/V6hyJiLaPzJeKTNg+n11dPsPl47JKQ/SeLJH67WD5z7o32nxk/h3tQ63hbXDdi+LuO9zahKLsgXy/LULpgfikwU1eNMjCHOw5pXgOYduUgpfdSYI4eyeiYwMx9TK4cGAAmsPcU1gLlw3ZQ77h0K1Eds6ALV3Sc1w9O45oy2n9m0Tq+tY5vR3F86456JeScuePb86y4TxENr2ZoqekPL0iz7Xo21u9H3V7O73ayabwWHnJvSUQDoBIKauHpTb0li/QOYLurd1a2XHvwcYKDjxfbcaMmO670oGxytrFez9Wmatj1DmXY1TI6pqtNw0dXRvp8qfDRlVTDPDPVrmlXkTR9c8twGIFlMfrGqFdG1Zlw49daoAp5RqkwSgVjyVVFfgF5Bdo5p5hfovMzAoLDH8oNRB76W7l8XvprZUbK1/BLb+crXmdv7uenNcZiLXvWT3hWGGp/7G/V+2OFV8vSwmU7SSg7LzOKg6JRnFTbHe1sNigtVVCEX/PCGFxY1p3ZB1HXbs7AKgMM2VrGVLM7csO3bMCYxEba8n0tHbqTFLuwdn0eh4OlsfLl54E85kD7a8A8gEFiVnafhkbdIGn83swLX+DOv22kFCn/QpR9+T8=&lt;/diagram&gt;&lt;/mxfile&gt;">
<defs/>
<g>
<rect x="10" y="200" width="620" height="390" fill="none" stroke="#666666" stroke-width="2" stroke-dasharray="6 6" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-end; justify-content: unsafe flex-start; width: 618px; height: 1px; padding-top: 587px; margin-left: 12px;">
<div style="box-sizing: border-box; font-size: 0; text-align: left; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #666666; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">
oCIS System
<br/>
[Software System]
</div>
</div>
</div>
</foreignObject>
<text x="12" y="587" fill="#666666" font-family="Helvetica" font-size="12px">
oCIS System...
</text>
</switch>
</g>
<ellipse cx="175" cy="23.33" rx="23.333333333333332" ry="23.333333333333332" fill="#08427b" stroke="none" pointer-events="all"/>
<path d="M 120 60.67 C 120 47.78 130.45 37.33 143.33 37.33 L 206.67 37.33 C 212.86 37.33 218.79 39.79 223.17 44.17 C 227.54 48.54 230 54.48 230 60.67 L 230 116.67 C 230 129.55 219.55 140 206.67 140 L 143.33 140 C 130.45 140 120 129.55 120 116.67 Z" fill="#08427b" stroke="none" pointer-events="all"/>
<ellipse cx="175" cy="23.33" rx="23.333333333333332" ry="23.333333333333332" fill="#08427b" stroke="none" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 89px; margin-left: 175px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #ffffff; line-height: 1.2; pointer-events: all; white-space: nowrap; ">
<b>
Einstein
</b>
<div>
[Person]
</div>
<br/>
<div>
End user
</div>
</div>
</div>
</div>
</foreignObject>
<text x="175" y="92" fill="#ffffff" font-family="Helvetica" font-size="12px" text-anchor="middle">
Einstein...
</text>
</switch>
</g>
<rect x="60" y="220" width="230" height="110" rx="11" ry="11" fill="#438dd5" stroke="#3c7fc0" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 228px; height: 1px; padding-top: 275px; margin-left: 61px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #ffffff; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">
<b>
Client
</b>
<div>
[Container: C++, Kotlin, Swift or Vue]
</div>
<br/>
<div>
A desktop, mobile or web Client
</div>
</div>
</div>
</div>
</foreignObject>
<text x="175" y="279" fill="#ffffff" font-family="Helvetica" font-size="12px" text-anchor="middle">
Client...
</text>
</switch>
</g>
<rect x="440" y="430" width="160" height="110" rx="11" ry="11" fill="#438dd5" stroke="#3c7fc0" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 158px; height: 1px; padding-top: 485px; margin-left: 441px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #ffffff; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">
<b>
Storage Space Registry
</b>
<div>
[Container: golang, HTTP, libregraph]
</div>
<br/>
<div>
Manages spaces for users
</div>
</div>
</div>
</div>
</foreignObject>
<text x="520" y="489" fill="#ffffff" font-family="Helvetica" font-size="12px" text-anchor="middle">
Storage Space Registry...
</text>
</switch>
</g>
<rect x="40" y="430" width="160" height="110" rx="11" ry="11" fill="#438dd5" stroke="#3c7fc0" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 158px; height: 1px; padding-top: 485px; margin-left: 41px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #ffffff; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">
<b>
Storage Provider
</b>
<div>
[Container: golang]
</div>
<br/>
<div>
Persists storage spaces using reva
</div>
</div>
</div>
</div>
</foreignObject>
<text x="120" y="489" fill="#ffffff" font-family="Helvetica" font-size="12px" text-anchor="middle">
Storage Provider...
</text>
</switch>
</g>
<rect x="40" y="670" width="160" height="110" rx="11" ry="11" fill="#999999" stroke="#8a8a8a" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 158px; height: 1px; padding-top: 725px; margin-left: 41px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #ffffff; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">
<b>
Storage System
</b>
<div>
[Software System]
</div>
<br/>
<div>
provides persistent storage
</div>
</div>
</div>
</div>
</foreignObject>
<text x="120" y="729" fill="#ffffff" font-family="Helvetica" font-size="12px" text-anchor="middle">
Storage System...
</text>
</switch>
</g>
<ellipse cx="467" cy="23.33" rx="23.333333333333332" ry="23.333333333333332" fill="#08427b" stroke="none" pointer-events="all"/>
<path d="M 412 60.67 C 412 47.78 422.45 37.33 435.33 37.33 L 498.67 37.33 C 504.86 37.33 510.79 39.79 515.17 44.17 C 519.54 48.54 522 54.48 522 60.67 L 522 116.67 C 522 129.55 511.55 140 498.67 140 L 435.33 140 C 422.45 140 412 129.55 412 116.67 Z" fill="#08427b" stroke="none" pointer-events="all"/>
<ellipse cx="467" cy="23.33" rx="23.333333333333332" ry="23.333333333333332" fill="#08427b" stroke="none" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 89px; margin-left: 467px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #ffffff; line-height: 1.2; pointer-events: all; white-space: nowrap; ">
<b>
Moss
</b>
<div>
[Person]
</div>
<br/>
<div>
Administrator
</div>
</div>
</div>
</div>
</foreignObject>
<text x="467" y="92" fill="#ffffff" font-family="Helvetica" font-size="12px" text-anchor="middle">
Moss...
</text>
</switch>
</g>
<path d="M 120 540 L 120 661.76" fill="none" stroke="#707070" stroke-width="2" stroke-miterlimit="10" stroke-dasharray="6 6" pointer-events="stroke"/>
<path d="M 120 667.76 L 116 659.76 L 120 661.76 L 124 659.76 Z" fill="#707070" stroke="#707070" stroke-width="2" stroke-miterlimit="10" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 640px; margin-left: 120px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #707070; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">
<div style="text-align: left">
<div style="text-align: center">
<b>
Reads from and writes to
</b>
</div>
<div style="text-align: center">
[POSIX, S3]
</div>
</div>
</div>
</div>
</div>
</foreignObject>
<text x="120" y="643" fill="#707070" font-family="Helvetica" font-size="11px" text-anchor="middle">
Reads from and writes to...
</text>
</switch>
</g>
<path d="M 117.5 330 L 118.98 421.77" fill="none" stroke="#707070" stroke-width="2" stroke-miterlimit="10" stroke-dasharray="6 6" pointer-events="stroke"/>
<path d="M 119.08 427.76 L 114.95 419.83 L 118.98 421.77 L 122.95 419.7 Z" fill="#707070" stroke="#707070" stroke-width="2" stroke-miterlimit="10" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 380px; margin-left: 118px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #707070; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">
<div style="text-align: left">
<div style="text-align: center">
<b>
Reads from and writes to
</b>
</div>
<div style="text-align: center">
[WebDAV, libregraph, CS3, tus]
</div>
</div>
</div>
</div>
</div>
</foreignObject>
<text x="118" y="383" fill="#707070" font-family="Helvetica" font-size="11px" text-anchor="middle">
Reads from and writes to...
</text>
</switch>
</g>
<path d="M 232.5 330 L 432.75 437.96" fill="none" stroke="#707070" stroke-width="2" stroke-miterlimit="10" stroke-dasharray="6 6" pointer-events="stroke"/>
<path d="M 438.03 440.81 L 429.09 440.53 L 432.75 437.96 L 432.89 433.49 Z" fill="#707070" stroke="#707070" stroke-width="2" stroke-miterlimit="10" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 386px; margin-left: 336px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #707070; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">
<div style="text-align: left">
<div style="text-align: center">
<b>
Manages the users Storage Spaces in
</b>
</div>
<div style="text-align: center">
[libregraph]
</div>
</div>
</div>
</div>
</div>
</foreignObject>
<text x="336" y="389" fill="#707070" font-family="Helvetica" font-size="11px" text-anchor="middle">
Manages the users Storage Spac...
</text>
</switch>
</g>
<path d="M 175 140 L 175 211.76" fill="none" stroke="#707070" stroke-width="2" stroke-miterlimit="10" stroke-dasharray="6 6" pointer-events="stroke"/>
<path d="M 175 217.76 L 171 209.76 L 175 211.76 L 179 209.76 Z" fill="#707070" stroke="#707070" stroke-width="2" stroke-miterlimit="10" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 180px; margin-left: 175px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #707070; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">
<div style="text-align: left">
<div style="text-align: center">
<b>
Manages resources with
</b>
</div>
<div style="text-align: center">
[Web UI or native clients]
</div>
</div>
</div>
</div>
</div>
</foreignObject>
<text x="175" y="183" fill="#707070" font-family="Helvetica" font-size="11px" text-anchor="middle">
Manages resources with...
</text>
</switch>
</g>
<path d="M 200 485 L 431.76 485" fill="none" stroke="#707070" stroke-width="2" stroke-miterlimit="10" stroke-dasharray="6 6" pointer-events="stroke"/>
<path d="M 437.76 485 L 429.76 489 L 431.76 485 L 429.76 481 Z" fill="#707070" stroke="#707070" stroke-width="2" stroke-miterlimit="10" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 485px; margin-left: 320px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #707070; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">
<div style="text-align: left">
<div style="text-align: center">
<b>
Registers itself at and
<br/>
sends space root etag changes to
</b>
</div>
<div style="text-align: center">
[CS3, libregraph?, PUSH]
</div>
</div>
</div>
</div>
</div>
</foreignObject>
<text x="320" y="488" fill="#707070" font-family="Helvetica" font-size="11px" text-anchor="middle">
Registers itself at and...
</text>
</switch>
</g>
<path d="M 471.31 140 L 474.49 191.78" fill="none" stroke="#707070" stroke-width="2" stroke-miterlimit="10" stroke-dasharray="6 6" pointer-events="stroke"/>
<path d="M 474.86 197.77 L 470.38 190.03 L 474.49 191.78 L 478.36 189.54 Z" fill="#707070" stroke="#707070" stroke-width="2" stroke-miterlimit="10" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 170px; margin-left: 473px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #707070; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">
<div style="text-align: left">
<div style="text-align: center">
<b>
Manages organizational Storage Spaces in
</b>
</div>
<div style="text-align: center">
[WebDAV, libregraph, CS3, CLI]
</div>
</div>
</div>
</div>
</div>
</foreignObject>
<text x="473" y="173" fill="#707070" font-family="Helvetica" font-size="11px" text-anchor="middle">
Manages organizational Storage...
</text>
</switch>
</g>
<rect x="395" y="670" width="160" height="110" rx="11" ry="11" fill="#999999" stroke="#8a8a8a" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 158px; height: 1px; padding-top: 725px; margin-left: 396px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #ffffff; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">
<b>
Identity Management System
</b>
<div>
[Software System]
</div>
<br/>
<div>
provides users and groups
</div>
</div>
</div>
</div>
</foreignObject>
<text x="475" y="729" fill="#ffffff" font-family="Helvetica" font-size="12px" text-anchor="middle">
Identity Management System...
</text>
</switch>
</g>
<path d="M 475 590 L 475 661.76" fill="none" stroke="#707070" stroke-width="2" stroke-miterlimit="10" stroke-dasharray="6 6" pointer-events="stroke"/>
<path d="M 475 667.76 L 471 659.76 L 475 661.76 L 479 659.76 Z" fill="#707070" stroke="#707070" stroke-width="2" stroke-miterlimit="10" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 640px; margin-left: 475px;">
<div style="box-sizing: border-box; font-size: 0; text-align: center; ">
<div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #707070; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">
<div style="text-align: left">
<div style="text-align: center">
<b>
Authenticates users and searches recipients with
</b>
</div>
<div style="text-align: center">
[OpenID Connect, LDAP, REST]
</div>
</div>
</div>
</div>
</div>
</foreignObject>
<text x="475" y="643" fill="#707070" font-family="Helvetica" font-size="11px" text-anchor="middle">
Authenticates users and search...
</text>
</switch>
</g>
<rect x="0" y="790" width="620" height="110" fill="none" stroke="none" pointer-events="all"/>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe flex-start; width: 612px; height: 1px; padding-top: 780px; margin-left: 5px;">
<div style="box-sizing: border-box; font-size: 0; text-align: left; max-height: 120px; overflow: hidden; ">
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">
<h1>
C4 Container diagram for the oCIS System
</h1>
<p>
As a platform, the oCIS system may not only includes web, mobile and desktop clients but also the underlying storage system or an identity management system
</p>
<p>
Date: 2021-07-22T16:43
</p>
</div>
</div>
</div>
</foreignObject>
<text x="5" y="792" fill="#000000" font-family="Helvetica" font-size="12px">
C4 Container diagram for the oCIS System...
</text>
</switch>
</g>
</g>
<switch>
<g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/>
<a transform="translate(0,-5)" xlink:href="https://www.diagrams.net/doc/faq/svg-export-text-problems" target="_blank">
<text text-anchor="middle" font-size="10px" x="50%" y="100%">
Viewer does not support full SVG 1.1
</text>
</a>
</switch>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 33 KiB

docs/extensions/storage/static/storageprovider.drawio.svg
+308 -82
View File
File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 97 KiB

After

Width:  |  Height:  |  Size: 26 KiB

docs/extensions/storage/static/storageregistry-spaces.drawio.svg
-327
View File
File diff suppressed because one or more lines are too long
Side by Side

Before

Width:  |  Height:  |  Size: 149 KiB

docs/extensions/storage/storages.md
+9 -1
View File
@@ -78,7 +78,15 @@ The storage keeps an activity history, tracking the different actions that have
## Storage drivers
Reva currently has four storage driver implementations that can be used for *storage providers* an well as *data providers*.
Reva currently has several storage driver implementations that can be used for *storage providers* an well as *data providers*.
### OCIS and S3NG Storage Driver
The oCIS storage driver is the default storage driver. It decomposes the metadata and persists it in a POSIX filesystem. Blobs are stored on the filesystem as well. The layout makes extensive use of symlinks and extended attributes. A filesystem like xfs or zfs without inode size limitations is recommended. We will evolve this to further integrate with file systems like cephfs or gpfs.
The S3NG storage driver uses the same metadata layout on a POSIX storage as the oCIS driver, but it uses S3 as the blob storage.
TODO add list of capabilities / tradeoffs
### Local Storage Driver
docs/extensions/storage/terminology.md
+32 -126
View File
@@ -9,66 +9,50 @@ geekdocFilePath: terminology.md
Communication is hard. And clear communication is even harder. You may encounter the following terms throughout the documentation, in the code or when talking to other developers. Just keep in mind that whenever you hear or read *storage*, that term needs to be clarified, because on its own it is too vague. PR welcome.
## Resources
A *resource* is a logical concept. Resources can be of [different types](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceType):
## Logical concepts
### Resources
A *resource* is the basic building block that oCIS manages. It can be of [different types](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceType):
- an actual *file*
- a *container*, e.g. a folder or bucket
- a *symlink*, or
- a [*reference*]({{< ref "#references" >}}) which can point to a resource in another [*storage provider*]({{< ref "#storage-providers" >}})
## References
### References
A *reference* is a logical concept that identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) consists of either
- a *path* based reference, used to identify a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It must start with a `/`.
- a [CS3 *id* based reference](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId), uniquely identifying a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It consists of a `storage provider id` and an `opaque id`. The `storage provider id` must NOT start with a `/`.
{{< hint info >}}
The `/` is important because currently the static [*storage registry*]({{< ref "#storage-space-registries" >}}) uses a map to look up which [*storage provider*]({{< ref "#storage-providers" >}}) is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map.
{{< /hint >}}
A *reference* identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) can carry a *path* and a [CS3 *resource id*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The references come in two flavors: absolute and combined.
Absolute references have either the *path* or the *resource id* set:
- An absolute *path* MUST start with a `/`. The *resource id* MUST be empty.
- An absolute *resource id* uniquely identifies a [*resource*]({{< ref "#resources" >}}) and is used as a stable identifier for sharing. The *path* MUST be empty.
Combined references have both, *path* and *resource id* set:
- the *resource id* identifies the root [*resource*]({{< ref "#resources" >}})
- the *path* is relative to that root. It MUST start with `.`
{{< hint warning >}}
### Alternative: reference triple ####
A *reference* is a logical concept. It identifies a [*resource*]({{< ref "#resources" >}}) and consists of
a `storage_space`, a `<root_id>` and a `<path>`
```
<storage_space>!<root_id>:<path>
```
While all components are optional, only three cases are used:
| format | example | description |
|-|-|-|
| `!:<absolute_path>` | `!:/absolute/path/to/file.ext` | absolute path |
| `<storage_space>!:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space |
| `<storage_space>!<root>:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths |
### Storage Spaces
A *storage space* organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*),
a *quota*, *permissions* and is identified by a `storage space id`.
`<storage_space>` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`.
{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}}
A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path
Examples would be every user's personal storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ...
| Name | Description | Who resolves it? |
|------|-------------|-|
| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource |
Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs.
Now, the same file is accessed as a share
| Name | Description |
|------|-------------|
| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty |
### Shares
*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file indvidual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.*
The `:`, `!` and `$` are chosen from the set of [RFC3986 sub delimiters](https://tools.ietf.org/html/rfc3986#section-2.2) on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set:
| reference | interpretation |
|-|-|
| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted |
| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present |
| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` |
### Storage Space Registries
{{< /hint >}}
A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: it is used by *clients* to look up storage spaces a user has access to, the `/dav/spaces` endpoint to access it via WabDAV, and where the client should mount it in the users personal namespace.
## Storage Drivers
{{< svg src="extensions/storage/static/spacesregistry.drawio.svg" >}}
## Technical concepts
### Storage Drivers
A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}):
@@ -77,38 +61,14 @@ It maps the *path* and *id* based CS3 *references* to an appropriate [*storage s
- posix inodes or paths
- deconstructed filesystem nodes
{{< hint warning >}}
**Proposed Change**
iOS clients can only queue single requests to be executed in the background. The queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause worksflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant.
Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider.
Technically, this means that every storage driler needs to have a map of a `uuid` to in internal resource identifier. This internal resource identifier can be
- an eos fileid, because eos can look up files by id
- an inode if the filesystem and the storage driver support lookung up by inode
- a path if the storage driver has no way of looking up files by id.
- In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date.
- to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `<parentuuid>:<childname>`, allowing to trade writes for reads
{{< /hint >}}
## Storage Providers
### Storage Providers
A *storage provider* manages [*resources*]({{< ref "#resources" >}}) identified by a [*reference*]({{< ref "#references" >}})
by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}).
{{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}}
{{< hint warning >}}
**Proposed Change**
A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}})
by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}).
{{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}}
By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space.
{{< /hint >}}
## Storage Space Registries
### Storage Registry
A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}):
It is used by the *gateway*
@@ -117,65 +77,11 @@ that should handle a [*reference*]({{< ref "#references" >}}).
{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}}
{{< hint warning >}}
**Proposed Change**
A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*:
It is used by the *gateway*
to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}})
that is currently serving a [*storage space*]({{< ref "#storage-space" >}}).
{{< svg src="extensions/storage/static/storageregistry-spaces.drawio.svg" >}}
By making *storage registries* aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can query them for a listing of all [*storage spaces*]({{< ref "#storage-spaces" >}}) a user has access to. Including his home, received shares, project folders or group drives. See [a WIP PR for spaces in the oCIS repo (#1827)](https://github.com/owncloud/ocis/pull/1827) for more info.
{{< /hint >}}
## Storage Spaces
A *storage space* is a logical concept:
It is a tree of [*resources*]({{< ref "#resources" >}})*resources*
with a single *owner* (*user* or *group*),
a *quota* and *permissions*, identified by a `storage space id`.
{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}}
Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ...
Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs.
## Shares
*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file indvidual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.*
## Storage Systems
### Storage Systems
Every *storage system* has different native capabilities like id and path based lookups, recursive change time propagation, permissions, trash, versions, archival and more.
A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "#storage-drivers" >}}).
There migt be multiple [*storage drivers*]({{< ref "#storage-drivers" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements.
## Gateways
### Gateways
A *gateway* acts as a facade to the storage related services. It authenticates and forwards API calls that are publicly accessible.
{{< hint warning >}}
**Proposed Change**
Currently, the *gateway* treats `/home/shares` different than any other path: it will stat all children and calculate an etag to allow clients to discover changes in accepted shares. This requires the storage provider to cooperate and provide this special `/shares` folder in the root of a users home when it is accessed as a home storage, which is a config flag that needs to be set for every storage driver.
The `enable_home` flag will cause drivers to jail path based requests into a `<userlayout>` subfolder. In effect it divides a storage provider into multiple [*storage spaces*]({{< ref "#storage-spaces" >}}): when calling `CreateHome` a subfolder following the `<userlayout>` is created and market as the root of a users home. Both, the eos and ocis storage drivers use extended attributes to mark the folder as the end of the size aggregation and tree mtime propagation mechanism. Even setting the quota is possible like that. All this literally is a [*storage space*]({{< ref "#storage-spaces" >}}).
We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either
- iterating over the root of the storage and treating every folder following the `<userlayout>` as a `home` *storage space*,
- iterating over the root of the storage and treating every folder following a new `<projectlayout>` as a `project` *storage space*, or
- iterating over the root of the storage and treating every folder following a generic `<layout>` as a *storage space* for a configurable space type, or
- we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like
```
home=/var/lib/ocis/storage/home/{{substr 0 1 .Owner.Username}}/{{.Owner.Username}}
spaces=/spaces/var/lib/ocis/storage/projects/{{.Name}}
```
This would make the `GetHome()` call return the path to the *storage provider* including the relative path to the *storage space*. No need for a *storage provider* mounted at `/home`. This is just a UI alias for `/users/<userlayout>`. Just like a normal `/home/<username>` on a linux machine.
But if we have no `/home` where do we find the shares, and how can clients discover changes in accepted shares?
The `/shares` namespace should be provided by a *storage provider* that lists all accepted shares for the current user... but what about copy pasting links from the browser? Well this storage is only really needed to have a path to ocm shares that actually reside on other instances. In the UI the shares would be listed by querying a *share manager*. It returns ResourceIds, which can be stated to fetch a path that is then accessible in the CS3 global namespace. Two caveats:
- This only works for resources that are actually hosted by the current instance. For those it would leak the parent path segments to a shared resource.
- For accepted OCM shares there must be a path in the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}) that has to be the same for all users, otherwise they cannot copy and share those URLs.
{{< /hint >}}
docs/extensions/storage/updating.md
+2 -2
View File
@@ -11,9 +11,9 @@ geekdocFilePath: updating.md
## Updating reva
1. Run `go get github.com/cs3org/reva@master`
1. Run `go get github.com/cs3org/reva@master` in all repos that depend on reva
2. Create a changelog entry containing changes that were done in [reva](https://github.com/cs3org/reva/commits/master)
3. Create a Pull Request to ocis-reva master with those changes
3. Create a Pull Request to ocis master with those changes
4. If test issues appear, you might need to adjust the tests
5. After the PR is merged, consider doing a [release of the storage submodule]({{< ref "releasing" >}})