sevenhill/QSfera
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

migration sideeffects

Browse Source 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
This commit is contained in:
Jörn Friedrich Dreyer
2021-04-28 14:45:50 +00:00
parent f7aa2383fa
commit 205b3a72bd
4 changed files with 55 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/extensions/storage/architecture.md
+12 -12
View File
@@ -28,7 +28,7 @@ The ocdav service not only handles all WebDAV requests under `(remote.php/)(web)
| `(remote.php/)webdav` | ocdav | storageprovider | `/home` | the old webdav endpoint | |
| `(remote.php/)dav/files/<username>` | ocdav | storageprovider | `/home` | the new webdav endpoint | |
| `(remote.php/)dav/meta/<fileid>/v` | ocdav | storageprovider | id based | versions | |
| `(remote.php/)dav/trash-bin/<username>` | ocdav | recycle | - | trash | should aggregate the trash of storage spaces the user has access to |
| `(remote.php/)dav/trash-bin/<username>` | ocdav | recycle | - | trash | should aggregate the trash of [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) the user has access to |
| `(remote.php/)dav/public-files/<token>` | ocdav | storageprovider | `/public/<token>` | public links | |
| `(remote.php/)dav/avatars/<username>` | ocdav | - | - | avatars, hardcoded | look up from user provider and cache |
| *CernBox setup:* |||||
@@ -38,32 +38,32 @@ The ocdav service not only handles all WebDAV requests under `(remote.php/)(web)
| `(remote.php/)webdav/users` | ocdav | storageprovider | `/users` | | |
| `(remote.php/)dav/files/<username>` | ocdav | storageprovider | `/users/<userlayout>` | | |
| *Spaces concept also needs a new endpoint:* |||||
| `(remote.php/)dav/spaces/<spaceid>/<relative_path>` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to storage spaces or a global endpoint | allow accessing spaces, listing is done by the graph api |
| `(remote.php/)dav/spaces/<spaceid>/<relative_path>` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) or a global endpoint | allow accessing spaces, listing is done by the graph api |
The correct endpoint for a users home storage space in oc10 is `remote.php/dav/files/<username>`. In oc10 All requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home.
The correct endpoint for a users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) in oc10 is `remote.php/dav/files/<username>`. In oc10 All requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home.
In oc10 we originally had `remote.php/webdav` which would render the current users home storage space. The early versions (pre OC7) would jail all received shares into a `remote.php/webdav/shares` subfolder. The semantics for syncing such a folder are [not trivially predictable](https://github.com/owncloud/core/issues/5349), which is why we made shares [freely mountable](https://github.com/owncloud/core/pull/8026) anywhere in the users home.
In oc10 we originally had `remote.php/webdav` which would render the current users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The early versions (pre OC7) would jail all received shares into a `remote.php/webdav/shares` subfolder. The semantics for syncing such a folder are [not trivially predictable](https://github.com/owncloud/core/issues/5349), which is why we made shares [freely mountable](https://github.com/owncloud/core/pull/8026) anywhere in the users home.
The current reva implementation jails shares into a `remote.php/webdav/Shares` folder for performance reasons. Obviously, this brings back the [special semantics for syncing](https://github.com/owncloud/product/issues/7). In the future we will follow [a different solution](https://github.com/owncloud/product/issues/302) and jail the received shares into a dedicated `/shares` space, on the same level as `/home` and `/spaces`. We will add a dedicated [API to list all *storage spaces*](https://github.com/owncloud/ocis/pull/1827) a user has access to and where they are mounted in the users *namespace*.
{{< hint warning >}}
Existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy like CernBox did.
For legacy clients the `remote.php/webdav` endpoint will no longer list the users home directly, but instead present the different types of storage spaces:
- `remote.php/webdav/home`: the users home is pushed down into a new `home` *storage space*
- `remote.php/webdav/shares`: all mounted shares will be moved to a new `shares` *storage space*
- `remote.php/webdav/spaces`: other *storage spaces* the user has access to, e.g. group or project drives
- `remote.php/webdav/home`: the users home is pushed down into a new `home` [*storage space*]({{< ref "./terminology.md#storage-spaces" >}})
- `remote.php/webdav/shares`: all mounted shares will be moved to a new `shares` [*storage space*]({{< ref "./terminology.md#storage-spaces" >}})
- `remote.php/webdav/spaces`: other [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) the user has access to, e.g. group or project drives
{{< /hint >}}
{{< hint warning >}}
An alternative would be to introduce a new `remote.php/dav/spaces` or `remote.php/dav/global` endpoint. However, `remote.php/dav` properly follows the WebDAV RFCs strictly. To ensure that all resources under that namespace are scoped to the user the URL would have to include the principal like `remote.php/dav/spaces/<username>`, a precondition for e.g. WebDAV [RFC5397](https://tools.ietf.org/html/rfc5397). For a history lesson start at [Replace WebDAV with REST
An alternative would be to introduce a new `remote.php/dav/spaces` or `remote.php/dav/global` endpoint. However, `remote.php/dav` properly follows the WebDAV RFCs strictly. To ensure that all resources under that [*namespace*]({{< ref "./terminology.md#namespaces" >}}) are scoped to the user the URL would have to include the principal like `remote.php/dav/spaces/<username>`, a precondition for e.g. WebDAV [RFC5397](https://tools.ietf.org/html/rfc5397). For a history lesson start at [Replace WebDAV with REST
owncloud/core#12504](https://github.com/owncloud/core/issues/12504#issuecomment-65218491) which spawned [Add extra layer in DAV to accomodate for other services like versions, trashbin, etc owncloud/core#12543](https://github.com/owncloud/core/issues/12543)
{{< /hint >}}
### Sharing
The [OCS Share API](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html) endpoint `/ocs/v1.php/apps/files_sharing/api/v1/shares` returns shares, which have their own share id and reference files using a path relative to the users home. They API also lists the numeric storage id as well as the string type `storage_id` (which is confusing ... but yeah) which would allow constructing combined references with a storage spacle id and a path relative to the root of that storage space. The web UI however assumes that it can take the path from the `file_target` and append it to the users home to access it.
The [OCS Share API](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html) endpoint `/ocs/v1.php/apps/files_sharing/api/v1/shares` returns shares, which have their own share id and reference files using a path relative to the users home. They API also lists the numeric storage id as well as the string type `storage_id` (which is confusing ... but yeah) which would allow constructing combined references with a `storage space id` and a `path` relative to the root of that [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The web UI however assumes that it can take the path from the `file_target` and append it to the users home to access it.
{{< hint >}}
The API [already returns the storage id](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html#example-request-response-payloads-4) (and numeric id) in addition to the file id:
@@ -88,13 +88,13 @@ The OCS service makes a stat request to the storage provider to get a [ResourceI
{{< hint >}}
The user and public share provider implementations identify the file using the `ResourceId`. The `ResourceInfo` is passed so the share provider can also store who the owner of the resource is. The *path* is not part of the other API calls, e.g. when listing shares.
The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.
The user and public share provider implementations identify the file using the [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The [`ResourceInfo`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo) is passed so the share provider can also store who the owner of the resource is. The *path* is not part of the other API calls, e.g. when listing shares.
The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.*
{{< /hint >}}
### User and Group provisioning
In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `<username>` instead of `<userid>`. You may have encountered `<userlayout>`, which refers to a template that can be configuted to build several path segments by filling in user properties, e.g. the first two characters of the username or the issuer.
In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `<username>` instead of `<userid>`. You may have encountered `<userlayout>`, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`)
{{< hint warning >}}
Make no mistake, the [OCS Provisioning API](https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html) uses `userid` while it actually is the username, because it is what you use to login.
docs/extensions/storage/namespaces.md
+3 -1
View File
@@ -12,7 +12,7 @@ In ownCloud 10 all paths are considered relative to the users home. The CS3 API
{{< svg src="extensions/storage/static/namespaces.drawio.svg" >}}
The different paths in the namespaces need to be translated while passing *references* from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a *storage space*.
The different paths in the namespaces need to be translated while passing [*references*]({{< ref "./terminology.md#references" >}}) from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The following table lists the various namespaces, paths and id based references:
| oc10 namespace | CS3 global namespace | storage provider | reference | content |
|--------------------------------------------------|----------------------------------------|---------|-------------------|-----------------|
@@ -29,3 +29,5 @@ oCIS currently is configured to jail users into the CS3 `/home` namespace in the
{{< hint warning >}}
In the global CS3 namespaces we plan to move `/home/Shares`, which currently lists all mounted shares of the currently logged in user to a dedicated `/shares` namespace. See [Move shares folder out from home directory to a separate mount reva#1584](https://github.com/cs3org/reva/pull/1584).
{{< /hint >}}
TODO link frontend / webdav because it contains more details
docs/extensions/storage/terminology.md
+6 -6
View File
@@ -41,7 +41,7 @@ While all components are optional, only three cases are used:
| `<storage_space>!:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space |
| `<storage_space>!<root>:<relative_path>` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths |
`<storage_space>` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-space" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`.
`<storage_space>` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`.
A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path
@@ -88,10 +88,10 @@ by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*stora
**Proposed Change**
A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}})
by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}).
{{< /hint >}}
{{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}}
{{< hint warning >}}
By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva. Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space.
By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space.
{{< /hint >}}
## Storage Registries
@@ -109,9 +109,9 @@ A *storage registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) fo
It is used by the *gateway*
to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}})
that is currently serving a [*storage space*]({{< ref "#storage-space" >}}).
{{< /hint >}}
{{< svg src="extensions/storage/static/storageregistry-spaces.drawio.svg" >}}
{{< hint warning >}}
By making *storage registries* aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can query them for a listing of all [*storage spaces*]({{< ref "#storage-spaces" >}}) a user has access to. Including his home, received shares, project folders or group drives. See [a WIP PR for spaces in the oCIS repo (#1827)](https://github.com/owncloud/ocis/pull/1827) for more info.
{{< /hint >}}