From 205b3a72bd340b8edc6a60316146313ef5d7c1d8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Wed, 28 Apr 2021 14:31:01 +0000 Subject: [PATCH] migration sideeffects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/extensions/storage/architecture.md | 24 ++++++++--------- docs/extensions/storage/namespaces.md | 4 ++- docs/extensions/storage/terminology.md | 12 ++++----- docs/ocis/migration.md | 34 +++++++++++++++++++++++++ 4 files changed, 55 insertions(+), 19 deletions(-) diff --git a/docs/extensions/storage/architecture.md b/docs/extensions/storage/architecture.md index 9cdb4d3aa..ab8683234 100644 --- a/docs/extensions/storage/architecture.md +++ b/docs/extensions/storage/architecture.md @@ -28,7 +28,7 @@ The ocdav service not only handles all WebDAV requests under `(remote.php/)(web) | `(remote.php/)webdav` | ocdav | storageprovider | `/home` | the old webdav endpoint | | | `(remote.php/)dav/files/` | ocdav | storageprovider | `/home` | the new webdav endpoint | | | `(remote.php/)dav/meta//v` | ocdav | storageprovider | id based | versions | | -| `(remote.php/)dav/trash-bin/` | ocdav | recycle | - | trash | should aggregate the trash of storage spaces the user has access to | +| `(remote.php/)dav/trash-bin/` | ocdav | recycle | - | trash | should aggregate the trash of [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) the user has access to | | `(remote.php/)dav/public-files/` | ocdav | storageprovider | `/public/` | public links | | | `(remote.php/)dav/avatars/` | ocdav | - | - | avatars, hardcoded | look up from user provider and cache | | *CernBox setup:* ||||| @@ -38,32 +38,32 @@ The ocdav service not only handles all WebDAV requests under `(remote.php/)(web) | `(remote.php/)webdav/users` | ocdav | storageprovider | `/users` | | | | `(remote.php/)dav/files/` | ocdav | storageprovider | `/users/` | | | | *Spaces concept also needs a new endpoint:* ||||| -| `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to storage spaces or a global endpoint | allow accessing spaces, listing is done by the graph api | +| `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) or a global endpoint | allow accessing spaces, listing is done by the graph api | -The correct endpoint for a users home storage space in oc10 is `remote.php/dav/files/`. In oc10 All requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home. +The correct endpoint for a users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) in oc10 is `remote.php/dav/files/`. In oc10 All requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home. -In oc10 we originally had `remote.php/webdav` which would render the current users home storage space. The early versions (pre OC7) would jail all received shares into a `remote.php/webdav/shares` subfolder. The semantics for syncing such a folder are [not trivially predictable](https://github.com/owncloud/core/issues/5349), which is why we made shares [freely mountable](https://github.com/owncloud/core/pull/8026) anywhere in the users home. +In oc10 we originally had `remote.php/webdav` which would render the current users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The early versions (pre OC7) would jail all received shares into a `remote.php/webdav/shares` subfolder. The semantics for syncing such a folder are [not trivially predictable](https://github.com/owncloud/core/issues/5349), which is why we made shares [freely mountable](https://github.com/owncloud/core/pull/8026) anywhere in the users home. The current reva implementation jails shares into a `remote.php/webdav/Shares` folder for performance reasons. Obviously, this brings back the [special semantics for syncing](https://github.com/owncloud/product/issues/7). In the future we will follow [a different solution](https://github.com/owncloud/product/issues/302) and jail the received shares into a dedicated `/shares` space, on the same level as `/home` and `/spaces`. We will add a dedicated [API to list all *storage spaces*](https://github.com/owncloud/ocis/pull/1827) a user has access to and where they are mounted in the users *namespace*. {{< hint warning >}} Existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy like CernBox did. For legacy clients the `remote.php/webdav` endpoint will no longer list the users home directly, but instead present the different types of storage spaces: -- `remote.php/webdav/home`: the users home is pushed down into a new `home` *storage space* -- `remote.php/webdav/shares`: all mounted shares will be moved to a new `shares` *storage space* -- `remote.php/webdav/spaces`: other *storage spaces* the user has access to, e.g. group or project drives +- `remote.php/webdav/home`: the users home is pushed down into a new `home` [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) +- `remote.php/webdav/shares`: all mounted shares will be moved to a new `shares` [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) +- `remote.php/webdav/spaces`: other [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) the user has access to, e.g. group or project drives {{< /hint >}} {{< hint warning >}} -An alternative would be to introduce a new `remote.php/dav/spaces` or `remote.php/dav/global` endpoint. However, `remote.php/dav` properly follows the WebDAV RFCs strictly. To ensure that all resources under that namespace are scoped to the user the URL would have to include the principal like `remote.php/dav/spaces/`, a precondition for e.g. WebDAV [RFC5397](https://tools.ietf.org/html/rfc5397). For a history lesson start at [Replace WebDAV with REST +An alternative would be to introduce a new `remote.php/dav/spaces` or `remote.php/dav/global` endpoint. However, `remote.php/dav` properly follows the WebDAV RFCs strictly. To ensure that all resources under that [*namespace*]({{< ref "./terminology.md#namespaces" >}}) are scoped to the user the URL would have to include the principal like `remote.php/dav/spaces/`, a precondition for e.g. WebDAV [RFC5397](https://tools.ietf.org/html/rfc5397). For a history lesson start at [Replace WebDAV with REST owncloud/core#12504](https://github.com/owncloud/core/issues/12504#issuecomment-65218491) which spawned [Add extra layer in DAV to accomodate for other services like versions, trashbin, etc owncloud/core#12543](https://github.com/owncloud/core/issues/12543) {{< /hint >}} ### Sharing -The [OCS Share API](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html) endpoint `/ocs/v1.php/apps/files_sharing/api/v1/shares` returns shares, which have their own share id and reference files using a path relative to the users home. They API also lists the numeric storage id as well as the string type `storage_id` (which is confusing ... but yeah) which would allow constructing combined references with a storage spacle id and a path relative to the root of that storage space. The web UI however assumes that it can take the path from the `file_target` and append it to the users home to access it. +The [OCS Share API](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html) endpoint `/ocs/v1.php/apps/files_sharing/api/v1/shares` returns shares, which have their own share id and reference files using a path relative to the users home. They API also lists the numeric storage id as well as the string type `storage_id` (which is confusing ... but yeah) which would allow constructing combined references with a `storage space id` and a `path` relative to the root of that [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The web UI however assumes that it can take the path from the `file_target` and append it to the users home to access it. {{< hint >}} The API [already returns the storage id](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html#example-request-response-payloads-4) (and numeric id) in addition to the file id: @@ -88,13 +88,13 @@ The OCS service makes a stat request to the storage provider to get a [ResourceI {{< hint >}} -The user and public share provider implementations identify the file using the `ResourceId`. The `ResourceInfo` is passed so the share provider can also store who the owner of the resource is. The *path* is not part of the other API calls, e.g. when listing shares. -The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares. +The user and public share provider implementations identify the file using the [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The [`ResourceInfo`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo) is passed so the share provider can also store who the owner of the resource is. The *path* is not part of the other API calls, e.g. when listing shares. +The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.* {{< /hint >}} ### User and Group provisioning -In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configuted to build several path segments by filling in user properties, e.g. the first two characters of the username or the issuer. +In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) {{< hint warning >}} Make no mistake, the [OCS Provisioning API](https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html) uses `userid` while it actually is the username, because it is what you use to login. diff --git a/docs/extensions/storage/namespaces.md b/docs/extensions/storage/namespaces.md index fce997a50..e20357906 100644 --- a/docs/extensions/storage/namespaces.md +++ b/docs/extensions/storage/namespaces.md @@ -12,7 +12,7 @@ In ownCloud 10 all paths are considered relative to the users home. The CS3 API {{< svg src="extensions/storage/static/namespaces.drawio.svg" >}} -The different paths in the namespaces need to be translated while passing *references* from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a *storage space*. +The different paths in the namespaces need to be translated while passing [*references*]({{< ref "./terminology.md#references" >}}) from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The following table lists the various namespaces, paths and id based references: | oc10 namespace | CS3 global namespace | storage provider | reference | content | |--------------------------------------------------|----------------------------------------|---------|-------------------|-----------------| @@ -29,3 +29,5 @@ oCIS currently is configured to jail users into the CS3 `/home` namespace in the {{< hint warning >}} In the global CS3 namespaces we plan to move `/home/Shares`, which currently lists all mounted shares of the currently logged in user to a dedicated `/shares` namespace. See [Move shares folder out from home directory to a separate mount reva#1584](https://github.com/cs3org/reva/pull/1584). {{< /hint >}} + +TODO link frontend / webdav because it contains more details \ No newline at end of file diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md index 3aadf131d..bd1d4c0bf 100644 --- a/docs/extensions/storage/terminology.md +++ b/docs/extensions/storage/terminology.md @@ -41,7 +41,7 @@ While all components are optional, only three cases are used: | `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space | | `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths | -`` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-space" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`. +`` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`. A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path @@ -88,10 +88,10 @@ by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*stora **Proposed Change** A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}}) by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). -{{< /hint >}} + {{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}} -{{< hint warning >}} -By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva. Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space. + +By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space. {{< /hint >}} ## Storage Registries @@ -109,9 +109,9 @@ A *storage registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) fo It is used by the *gateway* to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) that is currently serving a [*storage space*]({{< ref "#storage-space" >}}). -{{< /hint >}} + {{< svg src="extensions/storage/static/storageregistry-spaces.drawio.svg" >}} -{{< hint warning >}} + By making *storage registries* aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can query them for a listing of all [*storage spaces*]({{< ref "#storage-spaces" >}}) a user has access to. Including his home, received shares, project folders or group drives. See [a WIP PR for spaces in the oCIS repo (#1827)](https://github.com/owncloud/ocis/pull/1827) for more info. {{< /hint >}} diff --git a/docs/ocis/migration.md b/docs/ocis/migration.md index 98d6ad201..407cc2ad1 100644 --- a/docs/ocis/migration.md +++ b/docs/ocis/migration.md @@ -26,6 +26,40 @@ Introduce OpenID Connect to server and Clients ### Stage-2 Install and introduce ownCloud Web and let users test it voluntarily. + +{{< hint warning >}} +**Alternative 1** +Add a routable prefix to fileids in oc10, and replicate the prefix in ocis. +### Stage-2.1 +Let oc10 render file ids with prefixes: `$!`. This will allow clients to handle moved files. + +### Stage-2.2 +Roll out new clients that understand the spaces API and know how to convert local sync pairs for legacy oc10 `/webdav` or `/dav/files/` home folders into multiple sync pairs. +One pair for `/webdav/home` or `/dav/files//home` and another pair for every accepted share. The shares will be accessible at `/webdav/shares/` when the server side enables the spaces API. +Files can be identified using `$!` and moved to the correct sync pair. + +### Stage-2.3 +Enable spaces API in oc10: +- New clients will get a response from the spaces API and can set up new sync pairs. +- Legacy clients will still poll `/webdav` or `/dav/files/` where they will see new subfolders instead of the users home. They will move down the users files into `/home` and shares into `/shares`. Custom sync pairs will no longer be available, causing the legacy client to leave local files in place. They can be picked up manually when installing a new client. + +{{< /hint >}} + +{{< hint warning >}} +**Alternative 2** +An additional `uuid` property used only to detect moves. A lookup by uuid is not necessary for this. The `/dav/meta` endpoint would still take the fileid. Clients would use the `uuid` to detect moves and set up new sync pairs when migrating to a global namespace. +### Stage-2.1 +Generate a `uuid` for every file as a file property. Clients can submit a `uuid` when creating files. The server will create a `uuid` if the client did not provide one. + +### Stage-2.2 +Roll out new clients that understand the spaces API and know how to convert local sync pairs for legacy oc10 `/webdav` or `/dav/files/` home folders into multiple sync pairs. +One pair for `/webdav/home` or `/dav/files//home` and another pair for every accepted share. The shares will be accessible at `/webdav/shares/` when the server side enables the spaces API. Files can be identified using the `uuid` and moved to the correct sync pair. + +### Stage-3.1 +When reading the files from ocis return the same `uuid`. It can be migrated to an extended attribute or it can be read from oc10. If users change it the client will not be able to detect a move and maybe other weird stuff happens. *What if the uuid gets lost on the server side due to a partial restore?* + +{{< /hint >}} + ### Stage-3 Start oCIS backend and make read only tests on existing data using the `owncloud` storage driver which will read (and write) - blobs from the same datadirectory layout as in ownCloud 10 and