Merge pull request #2318 from owncloud/cleanup-spaces-docs
[docs-only] separate proposed changes from terminology
This commit is contained in:
Michael Barz
+5
-7
@@ -10,18 +10,16 @@ geekdocFilePath: _index.md
|
||||
{{< figure class="floatright" src="/media/is.png" width="70%" height="auto" >}}
|
||||
|
||||
## ownCloud Infinite Scale
|
||||
|
||||
Welcome to oCIS, the modern file-sync and share platform, which is based on our knowledge and experience with the PHP based [ownCloud server](https://owncloud.com/#server).
|
||||
|
||||
### The idea of federated storage
|
||||
To creata a truly federated storage architecture oCIS breaks down the old ownCloud 10 user specific namespace, which is assembled on the server side, and makes the individual parts accessible to clients as storage spaces and storage space registries.
|
||||
|
||||
To create a truly federated storage architecture oCIS breaks down the old ownCloud 10 user specific namespace, which is assembled on the server side, and makes the individual parts accessible to clients as storage spaces and storage space registries.
|
||||
|
||||
The below diagram shows the core concepts that are the foundation for the new architecture:
|
||||
The below diagram shows the core conceps that are the foundation for the new architecture:
|
||||
- End user devices can fetch the list of *storage spaces* a user has access to, by querying one or multiple *storage space registries*. The list contains a unique endpoint for every *storage space*.
|
||||
- [*Storage space registries*]({{< ref "../extensions/storage/terminology#storage-space-registries" >}}) manage the list of storage spaces a user has access to. They may subscribe to *storage spaces* in order to receive notifications about changes on behalf of an end users mobile or desktop client.
|
||||
- [*Storage spaces*]({{< ref "../extensions/storage/terminology#storage-spaces" >}}) represent a collection of files and folders. A users personal files are a *storage space*, a group or project drive is a *storage space*, and even incoming shares are treated and implemented as *storage spaces*. Each with properties like owners, permissions, quota and type.
|
||||
- [*Storage providers*]({{< ref "../extensions/storage/terminology#storage-providers" >}}) can hold multiple *storage spaces*. At an oCIS instance, there might be a dedicated *storage provider* responsible for users personal storage spaces. There might be multiple, sharing the load or there might be just one, hosting all types of *storage spaces*.
|
||||
- [*Storage spaces*]({{< ref "../extensions/storage/terminology#storage-spaces" >}}) represent a collection of files and folders. A users personal files are contained in a *storage space*, a group or project drive is a *storage space*, and even incoming shares are treated and implemented as *storage spaces*. Each with properties like owners, permissions, quota and type.
|
||||
- [*Storage providers*]({{< ref "../extensions/storage/terminology#storage-providers" >}}) can hold multiple *storage spaces*. At an oCIS instance, there might be a dedicated *storage provider* responsible for users personal storage spaces. There might be multiple, either to shard the load, provide different levels of redundancy or support custom workflows. Or there might be just one, hosting all types of *storage spaces*.
|
||||
|
||||
{{< svg src="ocis/static/idea.drawio.svg" >}}
|
||||
|
||||
@@ -42,7 +40,7 @@ The oCIS runtime allows us to dynamically manage services running in a single pr
|
||||
### oCIS extensions
|
||||
|
||||
Every oCIS extension uses [ocis-pkg](https://github.com/owncloud/ocis/tree/master/ocis-pkg), which implements the [go-micro](https://go-micro.dev/) interfaces for [servers](https://github.com/asim/go-micro/blob/v3.5.0/server/server.go#L17-L37) to register and [clients](https://github.com/asim/go-micro/blob/v3.5.0/client/client.go#L11-L23) to lookup nodes with a service [registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/registry.go).
|
||||
We are following the [12 Factor](https://12factor.net/) methodology with oCIS. The uniformity of services also allows us to use the same command, logging and configuration mechanism. Configurations are forwarded from the
|
||||
We are following the [12 Factor](https://12factor.net/) methodology with oCIS. The uniformity of services also allows us to use the same command, logging and configuration mechanism. Configurations are forwarded from the
|
||||
oCIS runtime to the individual extensions.
|
||||
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ is also responsible for redirecting requests on the OIDC discovery endpoints (e.
|
||||
|
||||
Keycloak add two containers: Keycloak itself and a PostgreSQL as database. Keycloak will be configured as oCIS' IDP instead of the internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}})
|
||||
|
||||
The other container is oCIS itself running all extensions in one container. In this example oCIS uses [oCIS storage driver]({{< ref "../../extensions/storage/storages#storage-drivers" >}})
|
||||
The other container is oCIS itself running all extensions in one container. In this example oCIS uses the [oCIS storage driver]({{< ref "../../extensions/storage/storagedrivers" >}})
|
||||
|
||||
## Server Deployment
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ geekdocFilePath: ocis_traefik.md
|
||||
|
||||
The docker stack consists of two containers. One of them is Traefik, a proxy which is terminating ssl and forwards the requests to oCIS in the internal docker network.
|
||||
|
||||
The other one is oCIS itself running all extensions in one container. In this example oCIS uses its internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}}) and the [oCIS storage driver]({{< ref "../../extensions/storage/storages#storage-drivers" >}})
|
||||
The other one is oCIS itself running all extensions in one container. In this example oCIS uses its internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}}) and the [oCIS storage driver]({{< ref "../../extensions/storage/storagedrivers" >}})
|
||||
|
||||
## Server Deployment
|
||||
|
||||
|
||||
+39
-37
@@ -25,7 +25,7 @@ Is the pre-migration stage when having a functional ownCloud 10 instance.
|
||||
<div class="editpage">
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -45,10 +45,10 @@ _TODO allow limiting the web ui switch to an 'early adopters' group_
|
||||
</div>
|
||||
|
||||
#### Validation
|
||||
Ensure switching back an forth between the classic ownCloud 10 web UI and ownCloud web works as at our https://demo.owncloud.com.
|
||||
Ensure switching back an forth between the classic ownCloud 10 web UI and ownCloud web works as at our https://demo.owncloud.com.
|
||||
|
||||
#### Rollback
|
||||
Should there be problems with ownCloud web at this point it can simply be removed from the menu and be undeployed.
|
||||
Should there be problems with ownCloud web at this point it can simply be removed from the menu and be undeployed.
|
||||
|
||||
#### Notes
|
||||
<div style="break-after: avoid"></div>
|
||||
@@ -56,10 +56,10 @@ The ownCloud 10 demo instance uses OAuth to obtain a token for ownCloud web and
|
||||
|
||||
<div class="editpage">
|
||||
|
||||
_TODO make oauth2 in oc10 trust the new web ui, based on `redirect_uri` and CSRF so no explicit consent is needed_
|
||||
_TODO make oauth2 in oc10 trust the new web ui, based on `redirect_uri` and CSRF so no explicit consent is needed?_
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -68,15 +68,16 @@ _Feel free to add your question as a PR to this document using the link at the t
|
||||
### Stage 2: introduce OpenID Connect
|
||||
|
||||
Basic auth requires us to properly store and manage user credentials. Something we would rather like to delegate to a tool specifically built for that task.
|
||||
While SAML and Shibboleth are protocols that solve that problem, they are limited to web clients. Desktop and mobile clients were an afterthought and keep running into timeouts. For these reasons, we decided to move to [OpenID Connect as our primary authentication protocol](https://owncloud.com/news/openid-connect-oidc-app/).
|
||||
While SAML and Shibboleth are protocols that solve that problem, they are limited to web clients. Desktop and mobile clients were an afterthought and keep running into timeouts. For these reasons, we decided to move to [OpenID Connect as our primary authentication protocol](https://owncloud.com/news/openid-connect-oidc-app/).
|
||||
|
||||
<div class="editpage">
|
||||
|
||||
_TODO @butonic add ADR for OpenID Connect_
|
||||
_TODO @butonic add ADR for OpenID Connect and flesh out pros and cons of the above_
|
||||
|
||||
</div>
|
||||
|
||||
#### User impact
|
||||
|
||||
When introducing OpenID Connect, the clients will detect the new authentication scheme when their current way of authenticating returns an error. Users will then have to
|
||||
reauthorize at the OpenID Connect IdP, which again, may be configured to skip the consent step for trusted clients.
|
||||
|
||||
@@ -111,19 +112,19 @@ While OpenID Connect providers will send an `iss` and `sub` claim that relying p
|
||||
<div class="editpage">
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
<div style="break-after: page"></div>
|
||||
|
||||
### Stage 3: introduce oCIS interally
|
||||
### Stage 3: introduce oCIS internally
|
||||
|
||||
Before letting oCIS handle end user requests we will first make it available in the internal network. By subsequently adding services we can add functionality and verify the services work as intended.
|
||||
|
||||
Start oCIS backend and make read only tests on existing data using the `owncloudsql` storage driver which will read (and write)
|
||||
- blobs from the same data directory layout as in ownCloud 10
|
||||
- metadata from the ownCloud 10 database:
|
||||
- blobs from the same datadirectory layout as in ownCloud 10
|
||||
- metadata from the ownCloud 10 database:
|
||||
The oCIS share manager will read share information from the ownCloud database using an `owncloud` driver as well.
|
||||
|
||||
<div class="editpage">
|
||||
@@ -143,7 +144,7 @@ We are going to run and explore a series of services that will together handle t
|
||||
|
||||
|
||||
##### Storage provider for file metadata
|
||||
1. Deploy OCIS storage provider with the `owncloudsql` driver.
|
||||
1. Deploy OCIS storage provider with owncloudsql driver.
|
||||
2. Set `read_only: true` in the storage provider config. <div class="editpage">_TODO @butonic add read only flag to storage drivers_</div>
|
||||
3. Use cli tool to list files using the CS3 api
|
||||
|
||||
@@ -172,7 +173,7 @@ Enable spaces API in oc10:
|
||||
|
||||
{{< hint warning >}}
|
||||
**Alternative 2**
|
||||
An additional `uuid` property used only to detect moves. A lookup by uuid is not necessary for this. The `/dav/meta` endpoint would still take the fileid. Clients would use the `uuid` to detect moves and set up new sync pairs when migrating to a global namespace.
|
||||
An additional `uuid` property used only to detect moves. A lookup by uuid is not necessary for this. The `/dav/meta` endpoint would still take the fileid. Clients would use the `uuid` to detect moves and set up new sync pairs when migrating to a global namespace.
|
||||
### Stage-3.1
|
||||
Generate a `uuid` for every file as a file property. Clients can submit a `uuid` when creating files. The server will create a `uuid` if the client did not provide one.
|
||||
|
||||
@@ -181,7 +182,7 @@ Roll out new clients that understand the spaces API and know how to convert loca
|
||||
One pair for `/webdav/home` or `/dav/files/<username>/home` and another pair for every accepted share. The shares will be accessible at `/webdav/shares/` when the server side enables the spaces API. Files can be identified using the `uuid` and moved to the correct sync pair.
|
||||
|
||||
### Stage-4.1
|
||||
When reading the files from oCIS return the same `uuid`. It can be migrated to an extended attribute or it can be read from oc10. If users change it the client will not be able to detect a move and maybe other weird stuff happens. *What if the uuid gets lost on the server side due to a partial restore?*
|
||||
When reading the files from oCIS return the same `uuid`. It can be migrated to an extended attribute or it can be read from oc10. If users change it the client will not be able to detect a move and maybe other weird stuff happens. *What if the uuid gets lost on the server side due to a partial restore?*
|
||||
|
||||
{{< /hint >}}
|
||||
</div>
|
||||
@@ -223,7 +224,7 @@ Multiple ownCloud instances can be merged into one oCIS instance. The file ids w
|
||||
<div class="editpage">
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -261,7 +262,7 @@ With write access it becomes possible to manipulate existing files and shares.
|
||||
<div class="editpage">
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -273,10 +274,10 @@ In the previous stages oCIS was only accessible for administrators with access t
|
||||
#### User impact
|
||||
The IP address of the ownCloud host changes. There is no change for the file sync and share functionality when requests are handled by the oCIS codebase as it uses the same database and storage system as owncloud 10.
|
||||
|
||||
#### Steps and verifications
|
||||
#### Steps and verifications
|
||||
|
||||
##### Deploy oCIS proxy
|
||||
1. Deploy the `ocis proxy`
|
||||
1. Deploy the `ocis proxy`
|
||||
2. Verify the requests are routed based on the ownCloud 10 routing policy `oc10` by default
|
||||
|
||||
##### Test user based routing
|
||||
@@ -300,7 +301,7 @@ The proxy is stateless, multiple instances can be deployed as needed.
|
||||
<div class="editpage">
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -340,12 +341,13 @@ _TODO @butonic we need a canary app that allows users to decide for themselves w
|
||||
<div style="break-after: page"></div>
|
||||
|
||||
#### Notes
|
||||
Running the two systems in parallel requires additional maintenance effort. Try to keep the duration of this stage short. Until now, we only added services and made the system more complex. oCIS aims to reduce the maintenance cost of an ownCloud instance. You will not get there if you keep both systems alive.
|
||||
Running the two systems in parallel stage
|
||||
Try to keep the duration of this stage short. Until now we only added services and made the system more complex. oCIS aims to reduce the maintenance cost of an ownCloud instance. You will not get there if you keep both systems alive.
|
||||
|
||||
<div class="editpage">
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -355,8 +357,8 @@ _Feel free to add your question as a PR to this document using the link at the t
|
||||
To encourage users to switch you can promote the workspaces feature that is built into oCIS. The ownCloud 10 storage backend can be used for existing users. New users and group or project spaces can be provided by storage providers that better suit the underlying storage system.
|
||||
|
||||
#### Steps
|
||||
First, the admin needs to
|
||||
- deploy a storage provider with the storage driver that best fits the underlying storage system and requirements.
|
||||
First, the admin needs to
|
||||
- deploy a storage provider with the storage driver that best fits the underlying storage system and requirements.
|
||||
- register the storage in the storage registry with a new storage id (we recommend a uuid).
|
||||
|
||||
Then a user with the necessary create storage space role can create a storage space and assign Managers.
|
||||
@@ -373,7 +375,7 @@ The new storage space should show up in the `/graph/drives` endpoint for the man
|
||||
#### Notes
|
||||
Depending on the requirements and acceptable tradeoffs, a database less deployment using the ocis or s3ng storage driver is possible. There is also a [cephfs driver](https://github.com/cs3org/reva/pull/1209) on the way, that directly works on the API level instead of POSIX.
|
||||
|
||||
### Stage-8: shut down ownCloud 10
|
||||
### Stage-8: shut down ownCloud 10
|
||||
Disable ownCloud 10 in the proxy, all requests are now handled by oCIS, shut down oc10 web servers and redis (or keep for calendar & contacts only? rip out files from oCIS?)
|
||||
|
||||
#### User impact
|
||||
@@ -385,7 +387,7 @@ _TODO @butonic recommend alternatives_
|
||||
|
||||
</div>
|
||||
|
||||
#### Steps
|
||||
#### Steps
|
||||
1. Shut down the apache servers that are running the ownCloud 10 PHP code.
|
||||
2. DO NOT SHUT DOWN THE DATABASE, YET!
|
||||
|
||||
@@ -402,7 +404,7 @@ The database needs to remain online until the storage layer and share metadata h
|
||||
<div class="editpage">
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -414,7 +416,7 @@ To get rid of the database we will move the metadata from the old ownCloud 10 da
|
||||
#### User impact
|
||||
Noticeable performance improvements because we effectively shard the storage logic and persistence layer.
|
||||
|
||||
#### Steps
|
||||
#### Steps
|
||||
1. User by user storage migration from `owncloud` or `ownclouds3` driver to `ocis`/`s3ng`/`cephfs`... currently this means copying the metadata from one storage provider to another using the cs3 api.
|
||||
2. Change the responsible storage provider for a storage space (e.g. a user home, a group or project space are a workspace) in the storage registry.
|
||||
|
||||
@@ -441,7 +443,7 @@ The storage space migration will become a seamless feature in the future that al
|
||||
<div class="editpage">
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -457,7 +459,7 @@ Depending on chosen the share manager provider some sharing requests should be f
|
||||
- For non HA scenarios they can be served from memory, backed by a simple json file.
|
||||
- TODO: implement share manager with redis / nats / ... key value store backend: use the micro store interface please ...
|
||||
|
||||
#### Steps
|
||||
#### Steps
|
||||
1. Start new share manager
|
||||
2. Migrate metadata using the CS3 API (copy from old to new)
|
||||
3. Shut down old share manager
|
||||
@@ -482,7 +484,7 @@ _TODO let the gateway write updates to multiple share managers ... or rely on th
|
||||
</div>
|
||||
|
||||
#### Rollback
|
||||
To switch the share manager to the database one revert routing users to the new share manager. If you already shut down the old share manager start it again. Use the tiered/chained share manager provider in reverse configuration (new share provider as read only, old as write) and migrate the shares again. You can alse restore a database backup if needed.
|
||||
To switch the share manager to the database one revert routing users to the new share manager. If you already shut down the old share manager start it again. Use the tiered/chained share manager provider in reverse configuration (new share provider as read only, old as write) and migrate the shares again. You can also restore a database backup if needed.
|
||||
|
||||
<div class="editpage">
|
||||
|
||||
@@ -490,7 +492,7 @@ To switch the share manager to the database one revert routing users to the new
|
||||
Profit! Well, on the one hand you do not need to maintain a clustered database setup and can rely on the storage system. On the other hand you are now in microservice wonderland and will have to relearn how to identify bottlenecks and scale oCIS accordingly. The good thing is that tools like jaeger and prometheus have evolved and will help you understand what is going on. But this is a different topic. See you on the other side!
|
||||
|
||||
#### FAQ
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
_Feel free to add your question as a PR to this document using the link at the top of this page!_
|
||||
|
||||
</div>
|
||||
|
||||
@@ -503,7 +505,7 @@ The fundamental difference between ownCloud 10 and oCIS is that the file metadat
|
||||
|
||||
## Data that will be migrated
|
||||
|
||||
Currently, oCIS focuses on file sync and share use cases.
|
||||
Currently, oCIS focuses on file sync and share use cases.
|
||||
|
||||
### Blob data
|
||||
|
||||
@@ -533,7 +535,7 @@ data
|
||||
│ │ │ └── Notes.md.v1540305560
|
||||
│ │ └── ownCloud Manual.pdf.v1396628249
|
||||
│ ├── thumbnails
|
||||
│ │ └── 123
|
||||
│ │ └── 123
|
||||
│ │ │ ├── 2048-1536-max.png
|
||||
│ │ │ └── 32-32.png // the file id, eg. of /Photos/Portugal.jpg
|
||||
│ └── uploads
|
||||
@@ -553,9 +555,9 @@ The *data directory* may also contain subfolders for ownCloud 10 applications li
|
||||
|
||||
When an object storage is used as the primary storage all file blobs are stored by their file id and a prefix, eg.: `urn:oid:<fileid>`.
|
||||
|
||||
The three types of blobs we need to migrate are stored in
|
||||
The three types of blobs we need to migrate are stored in
|
||||
- `files` for file blobs, the current file content,
|
||||
- `files_trashbin` for trashed files (and their versions) and
|
||||
- `files_trashbin` for trashed files (and their versions) and
|
||||
- `files_versions` for file blobs of older versions.
|
||||
|
||||
<div style="break-after: page"></div>
|
||||
@@ -586,7 +588,7 @@ The `filecache` table itself has more metadata:
|
||||
| `checksum` | varchar(255) | YES | | NULL | | *same as blob checksum* | SHOULD become the checksum in the storage provider. eos calculates it itself, `ocis` driver stores it in extended attributes |
|
||||
|
||||
|
||||
> Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. otherwise either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid
|
||||
> Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. otherwise either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid
|
||||
|
||||
<div style="break-after: page"></div>
|
||||
|
||||
@@ -730,7 +732,7 @@ _TODO clarify if metadata from ldap & user_shibboleth needs to be migrated_
|
||||
|
||||
</div>
|
||||
|
||||
The `dn` -> *owncloud internal username* mapping that currently lives in the `oc_ldap_user_mapping` table needs to move into a dedicated ownclouduuid attribute in the LDAP server. The idp should send it as a claim so the proxy does not have to look up the user using LDAP again. The username cannot be changed in ownCloud 10 and the oCIS provisioning API will not allow changing it as well. When we introduce the graph api we may allow changing usernames when all clients have moved to that api.
|
||||
The `dn` -> *owncloud internal username* mapping that currently lives in the `oc_ldap_user_mapping` table needs to move into a dedicated `ownclouduuid` attribute in the LDAP server. The idp should send it as a claim so the proxy does not have to look up the user using LDAP again. The username cannot be changed in ownCloud 10 and the oCIS provisioning API will not allow changing it as well. When we introduce the graph api we may allow changing usernames when all clients have moved to that api.
|
||||
|
||||
The problem is that the username in owncloud 10 and in oCIS also need to be the same, which might not be the case when the ldap mapping used a different column. In that case we should add another owncloudusername attribute to the ldap server.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user