sevenhill/QSfera
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial migration docs

Browse Source 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
This commit is contained in:
Jörn Friedrich Dreyer
2021-03-19 08:49:12 +00:00
parent ec0e962c7b
commit cd28ac3fbd
2 changed files with 221 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/0004-support-hot-migration.md
+61
View File
@@ -0,0 +1,61 @@
# 4. Support Hot Migration
* Status: proposed
* Deciders: @butonic, @micbar, @dragotin, @hodyroff, @pmaier1
* Date: 2021-03-16
Technical Story: \[description | ticket/issue URL\]
## Context and Problem Statement
Migration is one of the most important topics of the oCIS story. We need to provide a concept how to migrate from oC10 to oCIS.
## Decision Drivers
- Do not lose file blob or meta data.
- Existing shares like public links and federated shares must remain functional after the migration.
- Legacy clients need to keep working
## Considered Options
- Cold Migration
- Hot Migration
## Decision Outcome
Chosen option: "\[option 1\]", because \[justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)\].
### Positive Consequences
- \[e.g., improvement of quality attribute satisfaction, follow-up decisions required, …\]
- …
### Negative Consequences
- \[e.g., compromising quality attribute, follow-up decisions required, …\]
- …
## Pros and Cons of the Options
### Cold Migration
The migration happens while the service is offline. File metadata, blobs and share data is exported from ownCloud 10 and imported in oCIS.
- Good, because oCIS can be tested in a staging system without writing to the production system.
- Good, because file layout can be changed together with the architecture migration.
- Bad, because the export and import might require significant amounts of storage.
- Bad, because a rollback to the state before the migration might cause data loss of the changes that happend in between.
### Hot Migration
The migration happens in subsequent stages while the service is online.
- Good, because users can switch between the backends on the fly.
- Good, because users and admins can learn to trust the new system.
- Bad, because it is more complex and might drag on for a long time.
## Links
- [Clarify responsibilities of share providers and storage providers · Issue #1377 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/1377) because the share manager for ocis should store share information on the storage system. And [storageprovider should persist share creator · Issue #93 · cs3org/cs3apis (github.com)](https://github.com/cs3org/cs3apis/issues/93) finally: [eos: store share id in inherited xattr · Issue #543 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/543)
docs/ocis/migration.md
+160
View File
@@ -0,0 +1,160 @@
---
title: "Migration"
date: 2021-03-16T16:17:00+01:00
weight: 41
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis
geekdocFilePath: migration.md
---
## Migration
The migration happens in subsequent stages while the service is online.
### User Stories
As an admin I need to avoid downtime.
As an admin I want to migrate certain groups of users before others.
As a user I need a seamless migration and not loose data by any chance.
### Stage-0
Is the pre-migration stage having a functional ownCloud 10 instance
### Stage-1
Introduce OpenID Connect to server and Clients
### Stage-2
Introduce new web ui, let end users get used to it voluntarily
### Stage-3
start oCIS backend and make read only tests on existing data using owncloud storage driver and database backed share manager
### Stage-4
test updating data using oCIS backend
### Stage-5
introduce reverse proxy and switch over early adoptors, let admins gain trust in the new backend
### Stage-6
voluntary transition period and subsequent hard deadline for all users
### Stage-7
disable oc10 in the proxy, all requests are now handled by oCIS, shut down oc10 web servers and redis (or keep for calendar & contacts only? rip out files from oCIS?)
### Stage-8
user by user storage migration from owncloud driver to `ocis`/`s3ng`/`cephfs`...
### Stage-9
migrate share data to &lt;yet to determine&gt; share manager backend and shut down owncloud database
### Stage-10
profit! (db for file metadata no longer necessary, less maintenance effort)
## Architectural differences
The fundamental difference between ownCloud 10 and oCIS is that the file metadata is moved from the database in the `oc_filecache` table (which is misnamed, as it actually is an index) to the storage provider who can place metadata as close to the underlying storage system as possible. In effect the file metadata is sharded over multiple specialized services.
## Data that will be migrated
Currently, oCIS focuses on file sync and share use cases. In ownCloud 10 the files are laid out on disk in the *data directory* using the following layout:
```
data
├── einstein
│ ├── cache
│ ├── files
│ │ ├── Photos
│ │ │ └── Portugal.jpg
│ │ ├── Projects
│ │ │ └── Notes.md
│ │ └── ownCloud Manual.pdf
│ ├── files_external
│ ├── files_trashbin
│ │ ├── files
│ │ │ ├── Documents.d1564687985
│ │ │ ├── TODO.txt.d1565721976
│ │ │ └── welcome.txt.d1564775872
│ │ └── versions
│ │ │ ├── TODO.txt.v1564605543.d1565721976
│ │ │ └── TODO.txt.v1564775936.d1565721976
│ ├── files_versions
│ │ ├── Projects
│ │ │ ├── Notes.md.v1496912691
│ │ │ └── Notes.md.v1540305560
│ │ └── ownCloud Manual.pdf.v1396628249
│ ├── thumbnails
│ │ └── 123
│ │ │ ├── 2048-1536-max.png
│ │ │ └── 32-32.png // the file id, eg. of /Photos/Portugal.jpg
│ └── uploads
├── einstein
│ ├── files
│ ├── files_trash
│ └── files_versions
│ …
├── marie
…
```
The *data directory* may also contain subfolders for owncloud 10 applications like `avatars`, `gallery`, `files_external` and `cache`.
When an objectstorage is used as the primary storage all file blobs are stored by their file id and a prefix, eg.: `urn:oid:<fileid>`.
In both cases the file metadata, including a full replication of the file tree, is stored in the `oc_filecache` table of an ownCloud 10 database. The primary key of a row is the file id. It is used to attach additional metadata like shares, tags, favorites or arbitrary file properties.
The `filecache` table itself has more metadata:
| Field | Type | Null | Key | Default | Extra | Comment | Migration |
|--------------------|---------------|------|-----|---------|----------------|----------------|----------------|
| `fileid` | bigint(20) | NO | PRI | NULL | auto_increment | | MUST become the oCIS `opaqueid` of a file reference. `ocis` driver stores it in extendet attributes and can use numbers as node ids on disk. for eos see note below table |
| `storage` | int(11) | NO | MUL | 0 | | *the filecache holds metadata for multiple storages* | corresponds to an oCIS *storage space* |
| `path` | varchar(4000) | YES | | NULL | | *the path relative to the storages root* | MUST become the `path` relative to the storage root. `files` prefix needs to be trimmed. |
| `path_hash` | varchar(32) | NO | | | | *mysql once had problems indexing long paths, so we stored a hash for lookup by path. | - |
| `parent` | bigint(20) | NO | MUL | 0 | | *used to implement the hierarchy and listing children of a folder by id. redundant with `path`* | - |
| `name` | varchar(250) | YES | | NULL | | *basename of `path`* | - |
| `mimetype` | int(11) | NO | | 0 | | *joined with the `oc_mimetypes` table. only relevant for objectstorage deployments* | can be determined from blob / file extension |
| `mimepart` | int(11) | NO | | 0 | | *"* | can be determined from blob / file extension |
| `size` | bigint(20) | NO | | 0 | | *same as blob size unless encryption is used* | MAY become size, can be determined from blob |
| `mtime` | bigint(20) | NO | | 0 | | *same as blob mtime* | for files MAY become mtime (can be determined from blob as well), for directories MUST become tmtime |
| `encrypted` | int(11) | NO | | 0 | | *encrypted flag* | oCIS currently does not support encryption |
| `etag` | varchar(40) | YES | | NULL | | *used to propagate changes in a tree* | MUST be migrated (or calculated in the same way) to prevent clients from syncing unnecessarily |
| `unencrypted_size` | bigint(20) | NO | | 0 | | *same as blob size* | oCIS currently does not support encryption |
| `storage_mtime` | bigint(20) | NO | | 0 | | *used to detect external storage changes* | oCIS delegates that to the storage providers and drivers |
| `permissions` | int(11) | YES | | 0 | | *used as the basis for permissions. synced from disk when running a file scan. * | oCIS delegates that to the storage providers and drivers |
| `checksum` | varchar(255) | YES | | NULL | | *same as blob checksum* | SHOULD become the checksum in the storage provider. eos calculates it itself, `ocis` driver stores it in extendetd attributes |
> Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. otherwise either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid
### File blob data
## Goal
This ticket is about to decide which data of a typical oC10 installation can and will be migrated to oCIS.
Examples:
- User files
- Trash
- Versions
- Users, groups and permissions
- Public links
- Private shares with users and groups
- Federated shares
- Guest shares
- Activities
## User Stories
- As an admin I need a time efficient way of migration.
- As an admin I need to be able to explain which data was migrated and which not, to which consequences.
- As a user I want a seamless migration of my data.
- As a user I do not want to loose work that I put into the system, ie. by setting up shares.
- As a user I want to understand which data can not be migrated.
## Acceptance Criteria
- ADR that describes which data from oC10 will be migrated and which not.
- If data changes it's meaning or used differently (ie. Groups vs Roles) that is explained and the consequences are clear.
- It is explained which data is not migrated.