ocis command on a working state
This commit is contained in:
A.Unger
@@ -1,30 +0,0 @@
|
||||
---
|
||||
title: "Infinite Scale"
|
||||
date: 2020-02-27T20:35:00+01:00
|
||||
weight: -10
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: _index.md
|
||||
---
|
||||
|
||||
This tool provides a single entrypoint for the whole ownCloud Infinite Scale stack.
|
||||
|
||||
{{< mermaid class="text-center">}}
|
||||
graph TD
|
||||
ocis-proxy -->
|
||||
ocis-konnectd & ocis-phoenix & ocis-thumbnails & ocis-ocs & ocis-webdav
|
||||
|
||||
ocis-phoenix --> ocis-reva-fronted
|
||||
ocis-reva-fronted --> ocis-reva-gateway
|
||||
ocis-konnectd --> ocis-glauth
|
||||
|
||||
|
||||
ocis-reva-gateway --> ocis-reva-users
|
||||
ocis-reva-gateway --> ocis-reva-authbasic
|
||||
ocis-reva-gateway --> ocis-reva-auth-bearer
|
||||
|
||||
ocis-reva-gateway --> ocis-reva-sharing
|
||||
ocis-reva-gateway --> ocis-reva-storage-home-*
|
||||
ocis-reva-storage-home-* --> ocis-reva-storage-home-*-data
|
||||
ocis-reva-sharing --> redis
|
||||
{{< /mermaid >}}
|
||||
@@ -1,104 +0,0 @@
|
||||
---
|
||||
title: "Basic Remote Setup"
|
||||
date: 2020-02-27T20:35:00+01:00
|
||||
weight: 16
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: basic-remote-setup.md
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
Out of the box the ocis single binary and the `owncloud/ocis` docker image are configured to run on localhost for quick testing and development.
|
||||
|
||||
If you need to access ocis on a VM or a remote machine e.g. when testing a mobile client you need to configure ocis to run on a different host.
|
||||
|
||||
## Use the binary
|
||||
|
||||
If you start the ocis fullstack for the first time with `./bin/ocis server` it will generate a file `identifier-registration.yml` in the config folder relative to its location. This file is used to configure the clients for the built-in Identity Provider.
|
||||
|
||||
{{< hint warning >}}
|
||||
**Outdated version**\
|
||||
The `identifier-registration.yml` file will only be generated if there is no such file in place. You could miss updates on this file. Run `make clean` to delete the file and keep the development environment tidy otherwise as well.
|
||||
{{< /hint >}}
|
||||
|
||||
### Add your hostname to the idp config
|
||||
|
||||
Let us assume `your-host` is your remote domain name or IP adress. Add your host to the `identifier-registration.yml` like this:
|
||||
|
||||
```yaml {linenos=table,hl_lines=["15-17",21]}
|
||||
# OpenID Connect client registry.
|
||||
clients:
|
||||
- id: phoenix
|
||||
name: ownCloud web app
|
||||
application_type: web
|
||||
insecure: yes
|
||||
trusted: yes
|
||||
redirect_uris:
|
||||
- http://localhost:9100/
|
||||
- http://localhost:9100/oidc-callback.html
|
||||
- http://localhost:9100/oidc-silent-redirect.html
|
||||
- https://localhost:9200/
|
||||
- https://localhost:9200/oidc-callback.html
|
||||
- https://localhost:9200/oidc-silent-redirect.html
|
||||
- https://your-server:9200/
|
||||
- https://your-server:9200/oidc-callback.html
|
||||
- https://your-server:9200/oidc-silent-redirect.html
|
||||
origins:
|
||||
- http://localhost:9100
|
||||
- https://localhost:9200
|
||||
- https://your-server:9200
|
||||
```
|
||||
|
||||
In this example we do not change the default port (`9200`). But this could be changed to another port.
|
||||
|
||||
### Start the ocis fullstack server
|
||||
|
||||
You need to configure `your-host` in some services to provide the needed public resources. When using the owncloud storage driver (which is the default) oCIS currently needs a running Redis Server reachable locally on the machine at the default port (`localhost:6379`). You can change this using the following option `REVA_STORAGE_OWNCLOUD_REDIS_ADDR=some-host:6379`.
|
||||
|
||||
```bash
|
||||
PROXY_HTTP_ADDR=0.0.0.0:9200 \
|
||||
KONNECTD_ISS=https://your-server:9200 \
|
||||
REVA_OIDC_ISSUER=https://your-server:9200 \
|
||||
PHOENIX_OIDC_AUTHORITY=https://your-server:9200 \
|
||||
PHOENIX_WEB_CONFIG_SERVER=https://your-server:9200 \
|
||||
PHOENIX_OIDC_METADATA_URL=https://your-server:9200/.well-known/openid-configuration \
|
||||
REVA_DATAGATEWAY_URL=https://your-server:9200/data \
|
||||
REVA_FRONTEND_URL=https://your-server:9200 \
|
||||
PROXY_TRANSPORT_TLS_KEY=./certs/your-host.key \
|
||||
PROXY_TRANSPORT_TLS_CERT=./certs/your-host.crt \
|
||||
KONNECTD_TLS=0 \
|
||||
./bin/ocis server
|
||||
```
|
||||
|
||||
For more configuration options check the configuration secion in [ocis](https://owncloud.github.io/ocis/configuration/) and every ocis extension.
|
||||
|
||||
{{< hint info >}}
|
||||
**TLS Certificate**\
|
||||
In this example, we are replacing the default self signed cert with a CA signed one to avoid the certificate warning when accessing the login page.
|
||||
{{< /hint >}}
|
||||
|
||||
## Use Docker Compose
|
||||
|
||||
We are using our [docker compose playground](https://github.com/owncloud-docker/compose-playground) as a repository to share snippets that make our test setups easier and more aligned.
|
||||
|
||||
You can start oCIS with docker very easily on a different host using this snippet.
|
||||
|
||||
Let us assume your local IP is `192.168.103.195`
|
||||
|
||||
```bash
|
||||
git clone https://github.com/owncloud-docker/compose-playground.git
|
||||
cd compose-playground/compose/ocis
|
||||
|
||||
sed -i -e 's/your-url/192.168.103.195/g' config/identifier-registration.yml
|
||||
|
||||
cat << EOF > .env
|
||||
OCIS_BASE_URL=192.168.103.195
|
||||
OCIS_HTTP_PORT=9200
|
||||
OCIS_DOCKER_TAG=latest
|
||||
EOF
|
||||
|
||||
docker-compose -f ocis.yml -f ../cache/redis-ocis.yml up -d
|
||||
|
||||
curl -k https://192.168.103.195:9200/status.php
|
||||
```
|
||||
-254
@@ -1,254 +0,0 @@
|
||||
---
|
||||
title: "Bridge"
|
||||
date: 2020-02-27T20:35:00+01:00
|
||||
weight: 30
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: bridge.md
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
We are planning to build a bridge from ownCloud 10 to ocis. The idea is to have a reverse proxy infront of ownCloud 10 that will forward requests to ownCloud 10 or ocis-reva, depending on the migration status of the logged in user.
|
||||
|
||||
This document is a work in progress of the current setup.
|
||||
|
||||
## Current status
|
||||
|
||||
Using ocis and the ownCloud 10 openidconnect and graphapi plugins it is possible today to introduce openid connect based authentication to existing instances. That is a prerequisite for migrating to ocis.
|
||||
|
||||
## How to do it
|
||||
|
||||
### Install the owncloud 10 graphapi app
|
||||
|
||||
In an owncloud 10 apps folder
|
||||
```
|
||||
$ git clone git@github.com:owncloud/graphapi.git
|
||||
$ cd graphapi
|
||||
$ composer install
|
||||
```
|
||||
|
||||
### Enable the graphapi app
|
||||
|
||||
```
|
||||
occ a:e graphapi
|
||||
```
|
||||
|
||||
No configuration necessary. You can test with `curl`:
|
||||
```console
|
||||
$ curl https://cloud.example.com/index.php/apps/graphapi/v1.0/users -u admin | jq
|
||||
Enter host password for user 'admin':
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
100 694 100 694 0 0 4283 0 --:--:-- --:--:-- --:--:-- 4283
|
||||
{
|
||||
"value": [
|
||||
{
|
||||
"id": "admin",
|
||||
"displayName": "admin",
|
||||
"mail": null
|
||||
},
|
||||
{
|
||||
"id": "demo",
|
||||
"displayName": "Demo",
|
||||
"mail": null
|
||||
},
|
||||
...
|
||||
|
||||
],
|
||||
"@odata.nextLink": "https://oc.butonic.de/apps/graphapi/v1.0/users?$top=10&$skip=10"
|
||||
}
|
||||
```
|
||||
|
||||
> Note: The MS graph api actually asks for `Bearer` auth, but in order to check users passwords during an LDAP bind we are exploiting ownClouds authentication implementation that will grant access when `Basic` auth is used. An LDAP Bind you may ask? Read on!
|
||||
|
||||
### Start ocis-glauth
|
||||
|
||||
We are going to use the above ownCloud 10 and graphapi app to turn it into the datastore for an LDAP proxy.
|
||||
|
||||
#### Grab it!
|
||||
|
||||
In an `ocis` folder
|
||||
```
|
||||
$ git clone git@github.com:owncloud/ocis-glauth.git
|
||||
$ cd ocis-glauth
|
||||
$ make
|
||||
```
|
||||
This should give you a `bin/ocis-glauth` binary. Try listing the help with `bin/ocis-glauth --help`.
|
||||
|
||||
|
||||
#### Run it!
|
||||
|
||||
You need to point `ocis-glauth` to your owncloud domain:
|
||||
```console
|
||||
$ bin/ocis-glauth --log-level debug server --backend-datastore owncloud --backend-server https://cloud.example.com --backend-basedn dc=example,dc=com
|
||||
```
|
||||
|
||||
`--log-level debug` is only used to generate more verbose output
|
||||
`--backend-datastore owncloud` switches to tho owncloud datastore
|
||||
`--backend-server https://cloud.example.com` is the url to an ownCloud instance with an enabled graphapi app
|
||||
`--backend-basedn dc=example,dc=com` is used to construct the LDAP dn. The user `admin` will become `cn=admin,dc=example,dc=com`.
|
||||
|
||||
#### Check it is up and running
|
||||
|
||||
You should now be able to list accounts from your ownCloud 10 oc_accounts table using:
|
||||
```console
|
||||
$ ldapsearch -x -H ldap://localhost:9125 -b dc=example,dc=com -D "cn=admin,dc=example,dc=com" -W '(objectclass=posixaccount)'
|
||||
```
|
||||
|
||||
Groups should work as well:
|
||||
```console
|
||||
$ ldapsearch -x -H ldap://localhost:9125 -b dc=example,dc=com -D "cn=admin,dc=example,dc=com" -W '(objectclass=posixgroup)'
|
||||
```
|
||||
|
||||
> Note: This is currently a readonly implementation and minimal to the usecase of authenticating users with konnectd.
|
||||
|
||||
### Start ocis-phoenix
|
||||
|
||||
#### Get it!
|
||||
|
||||
In an `ocis` folder
|
||||
```
|
||||
$ git clone git@github.com:owncloud/ocis-phoenix.git
|
||||
$ cd ocis-phoenix
|
||||
$ make
|
||||
```
|
||||
This should give you a `bin/ocis-phoenix` binary. Try listing the help with `bin/ocis-phoenix --help`.
|
||||
|
||||
#### Run it!
|
||||
|
||||
Point `ocis-phoenix` to your owncloud domain and tell it where to find the openid connect issuing authority:
|
||||
```console
|
||||
$ bin/ocis-phoenix server --web-config-server https://cloud.example.com --oidc-authority https://192.168.1.100:9130 --oidc-metadata-url https://192.168.1.100:9130/.well-known/openid-configuration --oidc-client-id ocis
|
||||
```
|
||||
|
||||
`ocis-phoenix` needs to know
|
||||
- `--web-config-server https://cloud.example.com` is ownCloud url with webdav and ocs endpoints (oc10 or ocis)
|
||||
- `--oidc-authority https://192.168.1.100:9130` the openid connect issuing authority, in our case `oidc-konnectd`, running on port 9130
|
||||
- `--oidc-metadata-url https://192.168.1.100:9130/.well-known/openid-configuration` the openid connect configuration endpoint, typically the issuer host with `.well-known/openid-configuration`, but there are cases when another endpoint is used, eg. ping identity provides multiple endpoints to separate domains
|
||||
- `--oidc-client-id ocis` the client id we will register later with `ocis-konnectd` in the `identifier-registration.yaml`
|
||||
|
||||
### Start ocis-konnectd
|
||||
|
||||
#### Get it!
|
||||
|
||||
In an `ocis` folder
|
||||
```
|
||||
$ git clone git@github.com:owncloud/ocis-konnectd.git
|
||||
$ cd ocis-konnectd
|
||||
$ make
|
||||
```
|
||||
This should give you a `bin/ocis-konnectd` binary. Try listing the help with `bin/ocis-konnectd --help`.
|
||||
|
||||
#### Set environment variables
|
||||
|
||||
Konnectd needs environment variables to configure the LDAP server:
|
||||
```console
|
||||
export LDAP_URI=ldap://192.168.1.100:9125
|
||||
export LDAP_BINDDN="cn=admin,dc=example,dc=com"
|
||||
export LDAP_BINDPW="its-a-secret"
|
||||
export LDAP_BASEDN="dc=example,dc=com"
|
||||
export LDAP_SCOPE=sub
|
||||
export LDAP_LOGIN_ATTRIBUTE=uid
|
||||
export LDAP_EMAIL_ATTRIBUTE=mail
|
||||
export LDAP_NAME_ATTRIBUTE=givenName
|
||||
export LDAP_UUID_ATTRIBUTE=uid
|
||||
export LDAP_UUID_ATTRIBUTE_TYPE=text
|
||||
export LDAP_FILTER="(objectClass=posixaccount)"
|
||||
```
|
||||
Don't forget to use an existing user and the correct password.
|
||||
|
||||
### Configure clients
|
||||
|
||||
Now we need to configure a client we can later use to configure the ownCloud 10 openidconnect app. In the `assets/identifier-registration.yaml` have:
|
||||
```yaml
|
||||
---
|
||||
|
||||
# OpenID Connect client registry.
|
||||
clients:
|
||||
- id: ocis
|
||||
name: ownCloud Infinite Scale
|
||||
insecure: yes
|
||||
application_type: web
|
||||
redirect_uris:
|
||||
- https://cloud.example.com/apps/openidconnect/redirect
|
||||
- http://localhost:9100/oidc-callback.html
|
||||
- http://localhost:9100
|
||||
- http://localhost:9100/
|
||||
```
|
||||
You will need the `insecure: yes` if you are using self signed certificates.
|
||||
|
||||
Replace `cloud.example.com` in the redirect URI with your ownCloud 10 host and port.
|
||||
Replace `localhost:9100` in the redirect URIs with your the `ocis-phoenix` host and port.
|
||||
|
||||
#### Run it!
|
||||
|
||||
You can now bring up `ocis-connectd` with:
|
||||
```console
|
||||
$ bin/ocis-konnectd server --iss https://192.168.1.100:9130 --identifier-registration-conf assets/identifier-registration.yaml --signing-kid gen1-2020-02-27
|
||||
```
|
||||
|
||||
`ocis-konnectd` needs to know
|
||||
- `--iss https://192.168.1.100:9130` the issuer, which must be a reachable https endpoint. For testing an ip works. HTTPS is NOT optional. This url is exposed in the `https://192.168.1.100:9130/.well-known/openid-configuration` endpoint and clients need to be able to connect to it
|
||||
- `--identifier-registration-conf assets/identifier-registration.yaml` the identifier-registration.yaml you created
|
||||
- `--signing-kid gen1-2020-02-27` a signature key id, otherwise the jwks key has no name, which might cause problems with clients. a random key is ok, but it should change when the actual signing key changes.
|
||||
|
||||
|
||||
#### Check it is up and running
|
||||
|
||||
1. Try getting the configuration:
|
||||
```console
|
||||
$ curl https://192.168.1.100:9130/.well-known/openid-configuration
|
||||
```
|
||||
|
||||
2. Check if the login works at https://192.168.1.100:9130/signin/v1/identifier
|
||||
|
||||
> Note: If you later get a `Unable to find a key for (algorithm, kid):PS256, )` Error make sure you did set a `--signing-kid` when starting `ocis-konnectd` by checking it is present in https://192.168.1.100:9130/konnect/v1/jwks.json
|
||||
|
||||
### Patch owncloud
|
||||
|
||||
While the UserSession in ownCloud 10 is currently used to test all available IAuthModule implementations, it immediately logs out the user when an exception occurs. However, existing owncloud 10 instances use the oauth2 app to create Bearer tokens for mobile and desktop clients.
|
||||
|
||||
To give the openidconnect app a chance to verify the tokens we need to change the code a bit. See https://github.com/owncloud/core/pull/37043 for a possible solution.
|
||||
|
||||
> Note: The PR is hot ... as in *younger than this list of steps*. And it messes with authentication. Use with caution.
|
||||
|
||||
### Install the owncloud 10 openidconnect app
|
||||
|
||||
In an owncloud 10 apps folder
|
||||
```
|
||||
$ git clone git@github.com:owncloud/openidconnect.git
|
||||
$ cd openidconnect
|
||||
$ composer install
|
||||
```
|
||||
|
||||
After enabling the app configure it in `config/oidc.config.php`
|
||||
|
||||
```php
|
||||
$CONFIG = [
|
||||
'openid-connect' => [
|
||||
'provider-url' => 'https://192.168.1.100:9130',
|
||||
'client-id' => 'ocis',
|
||||
'loginButtonName' => 'OpenId Connect @ Konnectd',
|
||||
],
|
||||
'debug' => true, // if using self signed certificates
|
||||
// allow the different domains access to the ocs and wabdav endpoints:
|
||||
'cors.allowed-domains' => [
|
||||
'https://cloud.example.com',
|
||||
'http://localhost:9100',
|
||||
],
|
||||
];
|
||||
```
|
||||
|
||||
In the above configuration replace
|
||||
- `provider-url` with the URL to your `ocis-konnectd` issuer
|
||||
- `https://cloud.example.com` with the URL to your ownCloud 10 instance
|
||||
- `http://localhost:9100` with the URL to your phoenix instance
|
||||
|
||||
> Note: By default the openidconnect app will use the email of the user to match the user from the oidc userinfo endpoint with the ownCloud account. So make sure your users have a unique primary email.
|
||||
|
||||
## Next steps
|
||||
|
||||
Aside from the above todos these are the next stepo
|
||||
- tie it all together behind `ocis-proxy`
|
||||
- create an `ocis bridge` command that runs all the ocis services in one step with a properly preconfigured `ocis-konnectd` `identifier-registration.yaml` file for `phoenix` and the owncloud 10 `openidconnect` app, as well as a randomized `--signing-kid`.
|
||||
@@ -1,36 +0,0 @@
|
||||
---
|
||||
title: "Building the documentation"
|
||||
date: 2020-07-27T08:39:38+00:00
|
||||
weight: 99
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: building-docs.md
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Buildling the documentation
|
||||
|
||||
Following steps can be applied for every oCIS extension repository.
|
||||
|
||||
### Setting up
|
||||
|
||||
- Install [hugo](https://gohugo.io/getting-started/installing/)
|
||||
- Run `make docs`
|
||||
|
||||
### Viewing the documentation
|
||||
|
||||
To view the rendered docs in the browser run:
|
||||
```bash
|
||||
cd hugo
|
||||
hugo -D server
|
||||
```
|
||||
|
||||
Then open "http://localhost:1313/"
|
||||
|
||||
When making changes to the docs, run `make docs` again and the server will pick up the changes and reload the page automatically
|
||||
|
||||
### Deploying the documentation
|
||||
|
||||
The documentation is automatically deployed from the master branch to https://owncloud.github.io/phoenix/
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
---
|
||||
title: "Building"
|
||||
date: 2020-02-27T20:35:00+01:00
|
||||
weight: 50
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: building.md
|
||||
---
|
||||
|
||||
As this project is built with Go, so you need to install that first. The installation of Go is out of the scope of this document, please follow the official documentation for [Go](https://golang.org/doc/install), to build this project you have to install Go >= v1.13. After the installation of the required tools you need to get the sources:
|
||||
|
||||
{{< highlight txt >}}
|
||||
git clone https://github.com/owncloud/ocis.git
|
||||
cd ocis
|
||||
{{< / highlight >}}
|
||||
|
||||
All required tools besides Go itself and make are bundled or getting automatically installed within the `GOPATH`. All commands to build this project are part of our `Makefile`. To build the `ocis` binary run:
|
||||
|
||||
{{< highlight txt >}}
|
||||
make generate
|
||||
make build
|
||||
{{< / highlight >}}
|
||||
|
||||
Finally, you should have the binary within the `bin/` folder now, give it a try with `./bin/ocis -h` to see all available options.
|
||||
|
||||
|
||||
## Simple Ocis fo extonsions example
|
||||
|
||||
Currently, we are using a go build tag to allow building a more simple set of the binary. It was intended to let extension developers focus on only the necessary services.
|
||||
|
||||
{{< hint info >}}
|
||||
While it the tag based simple build demonstrates how to use ocis as a framework for a micro service architecture, we may change to an approach that uses an explicit command to run only a subset of the services.
|
||||
{{< / hint >}}
|
||||
|
||||
```console
|
||||
TAGS=simple make build
|
||||
```
|
||||
|
||||
The artifact lives in `/bin/ocis`
|
||||
|
||||
The generated simple ocis binary is a subset of the ocis command with a restricted set of services meant for ease up development. The services included are
|
||||
|
||||
```
|
||||
ocis-hello
|
||||
ocis-phoenix
|
||||
ocis-konnectd
|
||||
ocis-glauth
|
||||
micro's own services
|
||||
```
|
||||
@@ -1,238 +0,0 @@
|
||||
---
|
||||
title: "Configuration"
|
||||
date: "2020-08-21T17:50:21+0200"
|
||||
weight: 20
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: configuration.md
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Configuration
|
||||
|
||||
oCIS Single Binary is not responsible for configuring extensions. Instead, each extension could either be configured by environment variables, cli flags or config files.
|
||||
|
||||
Each extension has its dedicated documentation page (e.g. https://owncloud.github.io/extensions/ocis_proxy/configuration) which lists all possible configurations. Config files and environment variables are picked up if you use the `./bin/ocis server` command within the oCIS single binary. Command line flags must be set explicitly on the extensions subcommands.
|
||||
|
||||
### Configuration using config files
|
||||
|
||||
Out of the box extensions will attempt to read configuration details from:
|
||||
|
||||
```console
|
||||
/etc/ocis
|
||||
$HOME/.ocis
|
||||
./config
|
||||
```
|
||||
|
||||
For this configuration to be picked up, have a look at your extension `root` command and look for which default config name it has assigned. *i.e: ocis-proxy reads `proxy.json | yaml | toml ...`*.
|
||||
|
||||
So far we support the file formats `JSON` and `YAML`, if you want to get a full example configuration just take a look at [our repository](https://github.com/owncloud/ocis/tree/master/config), there you can always see the latest configuration format. These example configurations include all available options and the default values. The configuration file will be automatically loaded if it's placed at `/etc/ocis/ocis.yml`, `${HOME}/.ocis/ocis.yml` or `$(pwd)/config/ocis.yml`.
|
||||
|
||||
### Envrionment variables
|
||||
|
||||
If you prefer to configure the service with environment variables you can see the available variables below.
|
||||
|
||||
### Commandline flags
|
||||
|
||||
If you prefer to configure the service with commandline flags you can see the available variables below. Command line flags are only working when calling the subcommand directly.
|
||||
|
||||
## Root Command
|
||||
|
||||
ownCloud Infinite Scale Stack
|
||||
|
||||
Usage: `ocis [global options] command [command options] [arguments...]`
|
||||
|
||||
--config-file | $OCIS_CONFIG_FILE
|
||||
: Path to config file.
|
||||
|
||||
--log-level | $OCIS_LOG_LEVEL
|
||||
: Set logging level. Default: `info`.
|
||||
|
||||
--log-pretty | $OCIS_LOG_PRETTY
|
||||
: Enable pretty logging. Default: `true`.
|
||||
|
||||
--log-color | $OCIS_LOG_COLOR
|
||||
: Enable colored logging. Default: `true`.
|
||||
|
||||
## Sub Commands
|
||||
|
||||
### ocis server
|
||||
|
||||
Start fullstack server
|
||||
|
||||
Usage: `ocis server [command options] [arguments...]`
|
||||
|
||||
--tracing-enabled | $OCIS_TRACING_ENABLED
|
||||
: Enable sending traces.
|
||||
|
||||
--tracing-type | $OCIS_TRACING_TYPE
|
||||
: Tracing backend type. Default: `jaeger`.
|
||||
|
||||
--tracing-endpoint | $OCIS_TRACING_ENDPOINT
|
||||
: Endpoint for the agent. Default: `localhost:6831`.
|
||||
|
||||
--tracing-collector | $OCIS_TRACING_COLLECTOR
|
||||
: Endpoint for the collector. Default: `http://localhost:14268/api/traces`.
|
||||
|
||||
--tracing-service | $OCIS_TRACING_SERVICE
|
||||
: Service name for tracing. Default: `ocis`.
|
||||
|
||||
--debug-addr | $OCIS_DEBUG_ADDR
|
||||
: Address to bind debug server. Default: `0.0.0.0:9010`.
|
||||
|
||||
--debug-token | $OCIS_DEBUG_TOKEN
|
||||
: Token to grant metrics access.
|
||||
|
||||
--debug-pprof | $OCIS_DEBUG_PPROF
|
||||
: Enable pprof debugging.
|
||||
|
||||
--debug-zpages | $OCIS_DEBUG_ZPAGES
|
||||
: Enable zpages debugging.
|
||||
|
||||
--http-addr | $OCIS_HTTP_ADDR
|
||||
: Address to bind http server. Default: `0.0.0.0:9000`.
|
||||
|
||||
--http-root | $OCIS_HTTP_ROOT
|
||||
: Root path of http server. Default: `/`.
|
||||
|
||||
--grpc-addr | $OCIS_GRPC_ADDR
|
||||
: Address to bind grpc server. Default: `0.0.0.0:9001`.
|
||||
|
||||
### ocis health
|
||||
|
||||
Check health status
|
||||
|
||||
Usage: `ocis health [command options] [arguments...]`
|
||||
|
||||
--debug-addr | $OCIS_DEBUG_ADDR
|
||||
: Address to debug endpoint. Default: `0.0.0.0:9010`.
|
||||
|
||||
### List of available Extension subcommands
|
||||
|
||||
There are more subcommands to start the individual extensions. Please check the documentation about their usage and options in the dedicated section of the documentation.
|
||||
|
||||
#### ocis glauth
|
||||
|
||||
Start glauth server
|
||||
|
||||
#### ocis ocs
|
||||
|
||||
Start ocs server
|
||||
|
||||
#### ocis store
|
||||
|
||||
Start a go-micro store
|
||||
|
||||
#### ocis reva-auth-basic
|
||||
|
||||
Start reva auth-basic service
|
||||
|
||||
#### ocis reva-storage-oc
|
||||
|
||||
Start reva oc storage
|
||||
|
||||
#### ocis run
|
||||
|
||||
Runs an extension
|
||||
|
||||
#### ocis graph-explorer
|
||||
|
||||
Start graph explorer
|
||||
|
||||
#### ocis hello
|
||||
|
||||
Start hello server
|
||||
|
||||
#### ocis phoenix
|
||||
|
||||
Start phoenix server
|
||||
|
||||
#### ocis reva-storage-eos-data
|
||||
|
||||
Start reva eos storage dataprovider
|
||||
|
||||
#### ocis reva-frontend
|
||||
|
||||
Start reva frontend
|
||||
|
||||
#### ocis reva-storage-home
|
||||
|
||||
Start reva home storage
|
||||
|
||||
#### ocis webdav
|
||||
|
||||
Start webdav server
|
||||
|
||||
#### ocis konnectd
|
||||
|
||||
Start konnectd server
|
||||
|
||||
#### ocis reva-auth-bearer
|
||||
|
||||
Start reva auth-bearer service
|
||||
|
||||
#### ocis reva-storage-home-data
|
||||
|
||||
Start reva home storage dataprovider
|
||||
|
||||
#### ocis reva-users
|
||||
|
||||
Start reva users service
|
||||
|
||||
#### ocis accounts
|
||||
|
||||
Start accounts server
|
||||
|
||||
#### ocis graph
|
||||
|
||||
Start graph server
|
||||
|
||||
#### ocis reva-storage-oc-data
|
||||
|
||||
Start reva oc storage dataprovider
|
||||
|
||||
#### ocis proxy
|
||||
|
||||
Start proxy server
|
||||
|
||||
#### ocis reva-storage-eos
|
||||
|
||||
Start reva eos storage
|
||||
|
||||
#### ocis settings
|
||||
|
||||
Start settings server
|
||||
|
||||
#### ocis kill
|
||||
|
||||
Kill an extension by name
|
||||
|
||||
#### ocis list
|
||||
|
||||
Lists running ocis extensions
|
||||
|
||||
#### ocis import
|
||||
|
||||
Import a user exported by owncloud/data_exporter
|
||||
|
||||
#### ocis thumbnails
|
||||
|
||||
Start thumbnails server
|
||||
|
||||
#### ocis reva-gateway
|
||||
|
||||
Start reva gateway
|
||||
|
||||
#### ocis reva-sharing
|
||||
|
||||
Start reva sharing service
|
||||
|
||||
#### ocis reva-storage-public-link
|
||||
|
||||
Start reva public link storage
|
||||
|
||||
#### ocis reva-storage-root
|
||||
|
||||
Start reva root storage
|
||||
|
||||
@@ -1,217 +0,0 @@
|
||||
---
|
||||
title: "Debugging"
|
||||
date: 2020-03-19T08:21:00+01:00
|
||||
weight: 50
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: debugging.md
|
||||
---
|
||||
|
||||
## Debugging
|
||||
|
||||
As a single binary for easy deployment running `ocis server` just forks itself to start all the services, which makes debugging those processes a little harder.
|
||||
|
||||
Ultimately, we want to be able to stop a single service using eg. `ocis kill phoenix` so that you can start the service you want to debug in debug mode. We need to [change the way we fork processes](https://github.com/owncloud/ocis/issues/77) though, otherwise the runtime will automatically restart a service if killed.
|
||||
|
||||
### Start ocis
|
||||
|
||||
For debugging there are two workflows that work well, depending on your preferences.
|
||||
|
||||
#### Use the debug binary and attach to the process as needed
|
||||
|
||||
Run the debug binary with `OCIS_LOG_LEVEL=debug bin/ocis-debug server` and then find the service you want to debug using:
|
||||
|
||||
```console
|
||||
# ps ax | grep ocis
|
||||
12837 pts/1 Sl+ 0:00 bin/ocis-debug server
|
||||
12845 pts/1 Sl 0:00 bin/ocis-debug graph
|
||||
12847 pts/1 Sl 0:00 bin/ocis-debug reva-auth-bearer
|
||||
12848 pts/1 Sl 0:00 bin/ocis-debug graph-explorer
|
||||
12849 pts/1 Sl 0:00 bin/ocis-debug ocs
|
||||
12850 pts/1 Sl 0:00 bin/ocis-debug reva-storage-oc-data
|
||||
12863 pts/1 Sl 0:00 bin/ocis-debug webdav
|
||||
12874 pts/1 Sl 0:00 bin/ocis-debug reva-frontend
|
||||
12897 pts/1 Sl 0:00 bin/ocis-debug reva-sharing
|
||||
12905 pts/1 Sl 0:00 bin/ocis-debug reva-gateway
|
||||
12912 pts/1 Sl 0:00 bin/ocis-debug reva-storage-home
|
||||
12920 pts/1 Sl 0:00 bin/ocis-debug reva-users
|
||||
12929 pts/1 Sl 0:00 bin/ocis-debug glauth
|
||||
12940 pts/1 Sl 0:00 bin/ocis-debug reva-storage-home-data
|
||||
12948 pts/1 Sl 0:00 bin/ocis-debug konnectd
|
||||
12952 pts/1 Sl 0:00 bin/ocis-debug proxy
|
||||
12961 pts/1 Sl 0:00 bin/ocis-debug thumbnails
|
||||
12971 pts/1 Sl 0:00 bin/ocis-debug reva-storage-oc
|
||||
12981 pts/1 Sl 0:00 bin/ocis-debug web
|
||||
12993 pts/1 Sl 0:00 bin/ocis-debug api
|
||||
12998 pts/1 Sl 0:00 bin/ocis-debug registry
|
||||
13004 pts/1 Sl 0:00 bin/ocis-debug phoenix
|
||||
13015 pts/1 Sl 0:00 bin/ocis-debug reva-auth-basic
|
||||
```
|
||||
|
||||
Then you can set a breakpoint in the service you need and attach to the process via processid. To debug the `reva-sharing` service the VS Code `launch.json` would look like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "0.2.0",
|
||||
"configurations": [
|
||||
{
|
||||
"name": "ocis attach",
|
||||
"type": "go",
|
||||
"request": "attach",
|
||||
"mode": "local",
|
||||
"processId": 12897,
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Start all services independently to replace one of them with a debug process
|
||||
|
||||
1. You can use this `./ocis.sh` script to start all services independently, so they don't get restrarted by the runtime when you kill them:
|
||||
```bash
|
||||
#/bin/sh
|
||||
LOG_LEVEL="debug"
|
||||
|
||||
bin/ocis --log-level=$LOG_LEVEL micro &
|
||||
|
||||
bin/ocis --log-level=$LOG_LEVEL glauth &
|
||||
bin/ocis --log-level=$LOG_LEVEL graph-explorer &
|
||||
bin/ocis --log-level=$LOG_LEVEL graph &
|
||||
#bin/ocis --log-level=$LOG_LEVEL hello &
|
||||
bin/ocis --log-level=$LOG_LEVEL konnectd &
|
||||
#bin/ocis --log-level=$LOG_LEVEL ocs &
|
||||
bin/ocis --log-level=$LOG_LEVEL phoenix &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-auth-basic &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-auth-bearer &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-frontend &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-gateway &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-sharing &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-storage-home &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-storage-home-data &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-storage-oc &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-storage-oc-data &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-storage-root &
|
||||
bin/ocis --log-level=$LOG_LEVEL reva-users &
|
||||
#bin/ocis --log-level=$LOG_LEVEL webdav
|
||||
|
||||
bin/ocis --log-level=$LOG_LEVEL proxy &
|
||||
```
|
||||
|
||||
2. Get the list of running processes:
|
||||
|
||||
```console
|
||||
# ps ax | grep ocis
|
||||
12837 pts/1 Sl+ 0:00 bin/ocis-debug server
|
||||
12845 pts/1 Sl 0:00 bin/ocis-debug graph
|
||||
12847 pts/1 Sl 0:00 bin/ocis-debug reva-auth-bearer
|
||||
12848 pts/1 Sl 0:00 bin/ocis-debug graph-explorer
|
||||
12849 pts/1 Sl 0:00 bin/ocis-debug ocs
|
||||
12850 pts/1 Sl 0:00 bin/ocis-debug reva-storage-oc-data
|
||||
12863 pts/1 Sl 0:00 bin/ocis-debug webdav
|
||||
12874 pts/1 Sl 0:00 bin/ocis-debug reva-frontend
|
||||
12897 pts/1 Sl 0:00 bin/ocis-debug reva-sharing
|
||||
12905 pts/1 Sl 0:00 bin/ocis-debug reva-gateway
|
||||
12912 pts/1 Sl 0:00 bin/ocis-debug reva-storage-home
|
||||
12920 pts/1 Sl 0:00 bin/ocis-debug reva-users
|
||||
12929 pts/1 Sl 0:00 bin/ocis-debug glauth
|
||||
12940 pts/1 Sl 0:00 bin/ocis-debug reva-storage-home-data
|
||||
12948 pts/1 Sl 0:00 bin/ocis-debug konnectd
|
||||
12952 pts/1 Sl 0:00 bin/ocis-debug proxy
|
||||
12961 pts/1 Sl 0:00 bin/ocis-debug thumbnails
|
||||
12971 pts/1 Sl 0:00 bin/ocis-debug reva-storage-oc
|
||||
12981 pts/1 Sl 0:00 bin/ocis-debug web
|
||||
12993 pts/1 Sl 0:00 bin/ocis-debug api
|
||||
12998 pts/1 Sl 0:00 bin/ocis-debug registry
|
||||
13004 pts/1 Sl 0:00 bin/ocis-debug phoenix
|
||||
13015 pts/1 Sl 0:00 bin/ocis-debug reva-auth-basic
|
||||
```
|
||||
|
||||
3. Kill the service you want to start in debug mode:
|
||||
|
||||
|
||||
```console
|
||||
# kill 17628
|
||||
```
|
||||
|
||||
4. Start the service you are interested in in debug mode. When using make to build the binary there is already a `bin/ocis-debug` binary for you. When running an IDE tell it which service to start by providing the corresponding sub command, eg. `bin\ocis-debug reva-frontend`.
|
||||
|
||||
### Gather error messages
|
||||
|
||||
We recommend you collect all related information in a single file or in a github issue. Let us start with an error that pops up in the Web UI:
|
||||
|
||||
> Error while sharing.
|
||||
> error sending a grpc stat request
|
||||
|
||||
This popped up when I tried to add `marie` as a collaborator in phoenix. That triggers a request to the server which I copied as curl. We can strip a lot of headers and the gist of it is:
|
||||
|
||||
```console
|
||||
# curl 'https://localhost:9200/ocs/v1.php/apps/files_sharing/api/v1/shares' -d 'shareType=0&shareWith=marie&path=%2FNeuer+Ordner&permissions=1' -u einstein:relativity -k -v | xmllint -format -
|
||||
[... headers ...]
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<ocs>
|
||||
<meta>
|
||||
<status>error</status>
|
||||
<statuscode>998</statuscode>
|
||||
<message>error sending a grpc stat request</message>
|
||||
</meta>
|
||||
</ocs>
|
||||
```
|
||||
|
||||
{{< hint info >}}
|
||||
The username and password only work when basic auth is available. Otherwise you have to obtain a bearer token, eg. by grabbing it from the browser.
|
||||
{{< /hint >}}
|
||||
{{< hint danger >}}
|
||||
TODO add ocis cli tool to obtain a bearer token.
|
||||
{{< /hint >}}
|
||||
|
||||
We also have a few interesting log entries:
|
||||
|
||||
```
|
||||
0:43PM INF home/jfd/go/pkg/mod/github.com/cs3org/reva@v0.0.2-0.20200318111623-a2f97d4aa741/internal/grpc/interceptors/log/log.go:69 > unary code=OK end="18/Mar/2020:22:43:40 +0100" from=tcp://[::1]:44078 pid=17836 pkg=rgrpc start="18/Mar/2020:22:43:40 +0100" time_ns=95841 traceid=b4eb9a9f45921f7d3632523ca32a42b0 uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider user-agent=grpc-go/1.26.0
|
||||
10:43PM ERR home/jfd/go/pkg/mod/github.com/cs3org/reva@v0.0.2-0.20200318111623-a2f97d4aa741/internal/grpc/interceptors/log/log.go:69 > unary code=Unknown end="18/Mar/2020:22:43:40 +0100" from=tcp://[::1]:43910 pid=17836 pkg=rgrpc start="18/Mar/2020:22:43:40 +0100" time_ns=586115 traceid=b4eb9a9f45921f7d3632523ca32a42b0 uri=/cs3.gateway.v1beta1.GatewayAPI/Stat user-agent=grpc-go/1.26.0
|
||||
10:43PM ERR home/jfd/go/pkg/mod/github.com/cs3org/reva@v0.0.2-0.20200318111623-a2f97d4aa741/internal/http/services/owncloud/ocs/reqres.go:94 > error sending a grpc stat request error="rpc error: code = Unknown desc = gateway: error calling Stat: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp [::1]:9152: connect: connection refused\"" pid=17832 pkg=rhttp traceid=b4eb9a9f45921f7d3632523ca32a42b0
|
||||
```
|
||||
|
||||
{{< hint danger >}}
|
||||
TODO return the trace id in the response so we can correlate easier. For reva tracked in https://github.com/cs3org/reva/issues/587
|
||||
{{< /hint >}}
|
||||
|
||||
The last line gives us a hint where the log message originated: `.../github.com/cs3org/reva@v0.0.2-0.20200318111623-a2f97d4aa741/internal/http/services/owncloud/ocs/reqres.go:94`. Which looks like this:
|
||||
|
||||
```go
|
||||
89: // WriteOCSResponse handles writing ocs responses in json and xml
|
||||
90: func WriteOCSResponse(w http.ResponseWriter, r *http.Request, res *Response, err error) {
|
||||
91: var encoded []byte
|
||||
92:
|
||||
93: if err != nil {
|
||||
94: appctx.GetLogger(r.Context()).Error().Err(err).Msg(res.OCS.Meta.Message)
|
||||
95: }
|
||||
```
|
||||
|
||||
Ok, so this seems to be a convenience method that is called from multiple places an also handles errors. Unfortunately, this hides the actual source of the error. We could set a breakpoint in line 94 and reproduce the problem, which can be a lot harder than just clicking the share button or sending a curl request again. So let us see what else the log tells us.
|
||||
|
||||
The previous line tells us that a Stat request failed: `uri=/cs3.gateway.v1beta1.GatewayAPI/Stat`. This time the line is written by the grpc log interceptor. What else is there?
|
||||
|
||||
The first line tells us that looking up the responsible storage provider seems to have succeeded: `uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider`.
|
||||
|
||||
At this point it your familiarity with the codebase starts to become a factor. If you are new you should probably go back to setting a break point on the log line and check the stack trace.
|
||||
|
||||
Debug wherever the call trace leads you to ... good luck!
|
||||
|
||||
### Managing dependencies and testing changes
|
||||
|
||||
You can either run and manage the services independently, or you can update the `go.mod` file and replace dependencies with your local version.
|
||||
|
||||
To debug the reva frontend we need to add two replacements:
|
||||
```
|
||||
// use the local ocis-reva repo
|
||||
replace github.com/owncloud/ocis-reva => ../ocis-reva
|
||||
// also use the local reva repo
|
||||
replace github.com/cs3org/reva => ../reva
|
||||
```
|
||||
{{< hint info >}}
|
||||
The username and password only work when basic auth is available. Otherwise you have to obtain a bearer token, eg. by grabbing it from the browser.
|
||||
{{< /hint >}}
|
||||
|
||||
Rebuild ocis to make sure the dependency is used. It should be sufficient to just restart the service you want to debug.
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
---
|
||||
title: "Getting Started with Development"
|
||||
date: 2020-07-07T20:35:00+01:00
|
||||
weight: 15
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: development.md
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Docker dev environment
|
||||
|
||||
### Option 1: Plain docker
|
||||
|
||||
To build and run your local ocis code with default storage driver
|
||||
|
||||
```
|
||||
docker run --rm -ti --name ocis -v $PWD:/ocis -p 9200:9200 owncloud/eos-ocis-dev
|
||||
```
|
||||
|
||||
The eos-ocis-dev container will build and run ocis using the owncloud storage driver and store files in the container at `/var/tmp/reva/data/<username>/files`
|
||||
|
||||
To check the uploaded files start digging with: `docker exec -it ocis ls -l /var/tmp/reva/`
|
||||
|
||||
{{< hint info >}}
|
||||
On MacOS do not mount a local folder to the `/var/tmp/reva/` path. The fuse driver used by docker [does not support extended attributes](https://docs.docker.com/v18.09/docker-for-mac/osxfs/). See [#182](https://github.com/owncloud/ocis/issues/182) for more details.
|
||||
{{< /hint >}}
|
||||
|
||||
|
||||
### Option 2: Docker compose
|
||||
|
||||
With the `docker-compose.yml` file in ocis repo you can also start ocis via compose:
|
||||
|
||||
```
|
||||
docker-compose up -d ocis
|
||||
```
|
||||
|
||||
{{< hint info >}}
|
||||
We are only starting the `ocis` container here.
|
||||
{{< /hint >}}
|
||||
|
||||
## Verification
|
||||
|
||||
Check the services are running
|
||||
|
||||
```
|
||||
$ docker-compose exec ocis ./bin/ocis list
|
||||
+--------------------------+-----+
|
||||
| EXTENSION | PID |
|
||||
+--------------------------+-----+
|
||||
| accounts | 172 |
|
||||
| api | 204 |
|
||||
| glauth | 187 |
|
||||
| graph | 41 |
|
||||
| graph-explorer | 55 |
|
||||
| konnectd | 196 |
|
||||
| ocs | 59 |
|
||||
| phoenix | 29 |
|
||||
| proxy | 22 |
|
||||
| registry | 226 |
|
||||
| reva-auth-basic | 96 |
|
||||
| reva-auth-bearer | 104 |
|
||||
| reva-frontend | 485 |
|
||||
| reva-gateway | 78 |
|
||||
| reva-sharing | 286 |
|
||||
| reva-storage-eos | 129 |
|
||||
| reva-storage-eos-data | 134 |
|
||||
| reva-storage-home | 442 |
|
||||
| reva-storage-home-data | 464 |
|
||||
| reva-storage-oc | 149 |
|
||||
| reva-storage-oc-data | 155 |
|
||||
| reva-storage-public-link | 168 |
|
||||
| reva-users | 420 |
|
||||
| settings | 23 |
|
||||
| thumbnails | 201 |
|
||||
| web | 218 |
|
||||
| webdav | 63 |
|
||||
+--------------------------+-----+
|
||||
```
|
||||
-220
@@ -1,220 +0,0 @@
|
||||
---
|
||||
title: "EOS"
|
||||
date: 2020-02-27T20:35:00+01:00
|
||||
weight: 30
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: eos.md
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
OCIS can be configured to run on top of [eos](https://eos.web.cern.ch/). While the [eos documentation](http://eos-docs.web.cern.ch/) does cover a lot of topics it leaves out some details that you may have to either pull from various [docker containers](https://gitlab.cern.ch/eos/eos-docker), the [forums](https://eos-community.web.cern.ch/) or even the [source](https://github.com/cern-eos/eos) itself.
|
||||
|
||||
This document is a work in progress of the current setup.
|
||||
|
||||
## Docker dev environment for eos storage
|
||||
|
||||
We begin with the docker-compose.yml found in https://github.com/owncloud/ocis/ and
|
||||
switch it to eos-storage.
|
||||
|
||||
### 1. Start eos & ocis containers
|
||||
|
||||
Start the eos cluster and ocis via the compose stack.
|
||||
```
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
{{< hint info >}}
|
||||
The first time the **ocis** container starts up, it will compile ocis from scratch which can take a while.
|
||||
To follow progress, run `docker-compose logs -f --tail=10 ocis`
|
||||
{{< /hint >}}
|
||||
|
||||
### 2. LDAP support
|
||||
|
||||
Configure the OS to resolve users and groups using ldap
|
||||
|
||||
```
|
||||
docker-compose exec -d ocis /start-ldap
|
||||
```
|
||||
|
||||
Check that the OS in the ocis container can now resolve einstein or the other demo users
|
||||
|
||||
```
|
||||
$ docker-compose exec ocis id einstein
|
||||
uid=20000(einstein) gid=30000(users) groups=30000(users),30001(sailing-lovers),30002(violin-haters),30007(physics-lovers)
|
||||
```
|
||||
|
||||
{{< hint info >}}
|
||||
If the user is not found at first you might need to wait a few more minutes in case the ocis container is still compiling.
|
||||
{{< /hint >}}
|
||||
|
||||
We also need to restart the reva-users service so it picks up the changed environment. Without a restart it is not able to resolve users from LDAP.
|
||||
```
|
||||
docker-compose exec ocis ./bin/ocis kill reva-users
|
||||
docker-compose exec ocis ./bin/ocis run reva-users
|
||||
```
|
||||
|
||||
### 3. Home storage
|
||||
|
||||
Kill the home storage. By default it uses the `owncloud` storage driver. We need to switch it to the `eoshome` driver and make it use the storage id of the eos storage provider:
|
||||
|
||||
```
|
||||
docker-compose exec ocis ./bin/ocis kill reva-storage-home
|
||||
docker-compose exec -e REVA_STORAGE_HOME_DRIVER=eoshome -e REVA_STORAGE_HOME_MOUNT_ID=1284d238-aa92-42ce-bdc4-0b0000009158 ocis ./bin/ocis run reva-storage-home
|
||||
```
|
||||
|
||||
### 4. Home data provider
|
||||
|
||||
Kill the home data provider. By default it uses the `owncloud` storage driver. We need to switch it to the `eoshome` driver and make it use the storage id of the eos storage provider:
|
||||
|
||||
```
|
||||
docker-compose exec ocis ./bin/ocis kill reva-storage-home-data
|
||||
docker-compose exec -e REVA_STORAGE_HOME_DATA_DRIVER=eoshome ocis ./bin/ocis run reva-storage-home-data
|
||||
```
|
||||
|
||||
{{< hint info >}}
|
||||
The difference between the *home storage* and the *home data provider* are that the former is responsible for metadata changes while the latter is responsible for actual data transfer. The *home storage* uses the cs3 api to manage a folder hierarchy, while the *home data provider* is responsible for moving bytes to and from the storage.
|
||||
{{< /hint >}}
|
||||
|
||||
## Verification
|
||||
|
||||
Login with `einstein / relativity`, upload a file to einsteins home and verify the file is there using
|
||||
|
||||
```
|
||||
docker-compose exec ocis eos ls -l /eos/dockertest/reva/users/4/4c510ada-c86b-4815-8820-42cdf82c3d51/
|
||||
-rw-r--r-- 1 einstein users 10 Jul 1 15:24 newfile.txt
|
||||
```
|
||||
|
||||
If the problem persists, please check the [troubleshooting section about uploads](#creation-and-upload-of-files-does-not-work).
|
||||
|
||||
## Further exploration
|
||||
|
||||
EOS has a built in shell that you can enter using
|
||||
```
|
||||
$ docker-compose exec mgm-master eos
|
||||
# ---------------------------------------------------------------------------
|
||||
# EOS Copyright (C) 2011-2019 CERN/Switzerland
|
||||
# This program comes with ABSOLUTELY NO WARRANTY; for details type `license'.
|
||||
# This is free software, and you are welcome to redistribute it
|
||||
# under certain conditions; type `license' for details.
|
||||
# ---------------------------------------------------------------------------
|
||||
EOS_INSTANCE=eostest
|
||||
EOS_SERVER_VERSION=4.6.5 EOS_SERVER_RELEASE=1
|
||||
EOS_CLIENT_VERSION=4.6.5 EOS_CLIENT_RELEASE=1
|
||||
EOS Console [root://localhost] |/> help
|
||||
access Access Interface
|
||||
accounting Accounting Interface
|
||||
acl Acl Interface
|
||||
archive Archive Interface
|
||||
attr Attribute Interface
|
||||
backup Backup Interface
|
||||
clear Clear the terminal
|
||||
cd Change directory
|
||||
chmod Mode Interface
|
||||
chown Chown Interface
|
||||
config Configuration System
|
||||
console Run Error Console
|
||||
cp Cp command
|
||||
debug Set debug level
|
||||
exit Exit from EOS console
|
||||
file File Handling
|
||||
fileinfo File Information
|
||||
find Find files/directories
|
||||
newfind Find files/directories (new implementation)
|
||||
fs File System configuration
|
||||
fsck File System Consistency Checking
|
||||
fuse Fuse Mounting
|
||||
fusex Fuse(x) Administration
|
||||
geosched Geoscheduler Interface
|
||||
group Group configuration
|
||||
health Health information about system
|
||||
help Display this text
|
||||
info Retrieve file or directory information
|
||||
inspector Interact with File Inspector
|
||||
io IO Interface
|
||||
json Toggle JSON output flag for stdout
|
||||
license Display Software License
|
||||
ls List a directory
|
||||
ln Create a symbolic link
|
||||
map Path mapping interface
|
||||
member Check Egroup membership
|
||||
mkdir Create a directory
|
||||
motd Message of the day
|
||||
mv Rename file or directory
|
||||
node Node configuration
|
||||
ns Namespace Interface
|
||||
pwd Print working directory
|
||||
quit Exit from EOS console
|
||||
quota Quota System configuration
|
||||
reconnect Forces a re-authentication of the shell
|
||||
recycle Recycle Bin Functionality
|
||||
rmdir Remove a directory
|
||||
rm Remove a file
|
||||
role Set the client role
|
||||
route Routing interface
|
||||
rtlog Get realtime log output from mgm & fst servers
|
||||
silent Toggle silent flag for stdout
|
||||
space Space configuration
|
||||
stagerrm Remove disk replicas of a file if it has tape replicas
|
||||
stat Run 'stat' on a file or directory
|
||||
squash Run 'squashfs' utility function
|
||||
test Run performance test
|
||||
timing Toggle timing flag for execution time measurement
|
||||
touch Touch a file
|
||||
token Token interface
|
||||
tracker Interact with File Tracker
|
||||
transfer Transfer Interface
|
||||
version Verbose client/server version
|
||||
vid Virtual ID System Configuration
|
||||
whoami Determine how we are mapped on server side
|
||||
who Statistics about connected users
|
||||
? Synonym for 'help'
|
||||
.q Exit from EOS console
|
||||
EOS Console [root://localhost] |/>
|
||||
```
|
||||
|
||||
But this is a different adventure. See the links at the top of this page for other sources of information on eos.
|
||||
|
||||
## Cleaning up
|
||||
|
||||
To clean up and start completely from scratch, run `docker-compose down -v`.
|
||||
Then delete the local "bin" folder as root which contains the ocis binaries compiled by the "ocis" docker.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Docker-compose exits right away
|
||||
|
||||
When running `docker-compose up -d` ocis exits right away.
|
||||
|
||||
You can check the error code using `docker-compose ps` and investigate further by running only ocis again using `docker-compose up ocis` (without `-d` so you can see what is going on in the foreground).
|
||||
One reason might be that the binary was already built but does not match the container env. Try running `make clean` before running `docker-compose up ocis` so it gets built inside the container.
|
||||
|
||||
### Where are the logs ?
|
||||
|
||||
The ocis logs can be accessed using `docker-compose logs ocis`. Add `-f` for following.
|
||||
|
||||
### How do I update a service in the ocis container?
|
||||
|
||||
1. `docker-compose exec ocis make clean build` to update the binary
|
||||
2. `docker-compose exec ocis ./bin/ocis kill <service>` to kill the service
|
||||
3. `docker-compose exec ocis ./bin/ocis run <service>` to start the service. Do not forget to set any env vars, eg.
|
||||
`docker-compose exec -e REVA_STORAGE_EOS_LAYOUT="{{substr 0 1 .Id.OpaqueId}}/{{.Id.OpaqueId}}" -e REVA_STORAGE_HOME_DRIVER=eoshome ocis ./bin/ocis run reva-storage-home`
|
||||
|
||||
### Creation and upload of files does not work
|
||||
|
||||
If the upload did not work, please check the status of the eos space using the command `docker-compose exec mgm-master eos fs ls`.
|
||||
In case the default space appears as offline, run `docker-compose exec mgm-master eos space set default on`.
|
||||
|
||||
### Uploading big files appears to hang
|
||||
|
||||
Please note that the uploads first go into the "ocis" docker and land in its "/tmp" folder, then gets copied over to the EOS docker using `xrdcopy`.
|
||||
This is why uploading first transfers all bytes and then seem to hang for a while during the final copy.
|
||||
|
||||
### Running out of space quickly
|
||||
|
||||
The EOS dockers are configured with replication, so every file uploaded there will be replicated 4 times,
|
||||
so make sure there is enough physical space on disk when testing.
|
||||
|
||||
Also please note that older failed uploads might still be present in the "/tmp" directory of the "ocis" container.
|
||||
|
||||
@@ -1,237 +0,0 @@
|
||||
---
|
||||
title: "Extension"
|
||||
date: 2020-02-27T20:35:00+01:00
|
||||
weight: 40
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: extensions.md
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## How to build and run ocis-simple
|
||||
|
||||
ocis uses build tags to build different flavors of the binary. In order to work on a new extension we are going to reduce the scope a little and use the `simple` tag. Let us begin by creating a dedicated folder:
|
||||
|
||||
```console
|
||||
mkdir ocis-extension-workshop && ocis-extension-workshop
|
||||
```
|
||||
|
||||
Following https://github.com/owncloud/ocis
|
||||
|
||||
```console
|
||||
git clone https://github.com/owncloud/ocis.git
|
||||
cd ocis
|
||||
|
||||
TAGS=simple make generate build
|
||||
```
|
||||
|
||||
*Q: Can you specify which version of phoenix to use?*
|
||||
*A: No, the phoenix that is used is compiled into the [assets of ocis-phoenix](https://github.com/owncloud/ocis-phoenix/blob/master/pkg/assets/embed.go) which is currently not automatically updated. We'll see how to use a custom phoenix later.*
|
||||
|
||||
`bin/ocis server`
|
||||
|
||||
Open the browser at http://localhost:9100
|
||||
|
||||
1. You land on the login screen. click login
|
||||
2. You are redirected to an idp at http://localhost:9140/oauth2/auth with a login mask. Use `einstein:relativity`to login (one of the three demo users)
|
||||
3. You are redirected to http://localhost:9100/#/hello the ocis-hello app
|
||||
4. Replace `World` with something else and submit. You should see `Hello %something else%`
|
||||
|
||||
*Q: One of the required ports is already in use. Ocis seems to be trying to restart the service over and over. What gives?*
|
||||
*A: Using the ocis binary to start the server will case ocis to keep track of the different services and restart them in case they crash.*
|
||||
|
||||
## Hacking ocis-hello
|
||||
|
||||
go back to the ocis-extension-workshop folder
|
||||
|
||||
```console
|
||||
cd ..
|
||||
```
|
||||
|
||||
Following https://github.com/owncloud/ocis-hello
|
||||
|
||||
```
|
||||
git clone https://github.com/owncloud/ocis-hello.git
|
||||
cd ocis-hello
|
||||
|
||||
yarn install
|
||||
# this actually creates the assets
|
||||
yarn build
|
||||
|
||||
# this will compile the assets into the binary
|
||||
make generate build
|
||||
```
|
||||
|
||||
Two options:
|
||||
1. run only the necessery services from ocis and ocis-hello independently
|
||||
2. compile ocis with the updated ocis-hello
|
||||
|
||||
### Option 1:
|
||||
get a list of ocis services:
|
||||
|
||||
```console
|
||||
ps ax | grep ocis
|
||||
```
|
||||
|
||||
Try to kill `ocis hello`
|
||||
|
||||
Remember: for now, killing a service will cause ocis to restart it. This is subject to change.
|
||||
|
||||
In order to be able to manage the processes ourselves we need to start them independently:
|
||||
|
||||
`bin/ocis server` starts the same services as:
|
||||
|
||||
```
|
||||
bin/ocis micro &
|
||||
bin/ocis phoenix &
|
||||
bin/ocis hello &
|
||||
bin/ocis reva &
|
||||
```
|
||||
|
||||
Now we can kill the `ocis hello` and use our custom built ocis-hello binary:
|
||||
|
||||
```console
|
||||
cd ../ocis-hello
|
||||
bin/ocis-hello server
|
||||
```
|
||||
|
||||
## Hacking phoenix (and ocis-phoenix)
|
||||
|
||||
Following https://github.com/owncloud/phoenix we are going to build the current phoenix
|
||||
|
||||
```
|
||||
git clone https://github.com/owncloud/phoenix.git
|
||||
cd phoenix
|
||||
|
||||
yarn install
|
||||
yarn dist
|
||||
```
|
||||
|
||||
We can tell ocis to use the compiled assets:
|
||||
|
||||
Kill `ocis phoenix`, then use the compiled assets when starting phoenix.
|
||||
|
||||
```console
|
||||
cd ../ocis
|
||||
PHOENIX_ASSET_PATH="`pwd`/../phoenix/dist" bin/ocis phoenix
|
||||
```
|
||||
|
||||
## The ownCloud design system
|
||||
|
||||
The owncloud design system contains a set of ownCloud vue components for phoenix or your own ocis extensions. Use it for a consistent look and feel.
|
||||
|
||||
Point your browser to https://owncloud.github.io/owncloud-design-system and check the available components. Live editing the examples in the browser is supported.
|
||||
|
||||
note: There is a bug with navigation sub items: either click a nav item twice or refresh the page
|
||||
|
||||
## External phoenix apps
|
||||
|
||||
This is what hello is: copy and extend!
|
||||
|
||||
1. Phoenix is configured using the config.json which is served by the phoenix service (either `bin/ocis phoenix` or `bin/ocis-phoenix server`)
|
||||
|
||||
2. point ocis phoenix to the web config which you extended with an external app:
|
||||
`PHOENIX_WEB_CONFIG="`pwd`/../phoenix/config.json" PHOENIX_ASSET_PATH="`pwd`/../phoenix/dist" bin/ocis phoenix`
|
||||
|
||||
```json
|
||||
{
|
||||
"server": "http://localhost:9140",
|
||||
"theme": "owncloud",
|
||||
"version": "0.1.0",
|
||||
"openIdConnect": {
|
||||
"metadata_url": "http://localhost:9140/.well-known/openid-configuration",
|
||||
"authority": "http://localhost:9140",
|
||||
"client_id": "phoenix",
|
||||
"response_type": "code",
|
||||
"scope": "openid profile email"
|
||||
},
|
||||
"apps": [],
|
||||
"external_apps": [
|
||||
{
|
||||
"id": "hello",
|
||||
"path": "http://localhost:9105/hello.js",
|
||||
"config": {
|
||||
"url": "http://localhost:9105"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "myapp",
|
||||
"path": "http://localhost:6789/superapp.js",
|
||||
"config": {
|
||||
"backend": "http://someserver:1234",
|
||||
"myconfig": "is awesome"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Phoenix extension points
|
||||
|
||||
{{< hint info >}}
|
||||
For an up to date list check out [the phoenix documentation](https://github.com/owncloud/phoenix/issues/2423).
|
||||
{{< /hint >}}
|
||||
|
||||
Several ones available:
|
||||
|
||||
### Phoenix core
|
||||
- App switcher (defined in config.json)
|
||||
- App container (loads UI of your extension)
|
||||
|
||||
### Files app
|
||||
- File action
|
||||
- Create new file action
|
||||
- Sidebar
|
||||
- Quick access for sidebar inside of file actions (in the file row)
|
||||
|
||||
Example of a file action in the `app.js`:
|
||||
```js
|
||||
const appInfo = {
|
||||
name: 'MarkdownEditor',
|
||||
id: 'markdown-editor',
|
||||
icon: 'text',
|
||||
isFileEditor: true,
|
||||
extensions: [{
|
||||
extension: 'txt',
|
||||
newFileMenu: {
|
||||
menuTitle ($gettext) {
|
||||
return $gettext('Create new plain text file…')
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
extension: 'md',
|
||||
newFileMenu: {
|
||||
menuTitle ($gettext) {
|
||||
return $gettext('Create new mark-down file…')
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
For the side bar have a look at the files app, `defaults.js` & `fileSideBars`
|
||||
|
||||
## API driven development
|
||||
|
||||
Until now we only had a look at the ui and how the extensions are managed on the cli. But how do apps actually talk to the server?
|
||||
|
||||
Short answer: any way you like
|
||||
|
||||
Long answer: micro and ocis-hello follow a protocol driven development:
|
||||
|
||||
- specify the API using protobuf
|
||||
- generate client and server code
|
||||
- evolve based on the protocol
|
||||
|
||||
- CS3 api uses protobuf as well and uses GRPC
|
||||
|
||||
- ocis uses go-micro, which provides http and grpc gateways
|
||||
- the gateways and protocols are optional
|
||||
|
||||
- owncloud and kopano are looking into a [MS graph](https://developer.microsoft.com/de-de/graph) like api to handle phoenix requests.
|
||||
- they might be about user, contacrs, calendars ... which is covered by the graph api
|
||||
- we want to integrate with eg. kopano and provide a commen api (file sync and share is covered as well)
|
||||
|
||||
- as an example for protobuf take a look at [ocis-hello](https://github.com/owncloud/ocis-hello/tree/master/pkg/proto/v0)
|
||||
@@ -1,228 +0,0 @@
|
||||
---
|
||||
title: "Getting Started"
|
||||
date: 2020-02-27T20:35:00+01:00
|
||||
weight: 15
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: getting-started.md
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Installation
|
||||
|
||||
So far we are offering two different variants for the installation. You can choose between [Docker](https://www.docker.com/) or pre-built binaries which are stored on our download mirrors and GitHub releases. Maybe we will also provide system packages for the major distributions later if we see the need for it.
|
||||
|
||||
### Docker
|
||||
|
||||
Docker images for ocis are hosted on https://hub.docker.com/r/owncloud/ocis.
|
||||
|
||||
The `latest` tag always reflects the current master branch.
|
||||
|
||||
```console
|
||||
docker pull owncloud/ocis
|
||||
```
|
||||
|
||||
#### Dependencies
|
||||
|
||||
- Running ocis currently needs a working Redis caching server
|
||||
- The default storage location in the container is `/var/tmp/reva/data`. You may want to create a volume to persist the files in the primary storage
|
||||
|
||||
#### Docker compose
|
||||
|
||||
You can use our docker-compose [playground example](https://github.com/owncloud-docker/compose-playground/tree/master/ocis) to run ocis with dependencies with a single command in a docker network.
|
||||
|
||||
```console
|
||||
git clone git@github.com:owncloud-docker/compose-playground.git
|
||||
cd compose-playground/ocis
|
||||
docker-compose -f ocis.yml -f ../cache/redis-ocis.yml up
|
||||
```
|
||||
|
||||
### Binaries
|
||||
|
||||
The pre-built binaries for different platforms are downloadable at https://download.owncloud.com/ocis/ocis/ . Specific releases are organized in separate folders. They are in sync which every release tag on GitHub. The binaries from the current master branch can be found in https://download.owncloud.com/ocis/ocis/testing/
|
||||
|
||||
```console
|
||||
curl https://download.owncloud.com/ocis/ocis/1.0.0-beta1/ocis-1.0.0-beta1-darwin-amd64 --output ocis
|
||||
chmod +x ocis
|
||||
./ocis server
|
||||
```
|
||||
|
||||
#### Dependencies
|
||||
|
||||
- Running ocis currently needs a working Redis caching server
|
||||
- The default promary storage location is `/var/tmp/reva/data`. You can change that value by configuration.
|
||||
|
||||
## Usage
|
||||
|
||||
The program provides a few sub-commands on execution. The available configuration methods have already been mentioned above. Generally you can always see a formated help output if you execute the binary via `ocis --help`.
|
||||
|
||||
### Server
|
||||
|
||||
The server command is used to start the http and debug server on two addresses within a single process. The http server is serving the general webservice while the debug server is used for health check, readiness check and to server the metrics mentioned below. For further help please execute:
|
||||
|
||||
{{< highlight txt >}}
|
||||
ocis server --help
|
||||
{{< / highlight >}}
|
||||
|
||||
### Health
|
||||
|
||||
The health command is used to execute a health check, if the exit code equals zero the service should be up and running, if the exist code is greater than zero the service is not in a healthy state. Generally this command is used within our Docker containers, it could also be used within Kubernetes.
|
||||
|
||||
{{< highlight txt >}}
|
||||
ocis health --help
|
||||
{{< / highlight >}}
|
||||
|
||||
## Quickstart for Developers
|
||||
|
||||
Following https://github.com/owncloud/ocis#development
|
||||
|
||||
```console
|
||||
git clone https://github.com/owncloud/ocis.git
|
||||
cd ocis
|
||||
make generate build
|
||||
```
|
||||
|
||||
Open https://localhost:9200 and login using one of the demo accounts:
|
||||
|
||||
```console
|
||||
einstein:relativity
|
||||
marie:radioactivity
|
||||
feynman:superfluidity
|
||||
```
|
||||
|
||||
## Runtime
|
||||
|
||||
Included with the ocis binary is embedded a go-micro runtime that is in charge of starting services as a fork of the master process. This provides complete control over the services. Ocis extensions can be added as part of this runtime.
|
||||
|
||||
```console
|
||||
./bin/ocis micro
|
||||
```
|
||||
|
||||
This will currently boot:
|
||||
|
||||
```console
|
||||
com.owncloud.api
|
||||
com.owncloud.http.broker
|
||||
com.owncloud.proxy
|
||||
com.owncloud.registry
|
||||
com.owncloud.router
|
||||
com.owncloud.runtime
|
||||
com.owncloud.web
|
||||
go.micro.http.broker
|
||||
```
|
||||
|
||||
Further ocis extensions can be added to the runtime via the ocis command like:
|
||||
|
||||
```console
|
||||
./bin/ocis hello
|
||||
```
|
||||
|
||||
Which will register:
|
||||
|
||||
```console
|
||||
com.owncloud.web.hello
|
||||
com.owncloud.api.hello
|
||||
```
|
||||
|
||||
To the list of available services.
|
||||
|
||||
## Metrics
|
||||
|
||||
This service provides some [Prometheus](https://prometheus.io/) metrics through the debug endpoint, you can optionally secure the metrics endpoint by some random token, which got to be configured through one of the flag `--debug-token` or the environment variable `OCIS_DEBUG_TOKEN` mentioned above. By default the metrics endpoint is bound to `http://0.0.0.0:8001/metrics`.
|
||||
|
||||
go_gc_duration_seconds
|
||||
: A summary of the GC invocation durations
|
||||
|
||||
go_gc_duration_seconds_sum
|
||||
: A summary of the GC invocation durations
|
||||
|
||||
go_gc_duration_seconds_count
|
||||
: A summary of the GC invocation durations
|
||||
|
||||
go_goroutines
|
||||
: Number of goroutines that currently exist
|
||||
|
||||
go_info
|
||||
: Information about the Go environment
|
||||
|
||||
go_memstats_alloc_bytes
|
||||
: Number of bytes allocated and still in use
|
||||
|
||||
go_memstats_alloc_bytes_total
|
||||
: Total number of bytes allocated, even if freed
|
||||
|
||||
go_memstats_buck_hash_sys_bytes
|
||||
: Number of bytes used by the profiling bucket hash table
|
||||
|
||||
go_memstats_frees_total
|
||||
: Total number of frees
|
||||
|
||||
go_memstats_gc_cpu_fraction
|
||||
: The fraction of this program's available CPU time used by the GC since the program started
|
||||
|
||||
go_memstats_gc_sys_bytes
|
||||
: Number of bytes used for garbage collection system metadata
|
||||
|
||||
go_memstats_heap_alloc_bytes
|
||||
: Number of heap bytes allocated and still in use
|
||||
|
||||
go_memstats_heap_idle_bytes
|
||||
: Number of heap bytes waiting to be used
|
||||
|
||||
go_memstats_heap_inuse_bytes
|
||||
: Number of heap bytes that are in use
|
||||
|
||||
go_memstats_heap_objects
|
||||
: Number of allocated objects
|
||||
|
||||
go_memstats_heap_released_bytes
|
||||
: Number of heap bytes released to OS
|
||||
|
||||
go_memstats_heap_sys_bytes
|
||||
: Number of heap bytes obtained from system
|
||||
|
||||
go_memstats_last_gc_time_seconds
|
||||
: Number of seconds since 1970 of last garbage collection
|
||||
|
||||
go_memstats_lookups_total
|
||||
: Total number of pointer lookups
|
||||
|
||||
go_memstats_mallocs_total
|
||||
: Total number of mallocs
|
||||
|
||||
go_memstats_mcache_inuse_bytes
|
||||
: Number of bytes in use by mcache structures
|
||||
|
||||
go_memstats_mcache_sys_bytes
|
||||
: Number of bytes used for mcache structures obtained from system
|
||||
|
||||
go_memstats_mspan_inuse_bytes
|
||||
: Number of bytes in use by mspan structures
|
||||
|
||||
go_memstats_mspan_sys_bytes
|
||||
: Number of bytes used for mspan structures obtained from system
|
||||
|
||||
go_memstats_next_gc_bytes
|
||||
: Number of heap bytes when next garbage collection will take place
|
||||
|
||||
go_memstats_other_sys_bytes
|
||||
: Number of bytes used for other system allocations
|
||||
|
||||
go_memstats_stack_inuse_bytes
|
||||
: Number of bytes in use by the stack allocator
|
||||
|
||||
go_memstats_stack_sys_bytes
|
||||
: Number of bytes obtained from system for stack allocator
|
||||
|
||||
go_memstats_sys_bytes
|
||||
: Number of bytes obtained from system
|
||||
|
||||
go_threads
|
||||
: Number of OS threads created
|
||||
|
||||
promhttp_metric_handler_requests_in_flight
|
||||
: Current number of scrapes being served
|
||||
|
||||
promhttp_metric_handler_requests_total
|
||||
: Total number of scrapes by HTTP status code
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
title: "License"
|
||||
date: 2020-02-27T20:35:00+01:00
|
||||
weight: 100
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: license.md
|
||||
---
|
||||
|
||||
This project is licensed under the [Apache 2.0](https://github.com/owncloud/ocis/blob/master/LICENSE) license. For the license of the used libraries you have to check the respective sources.
|
||||
@@ -1,85 +0,0 @@
|
||||
---
|
||||
title: "Login Flow"
|
||||
date: 2020-05-04T20:47:00+01:00
|
||||
weight: 43
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: login-flow.md
|
||||
---
|
||||
|
||||
|
||||
## Login Flow
|
||||
|
||||
The following sequence diagram describes the [openid connect auth code flow](https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth). The eight numbered steps and notes correspond to the [openid connect auth code flow steps](https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowSteps). Example requests are based on the spec as well.:
|
||||
|
||||
{{< mermaid class="text-center">}}
|
||||
sequenceDiagram
|
||||
%% we have comments!! \o/
|
||||
%% this documents the login workflow
|
||||
%% examples taken from the oidc spec https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth
|
||||
%% TODO add PKCE, see https://developer.okta.com/blog/2019/08/22/okta-authjs-pkce#use-pkce-to-make-your-apps-more-secure
|
||||
participant user as User
|
||||
participant client as Client
|
||||
participant proxy as ocis-proxy
|
||||
participant idp as IdP
|
||||
participant glauth as ocis-glauth
|
||||
participant graph as ocis-graph
|
||||
participant accounts as ocis-accounts
|
||||
participant ldap as external LDAP server
|
||||
|
||||
user->>+client: What is the content of my home?
|
||||
|
||||
client->>+proxy: PROPFIND <br> no (or expired) auth
|
||||
Note over client,proxy: ocis needs to know the IdP that is<br>used to authenticate users. The<br>proxy will redirect unauthenticated<br>requests to that IdP.
|
||||
proxy-->>-client: 302 Found
|
||||
Note over client, idp: HTTP/1.1 302 Found<br>Location: https://server.example.com/authorize?<br>response_type=code&<br>scope=openid%20profile%20email<br>&client_id=s6BhdRkqt3<br>&state=af0ifjsldkj<br>&redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb
|
||||
|
||||
Note over client, idp: We should follow the OpenID Connect Discovery protocol
|
||||
Note over client, idp: Clients might fall back to the ocis server if the discovery failed.<br>We can provide a webfinger endpoint there to let guests use an idp<br>that is backed by the accounts service.
|
||||
Note over client, idp: For now, clients can only handle one IdP, which is configured in ocis.
|
||||
|
||||
client-->>client: 1. Client prepares an Authentication Request<br>containing the desired request parameters.
|
||||
|
||||
client->>+idp: 2. Client sends the request to the Authorization Server.
|
||||
Note over client, idp: GET /authorize?<br>response_type=code<br>&scope=openid%20profile%20email<br>&client_id=s6BhdRkqt3<br>&state=af0ifjsldkj<br>&redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb HTTP/1.1<br>Host: server.example.com
|
||||
Note over user, idp: 3. Authorization Server Authenticates the End-User.
|
||||
Note over idp,ldap: Either an IdP already exists or a new one is introduced. Since we are not yet using oidc discovery we can only use one IdP.
|
||||
alt all users managed by konnectd/ocis
|
||||
idp->>+glauth: LDAP query/bind
|
||||
glauth->>+graph: GET user with Basic Auth<br>GraphAPI
|
||||
graph->>+accounts: internal GRPC
|
||||
accounts-->>-graph: response
|
||||
graph-->>-glauth: OData response
|
||||
glauth-->>-idp: LDAP result
|
||||
Note over accounts,ldap: In case internal users are managed<br>in an external ldap they have to be<br>synced to the accounts service to<br>show up as recipients during sharing.
|
||||
else all users authenticated by an external idp
|
||||
idp->>+ldap: LDAP query/bind
|
||||
ldap-->>-idp: LDAP result
|
||||
alt guest accounts managed in ocis / lookup using glauth proxy:
|
||||
Note over idp,glauth: Idp is configured to use glauth as a<br>second ldap server.
|
||||
idp->>+glauth: LDAP query/bind
|
||||
glauth->>+graph: GET user with Basic Auth<br>GraphAPI
|
||||
graph->>+accounts: internal GRPC
|
||||
accounts-->>-graph: response
|
||||
graph-->>-glauth: OData response
|
||||
glauth-->>-idp: LDAP result
|
||||
else guest account provisioned by other means
|
||||
Note over accounts, ldap: In case guest accounts are managed<br>in an existing ldap they need to be<br>synced to the accounts service to<br>be able to login and show up as<br>recipients during sharing.
|
||||
end
|
||||
end
|
||||
Note over user, idp: 4. Authorization Server obtains End-User Consent/Authorization.
|
||||
idp-->>-client: 5. Authorization Server sends the End-User back<br>to the Client with an Authorization Code.
|
||||
Note over client, idp: HTTP/1.1 302 Found<br>Location: https://client.example.org/cb?<br>code=SplxlOBeZQQYbYS6WxSbIA&state=af0ifjsldkj
|
||||
|
||||
client->>+idp: 6. Client requests a response using the<br>Authorization Code at the Token Endpoint.
|
||||
Note over client, idp: POST /token HTTP/1.1<br>Host: server.example.com<br>Content-Type: application/x-www-form-urlencoded<br>grant_type=authorization_code&code=SplxlOBeZQQYbYS6WxSbIA<br>&redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb
|
||||
idp-->>-client: 7. Client receives a response that contains an<br>ID Token and Access Token in the response body.
|
||||
Note over client, idp: HTTP/1.1 200 OK<br>Content-Type: application/json<br>Cache-Control: no-store<br>Pragma: no-cache<br>{<br>"access_token": "SlAV32hkKG",<br>"token_type": "Bearer",<br>"refresh_token": "8xLOxBtZp8",<br>"expires_in": 3600,<br>"id_token": "a ... b.c ... d.e ... f" // must be a JWT<br>}
|
||||
|
||||
|
||||
client-->>client: 8. Client validates the ID token and<br>retrieves the End-User's Subject Identifier.
|
||||
|
||||
client->>+proxy: PROPFIND <br> With access token
|
||||
proxy-->>-client: 207 Multi-Status
|
||||
client-->>-user: List of Files X, Y, Z ...
|
||||
{{< /mermaid >}}
|
||||
@@ -1,15 +0,0 @@
|
||||
---
|
||||
title: "Public upload Flow"
|
||||
date: 2020-07-27T14:16:00+01:00
|
||||
weight: 47
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: public-upload-flow.md
|
||||
---
|
||||
|
||||
|
||||
## Public Upload flow
|
||||
|
||||
The following diagram describes the flow of requests:
|
||||
|
||||
{{< svg src="static/ocis/tus-public-upload.svg" >}}
|
||||
@@ -1,96 +0,0 @@
|
||||
---
|
||||
title: "Request Flow"
|
||||
date: 2020-04-27T16:07:00+01:00
|
||||
weight: 45
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: request-flow.md
|
||||
---
|
||||
|
||||
|
||||
## Request Flow
|
||||
|
||||
The following sequence diagram describes the general request flow. It shows where account provisioning and token minting are happening:
|
||||
|
||||
{{< mermaid class="text-center">}}
|
||||
sequenceDiagram
|
||||
%% we have comments!! \o/
|
||||
participant user as User
|
||||
participant client as Client
|
||||
participant proxy as ocis-proxy
|
||||
participant idp as IdP
|
||||
participant accounts as ocis-accounts
|
||||
participant ldap as corporate LDAP server
|
||||
|
||||
user->>+client: What is the content of my home?
|
||||
|
||||
client->>+proxy: PROPFIND <br> Bearer auth using oidc auth token
|
||||
Note over client,proxy: What is in a bearer token? <br> The spec recommends opaque tokens. <br> Treat it as random byte noise.
|
||||
Note over client,proxy: the proxy MUST authenticate users <br> using ocis-accounts because it needs <br> to decide where to send the request
|
||||
%% Mention introspection endpoint for opaque tokens
|
||||
%% konnectd uses jwt, so we can save a request
|
||||
%% either way the token can be used to look up the sub and iss of the user
|
||||
|
||||
%% or is token check enough?
|
||||
proxy->>+idp: GET /userinfo
|
||||
alt userinfo succeeds
|
||||
|
||||
idp-->>proxy: 200 OK
|
||||
Note over proxy,accounts: Content-Type: application/json<br>{<br>"sub": "248289761001",<br>"name": "Jane Doe",<br>"given_name": "Jane",<br>"family_name": "Doe",<br>"preferred_username": "j.doe",<br>"email": "janedoe@example.com",<br>"picture": "http://example.com/janedoe/me.jpg"<br>}
|
||||
%% see: https://openid.net/specs/openid-connect-core-1_0.html#UserInfoResponse
|
||||
|
||||
else userinfo fails
|
||||
|
||||
idp-->>-proxy: 401 Unauthorized
|
||||
Note over proxy,accounts: WWW-Authenticate: error="invalid_token",<br>error_description="The Access Token expired"
|
||||
|
||||
proxy-->>client: 401 Unauthorized or <br>302 Found with redirect to idp
|
||||
Note over client: start at login flow<br> or refresh the token
|
||||
|
||||
end
|
||||
|
||||
proxy->>+accounts: TODO API call to exchange sub@iss with account UUID
|
||||
Note over proxy,accounts: does not autoprovision users. They are explicitly provsioned later.
|
||||
|
||||
alt account exists or has been migrated
|
||||
|
||||
accounts-->>proxy: existing account UUID
|
||||
else account does not exist
|
||||
|
||||
opt oc10 endpoint is configured
|
||||
Note over proxy,oc10: Check if user exists in oc10
|
||||
proxy->>+oc10: GET /apps/graphapi/v1.0/users/<uuid>
|
||||
opt user exists in oc10
|
||||
oc10-->>-proxy: 200
|
||||
%% TODO auth using internal token
|
||||
proxy->>+oc10: PROPFIND
|
||||
Note over proxy,oc10: forward existing bearer auth
|
||||
oc10-->>-proxy: Multistatus response
|
||||
proxy-->>client: Multistatus response
|
||||
client-->>user: List of Files X, Y, Z ...
|
||||
end
|
||||
end
|
||||
|
||||
Note over proxy,accounts: provision a new account including displayname, email and sub@iss <br> TODO only if the user is allowed to login, based on group <br> membership in the ldap server
|
||||
proxy->>proxy: generate new uuid
|
||||
proxy->>+accounts: TODO create account with new generated uuid
|
||||
accounts-->>-proxy: OK / error
|
||||
|
||||
else account has been disabled
|
||||
|
||||
accounts-->>-proxy: account is disabled
|
||||
proxy-->>client: 401 Unauthorized or <br>302 Found with redirect to idp
|
||||
Note over client: start at login flow<br> or refresh the token
|
||||
|
||||
end
|
||||
proxy->>proxy: store uuid in context
|
||||
|
||||
%% what if oc10 does not support a certain request / API
|
||||
|
||||
proxy->>proxy: mint an internal jwt that includes the UUID and username using revas `x-access-token` header
|
||||
proxy->>+reva: PROPFIND <br>Token auth using internal JWT
|
||||
reva-->>-proxy: Multistatus response
|
||||
proxy-->>-client: Multistatus response
|
||||
|
||||
client-->>-user: List of Files X, Y, Z ...
|
||||
{{< /mermaid >}}
|
||||
-1
File diff suppressed because one or more lines are too long
Vendored
-3
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 66 KiB |
-107
@@ -1,107 +0,0 @@
|
||||
---
|
||||
title: "Testing"
|
||||
date: 2018-05-02T00:00:00+00:00
|
||||
weight: 37
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: testing.md
|
||||
---
|
||||
|
||||
|
||||
## Acceptance tests
|
||||
|
||||
We are using the ownCloud 10 acceptance testsuite against ocis. To set this up you need the owncloud 10 core repo, a ldap server that the acceptance tests can use to manage users, a redis server for file-versions and the ocis code.
|
||||
|
||||
### Getting the tests
|
||||
|
||||
All you need to do to get the acceptance tests is check out the core repo:
|
||||
```
|
||||
git clone https://github.com/owncloud/core.git
|
||||
```
|
||||
|
||||
### Run a redis server in a docker container
|
||||
|
||||
File versions need a redis server. Start one with docker by using:
|
||||
|
||||
`docker run -e REDIS_DATABASES=1 -p 6379:6379 -d webhippie/redis:latest`
|
||||
|
||||
### Run ocis
|
||||
|
||||
To start ocis:
|
||||
```
|
||||
bin/ocis server
|
||||
```
|
||||
|
||||
### Run the acceptance tests
|
||||
First we will need to clone the testing app in owncloud which contains the skeleton files required for running the tests.
|
||||
In the ownCloud 10 core clone the testing app with the following command:
|
||||
|
||||
```
|
||||
git clone https://github.com/owncloud/testing apps/testing
|
||||
```
|
||||
|
||||
Then run the api acceptance tests with the following command:
|
||||
```
|
||||
make test-acceptance-api \
|
||||
TEST_SERVER_URL=https://localhost:9200 \
|
||||
TEST_OCIS=true \
|
||||
OCIS_REVA_DATA_ROOT=/var/tmp/reva/ \
|
||||
SKELETON_DIR=apps/testing/data/apiSkeleton \
|
||||
BEHAT_FILTER_TAGS='~@notToImplementOnOCIS&&~@toImplementOnOCIS'
|
||||
```
|
||||
|
||||
Make sure to adjust the settings `TEST_SERVER_URL` and `OCIS_REVA_DATA_ROOT` according to your environment.
|
||||
|
||||
This will run all tests that are relevant to OCIS.
|
||||
|
||||
To run a single test add `BEHAT_FEATURE=<feature file>`
|
||||
|
||||
### use existing tests for BDD
|
||||
|
||||
As a lot of scenarios are written for oC10, we can use those tests for Behaviour driven development in ocis.
|
||||
Every scenario that does not work in OCIS with OC storage, is listed in `tests/acceptance/expected-failures-on-OC-storage.txt` with a link to the related issue.
|
||||
|
||||
Those scenarios are run in the ordinary acceptance test pipeline in CI. The scenarios that fail are checked against the
|
||||
expected failures. If there are any differences then the CI pipeline fails.
|
||||
Similarly, scenarios that do not work in OCIS with EOS storage are listed in `tests/acceptance/expected-failures-on-EOS-storage.txt`.
|
||||
Additionally, some issues have scenarios that demonstrate the current buggy behaviour in ocis(reva).
|
||||
Those scenarios are in this ocis repository in `tests/acceptance/features/apiOcisSpecific`.
|
||||
Have a look into the [documentation](https://doc.owncloud.com/server/developer_manual/testing/acceptance-tests.html#writing-scenarios-for-bugs) to understand why we are writing those tests.
|
||||
|
||||
If you want to work on a specific issue
|
||||
|
||||
1. adjust the core commit id to the latest commit in core so that CI will run the latest test code and scenarios from core.
|
||||
For that change `coreCommit` in the `config` section:
|
||||
|
||||
config = {
|
||||
'apiTests': {
|
||||
'coreBranch': 'master',
|
||||
'coreCommit': 'a06b1bd5ba8e5244bfaf7fa04f441961e6fb0daa',
|
||||
'numberOfParts': 2
|
||||
}
|
||||
}
|
||||
|
||||
2. locally run each of the tests marked with that issue in the expected failures file
|
||||
|
||||
E.g.:
|
||||
```
|
||||
make test-acceptance-api \
|
||||
TEST_SERVER_URL=https://localhost:9200 \
|
||||
TEST_OCIS=true \
|
||||
OCIS_REVA_DATA_ROOT=/var/tmp/reva/ \
|
||||
BEHAT_FEATURE='tests/acceptance/features/apiComments/comments.feature:123'
|
||||
```
|
||||
|
||||
3. the tests will fail, try to understand how and why they are failing
|
||||
4. fix the code
|
||||
5. go back to 2. and repeat till the tests are passing.
|
||||
6. remove those tests from the expected failures file
|
||||
7. run each of the local tests that were demonstrating the **buggy** behavior. They should fail.
|
||||
8. delete each of the local tests that were demonstrating the **buggy** behavior.
|
||||
9. make a PR that has the fixed code, relevant lines removed from the expected failures file and bug demonstration tests deleted.
|
||||
|
||||
If the changes also affect the `ocis-reva` repository make sure the changes get ported over there.
|
||||
|
||||
### Notes
|
||||
- in a normal case the test-code cleans up users after the test-run, but if a test-run is interrupted (e.g. by CTRL+C) users might have been left on the LDAP server. In that case rerunning the tests requires wiping the users in the ldap server, otherwise the tests will fail when trying to populate the users.
|
||||
- the tests usually create users in the OU `TestUsers` with usernames specified in the feature file. If not defined in the feature file, most users have the password `123456`, defined by `regularUserPassword` in `behat.yml`, but other passwords are also used, see [`\FeatureContext::getPasswordForUser()`](https://github.com/owncloud/core/blob/master/tests/acceptance/features/bootstrap/FeatureContext.php#L386) for mapping and [`\FeatureContext::__construct`](https://github.com/owncloud/core/blob/master/tests/acceptance/features/bootstrap/FeatureContext.php#L1668) for the password definitions.
|
||||
@@ -1,48 +0,0 @@
|
||||
---
|
||||
title: "Tracing"
|
||||
date: 2020-05-13T12:09:00+01:00
|
||||
weight: 55
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs
|
||||
geekdocFilePath: tracing.md
|
||||
---
|
||||
|
||||
By default, we use [Jaeger](https://www.jaegertracing.io) for request tracing within oCIS. You can follow these steps
|
||||
to get started:
|
||||
|
||||
1. Start Jaeger by using the all-in-one docker image:
|
||||
```console
|
||||
docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
||||
-p 5775:5775/udp \
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
-p 14250:14250 \
|
||||
-p 9411:9411 \
|
||||
jaegertracing/all-in-one:1.17
|
||||
```
|
||||
2. Every single oCIS service has its own environment variables for enabling and configuring tracing.
|
||||
1. You can enable and configure tracing on each service individually. For example, enable tracing
|
||||
in Reva when starting the oCIS single binary like this:
|
||||
```console
|
||||
REVA_TRACING_ENABLED=true \
|
||||
REVA_TRACING_ENDPOINT=localhost:6831 \
|
||||
REVA_TRACING_COLLECTOR=http://localhost:14268/api/traces \
|
||||
./bin/ocis server
|
||||
```
|
||||
2. Enabling and configuring tracing on oCIS itself will forward the configuration to all services:
|
||||
```console
|
||||
OCIS_TRACING_ENABLED=true \
|
||||
OCIS_TRACING_ENDPOINT=localhost:6831 \
|
||||
OCIS_TRACING_COLLECTOR=http://localhost:14268/api/traces \
|
||||
./bin/ocis server
|
||||
```
|
||||
If you want to set individual tracing configuration for each service, make sure to set
|
||||
`OCIS_TRACING_ENABLED=false`.
|
||||
3. Make the actual request that you want to trace.
|
||||
4. Open up the [Jaeger UI](http://localhost:16686) to analyze request traces.
|
||||
|
||||
For more information on Jaeger, please refer to their [Documentation](https://www.jaegertracing.io/docs/1.17/).
|
||||
Reference in New Issue
Block a user