Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,5 @@ website/pages/tre_agent/deploy.mdx
website/pages/submission/guides/submissionManagerAddingNewPerson.mdx
website/pages/five_safes_tes/reference_implementation/quickstart.mdx
website/pages/submission/guides/teswizard.mdx
website/pages/five_safes_tes/reference_implementation/overview.mdx
website/pages/five_safes_tes/reference_implementation/overview.mdx
website/pages/five_safes_tes/reference_implementation/TES/install_funnel.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,34 +4,39 @@ import { Steps, Callout, Tabs } from "nextra/components";

[Funnel](https://github.com/ohsu-comp-bio/funnel) is an implementation of the GA4GH Task Execution Schemas. It provides an API server,
multiple storage backends, multiple compute backends and a web UI.
For the purposes of the All In One stack we will be configuring the TRE Agent to use Funnel as the TES compatible execution backend and any outputs from the execution will be stored in the TRE Layer MinIO instance.
For the purposes of the DemoStack quickstart we will be configuring the TRE Agent to use Funnel as the TES compatible execution backend and any outputs from the execution will be first stored in the TRE Layer S3 instance and then copied over to the Submission Layer S3 instance.

<Steps>

## Set up MinIO Access key
## Set up S3 Access key

<Callout>
This guidance refers to the preset credentials for the All In One quickstart.
This guidance refers to the preset credentials for the DemoStack quickstart.
If you are setting up Funnel for a regular deployment, you'll need to use your
TRE Agent realm credentials.
</Callout>

Navigate to the MinIO instance of the TRE Layer. If you are using our deployment compose files as a starting point, it will be on the same host as the TRE Agent, at port 9003. If you've configured it to be different, you'll need to use the correct port.
- Navigate to the S3 instance of the TRE Layer. If you are using Demostack Quickstart, it will be on the same host as the TRE Agent, at port 9002. If you've configured it to be different, you'll need to use the correct port.

You can login with `Login with KeyCloak` option using the following credentials using your TRE admin credentials.
- Login with S3 TRE root credentials (which are set up in the environment variables in the `.env` file).

If you are using DemoStack, the TRE admin credentials are:
<Callout type="info">
If you are using Demostack Quickstart, the S3 TRE root credentials are:
```yaml copy
Username: s3-tre
Password: s3-tre-pass
```
</Callout>

```yaml copy
Username: globaladminuser
Password: password123
```
- To create the access keys, in the S3 (currently RustFS) console navigate to **_Access Keys -> Add Access Key_**

<Callout>
While you can log in with your KeyCloak credentials, using the option `Log in with KeyCloak`, to create access keys with efficient privileges, you need to log in with S3 root credentials.
</Callout>

You can also login with Minio root credentials (which are set up in the environment variables in the `.env` file) using the option `Use Credentials` under `Other Authentication methods`.
![RustFS](/images/S3-RustFS.png)

To create the access keys, in the MinIO console navigate to **_Access Keys -> Create access key_**

![MinIO](/images/MinioAccessKey.png)

## Install Funnel

Expand All @@ -43,10 +48,16 @@ Instructions on how to install Funnel can be found [here](https://ohsu-comp-bio.

<Tabs items={['Install Script', 'Homebrew', 'Manual Binary Install', 'Build from Source']}>
<Tabs.Tab>
This installs the latest release candidate at the time of writing `v0.11.7`.
This installs the latest release candidate at the time of writing `v0.11.12`.
```bash copy
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/ohsu-comp-bio/funnel/refs/heads/develop/install.sh)" -- v0.11.7
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/calypr/funnel/develop/install.sh)" -- "v0.11.12"
```
<Callout type="info">
You can add the destination of Funnel by adding it in the end of the command. For example, to install Funnel to `/opt/funnel`:
```bash copy
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/calypr/funnel/develop/install.sh)" -- "v0.11.12" "/opt/funnel"
```
</Callout>
</Tabs.Tab>

<Tabs.Tab>
Expand Down Expand Up @@ -123,15 +134,15 @@ funnel --help

## Funnel config file

Funnel will need access to the TRE layer MinIO. Once funnel is installed, create a `config.yml` file.
Funnel will need access to the TRE layer S3 storage. Once funnel is installed, create a `config.yml` file.

- `Endpoint` is where the instance of MinIO in the TRE Layer is hosted.
- `Access key` & `Secret` are created in the [previous step](#set-up-minio-access-key).
- `Endpoint` is where the instance of S3 in the TRE Layer is hosted.
- `Access key` & `Secret` are created in the [previous step](#set-up-s3-access-key).

```yaml filename="config.yml" copy
GenericS3:
- Disabled: false
Endpoint: "localhost:9002" # URL to MinIO. Change to match your environment. `9002` is the default port in our deployment samples.
Endpoint: "localhost:9002" # URL to S3 API. Change to match your environment. `9002` is the default port in our deployment samples.
Key: "<access key>"
Secret: "<secret>"
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -143,32 +143,28 @@ Activity logs for the stack.

**Hosted at**: [localhost:5341](http://localhost:5341)

### MinIO
### S3 storage

These are the object stores for the stack.
These are the Amazon S3 compatible object stores for the stack.

#### Submission Minio
#### Submission S3

Submission Layer MinIO stores any inputs and after execution, stores the results for a submission.
Submission Layer S3 stores any inputs and after execution, stores the results for a submission.

**Hosted at**: [localhost:9000](http://localhost:9000)
When using Demostack Quickstart, the Submission's S3 UI is hosted at: [localhost:9000](http://localhost:9000)

```yaml
Username: minio
Password: minio123
Root Username: s3-submission
Root Password: s3-submission-pass
```

#### TRE Minio
#### TRE S3

TRE Agent MinIO, stores any inputs copied over from the Submission layer, and the results of the analysis.
TRE Agent S3, stores any inputs copied over from the Submission layer, and the results of the analysis.

**Hosted at**: [localhost:9002](http://localhost:9002)
When using Demostack Quickstart, the TRE's S3 UI is hosted at: [localhost:9002](http://localhost:9002)

```yaml
Username: minio
Password: minio123
```

```

Root Username: s3-tre
Root Password: s3-tre-pass
```
89 changes: 48 additions & 41 deletions website/pages/submission/deploy.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,31 +12,31 @@ This page contains:

Here is a summary of the required components which make up the Submission Layer:

| Component | Notes |
| ------------------ | :-------------------------------------------------------------------------------------------------- |
| **KeyCloak** | The Submission apps require a Keycloak realm containing users and clients authorised to access them |
| **Minio** | An Amazon S3 compatible Submission Storage service |
| **RabbitMQ** | A message broker for queueing submissions for target TREs |
| **PostgreSQL** | A database for the Submission apps to keep local state |
| **Submission API** | A REST API for Submission Layer functionality, including the GA4GH TES API |
| **Submission GUI** | A Web Frontend for users to log into and interact with the Submission Layer |
| Component | Notes |
| ------------------------ | :-------------------------------------------------------------------------------------------------- |
| **KeyCloak** | The Submission apps require a Keycloak realm containing users and clients authorised to access them |
| **S3 (MinIO or RustFS)** | An Amazon S3 compatible Submission Storage service |
| **RabbitMQ** | A message broker for queueing submissions for target TREs |
| **PostgreSQL** | A database for the Submission apps to keep local state |
| **Submission API** | A REST API for Submission Layer functionality, including the GA4GH TES API |
| **Submission GUI** | A Web Frontend for users to log into and interact with the Submission Layer |

You may choose how to distribute your own deployment of these components, as long as they are able to communicate with each other over the network.

For some components, such as KeyCloak or MinIO, you may wish to use your own existing deployments.
For some components, such as KeyCloak or S3 (MinIO or RustFS), you may wish to use your own existing deployments.

{/* TODO: Keycloak realm setup guide */}

### User accessible components

A subset of the components will need to be accessible by users, outside of the host environent:

| Component | Reason |
| ------------------ | :--------------------------------------------------------------------------------------- |
| **KeyCloak** | So that users can authenticate and admins can manage the Submission KeyCloak Realm |
| **Minio** | So that Researchers and TREs can upload and download Submission data (inputs or outputs) |
| **Submission GUI** | So that users can log into and interact with the Submission Layer |
| **Submission API** | So that Researchers and TREs can interact with the Submission Layer via a REST API |
| Component | Reason |
| ------------------------ | :--------------------------------------------------------------------------------------- |
| **KeyCloak** | So that users can authenticate and admins can manage the Submission KeyCloak Realm |
| **S3 (MinIO or RustFS)** | So that Researchers and TREs can upload and download Submission data (inputs or outputs) |
| **Submission GUI** | So that users can log into and interact with the Submission Layer |
| **Submission API** | So that Researchers and TREs can interact with the Submission Layer via a REST API |

## Sample Deployment Guide

Expand All @@ -55,12 +55,12 @@ It also automatically configures the KeyCloak realm, ready for following our oth

Components which may need to be accessed from outside the host are forwarded to the host on specific ports

| Component | Ports |
| ------------------ | :--------------------------------- |
| **KeyCloak** | HTTP: `8085` |
| **MinIO** | API HTTP: `9000`, GUI HTTP: `9001` |
| **Submission GUI** | HTTP: `7220` |
| **Submission API** | HTTP: `5034` |
| Component | Ports |
| ------------------------ | :--------------------------------- |
| **KeyCloak** | HTTP: `8085` |
| **S3 (MinIO or RustFS)** | API HTTP: `9000`, GUI HTTP: `9001` |
| **Submission GUI** | HTTP: `7220` |
| **Submission API** | HTTP: `5034` |

This sample also includes some additional components useful for diagnostic purposes:

Expand All @@ -76,8 +76,8 @@ To use this sample as a starting point for a production deployment, some [genera
The ports specified above can be used for configuring a reverse proxy for access to the necessary components.

<Callout>
If you are doing reverse proxy for MinIO, the `Websockets Support` should be
enabled for MinIO GUI Proxy host.
If you are doing reverse proxy for S3 (MinIO or RustFS), the `Websockets
Support` should be enabled for S3 GUI Proxy host.
</Callout>

## Get Started
Expand Down Expand Up @@ -124,12 +124,19 @@ The compose manifests used in the Submission Layer are:
- `ServiceStack/compose-manifests/applications/submission-layer.yml` which defines the Submission Layer services, i.e., Submission API, Submission GUI.
- `ServiceStack/compose-manifests/shared/platform.yml` which defines the platform services, i.e., PostgreSQL, Adminer, Serilog, RabbitMQ.
- `ServiceStack/compose-manifests/shared/auth.yml` which defines the authentication services, i.e., Keycloak. This is optional and only needed if you want to use Keycloak for authentication internally within the Submission Layer.
- `ServiceStack/compose-manifests/storage/minio-submission.yml` which defines the Submission Layer's Minio server.

<Callout>
Comment or remove the line
If you don't want to use a built-in Keycloak service, remove the line
`../../ServiceStack/compose-manifests/shared/auth.yml` in the
`docker-compose.yml` file if you don't want to use it.
`docker-compose.yml` file and remove the `keycloak` dependency in the
`s3-submission.yml` file.
</Callout>

- `ServiceStack/compose-manifests/storage/s3-submission.yml` which defines the Submission Layer's S3 server.
<Callout type="info">
If the `Login with KeyCloak` option is not available in the RustFS login portal,
check the health of the KeyCloak server and restart the RustFS
container using the following command: `docker compose restart rustfs_submission`.
</Callout>

##### Configure the .env file
Expand Down Expand Up @@ -211,41 +218,41 @@ The descriptions of the essential environment variables and the guide to set the
</Td>
</Tr>
<Tr>
<Td>`MinioOpenidSecret`</Td>
<Td>`SubmissionS3OpenidSecret`</Td>
<Td>
The OpenID secret for the Minio client of Dare-Control realm. There is a
The OpenID secret for the S3 client of Dare-Control realm. There is a
default value in the realm configuration, but you should regenerate it
for production deployments. Do this by navigating to `Dare-Control`
realm -> `Clients` -> `Dare-Control-Minio` -> `Credentials`. Then click
realm -> `Clients` -> `Dare-Control-S3` -> `Credentials`. Then click
`Regenerate` and copy the new value into this environment variable.
</Td>
</Tr>
<Tr>
<Td>`MinioRootUser`</Td>
<Td>The root user for the Minio server</Td>
<Td>`SubmissionS3RootUser`</Td>
<Td>The root user for the S3 server</Td>
</Tr>
<Tr>
<Td>`MinioRootPass`</Td>
<Td>The root password for the Minio server</Td>
<Td>`SubmissionS3RootPass`</Td>
<Td>The root password for the S3 server</Td>
</Tr>
<Tr>
<Td>`submissionMinioUrl`</Td>
<Td>`SubmissionS3ApiUrl`</Td>
<Td>
The URL for the Submission Layer's Minio server, for internal access within Docker network.
For example, `http://minioSubmission:9000`
The URL for the Submission Layer's S3 server, for internal access within Docker network.
For example, `http://s3Submission:9000`
</Td>
</Tr>
<Tr>
<Td>`submissionMinioAdminConsole`</Td>
<Td>`SubmissionS3ConsoleUrl`</Td>
<Td>
The URL for the Submission Layer's Minio server's admin console. For example,
The URL for the Submission Layer's S3 server's admin console. For example,
`http://localhost:9001`
</Td>
</Tr>
<Tr>
<Td>`SubmissionAPIKeyCloakSecret`</Td>
<Td>
Same as `MinioOpenidSecret`, but this one is for `Dare-Control-API`
Same as `SubmissionS3OpenidSecret`, but this one is for `Dare-Control-API`
client.{" "}
</Td>
</Tr>
Expand All @@ -254,7 +261,7 @@ The descriptions of the essential environment variables and the guide to set the
<Td>`SubmissionUIClientSecret`</Td>
<Td>
{" "}
Same as `MinioOpenidSecret`, but this one is for `Dare-Control-UI` client.{" "}
Same as `SubmissionS3OpenidSecret`, but this one is for `Dare-Control-UI` client.{" "}
</Td>
</Tr>
<Tr>
Expand Down Expand Up @@ -384,7 +391,7 @@ you can access the Submission Layer UI by navigating to `http://<hostname>:7220`
- Submit a Hello World submission to the Submission Layer by following this [guide](/submission/tasks/quickstart).
- Submit a simple analysis submission to the Submission Layer by following this [guide](/submission/tasks/run_analysis).
- Add a new user in the `Dare-Control` Keycloak realm with appropriate permissions, then use this user credentials to access the Submission Layer UI.
- Update passwords or secrets for the Submission Layer, including the Keycloak client secrets, Minio server credentials, etc.
- Update passwords or secrets for the Submission Layer, including the Keycloak client secrets, S3 server credentials, etc.

#### Debugging

Expand Down
2 changes: 1 addition & 1 deletion website/pages/submission/tasks/run_analysis.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ It demonstrates running a basic statistical analysis (mean calculation) on measu
### Information Prerequisites

- Submission Layer endpoint and API Token which can be obtained by this [guide](/submission/guides/get_token).
- Submission Layer MinIO endpoint.
- Submission Layer S3 storage endpoint.
- Database host and credentials for each TRE

## Setup
Expand Down
Loading