Ada Installation Guide (Linux) - Version 0.10.1

(Expected time: 30-45 mins)

0. Preparation

Recommended OS: Ubuntu 16.04/18.04

Recommended resources:

Ada Server: 8 GB RAM, 8 CPUs, 20 GB disc space
Mongo DB: 8 GB RAM, 4 CPUs, 50 GB disc space
Elastic Search DB: 8 GB RAM, 4 CPUs, 50 GB disc space

1. Java 1.8

sudo apt install openjdk-8-jdk

2. Mongo DB

Install MongoDB (4.0.10) (Note that Ada is compatible with any 3.2, 3.4, 3.6, and 4.0 release of Mongo in case you fail to install the recommended version) (Also if you want to use a non-Ubuntu Linux distribution check the supported platforms/OS here)

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org=4.0.10 mongodb-org-server=4.0.10 mongodb-org-shell=4.0.10 mongodb-org-mongos=4.0.10 mongodb-org-tools=4.0.10

Configure memory and other settings in /etc/mongod.conf (set a reasonable cacheSizeGB, recommended to 50% of available RAM, ref)

  ...
  wiredTiger:
    engineConfig:
      cacheSizeGB: 5
      journalCompressor: none
    collectionConfig:
      blockCompressor: snappy

Create a limits file /etc/security/limits.d/mongodb.conf (if you are using Red Hat Enterprise Linux or CentOS read this)

mongodb    soft    nofile          1625538
mongodb    hard    nofile          1625538
mongodb    soft    nopro           64000
mongodb    hard    nopro           64000
mongodb    soft    memlock         unlimited
mongodb    hard    memlock         unlimited
mongodb    soft    fsize           unlimited
mongodb    hard    fsize           unlimited
mongodb    soft    cpu             unlimited
mongodb    hard    cpu             unlimited
mongodb    soft    as              unlimited
mongodb    hard    as              unlimited

To force your new limits to be loaded log out of all your current sessions and log back in.
If your Linux distribution uses systemd to manage services, create the following file /etc/systemd/system/mongod.service.d/ulimit.conf

[Service]

LimitNOFILE=1625538
LimitNPROC=64000
LimitFSIZE=infinity
LimitCPU=infinity
LimitAS=infinity

and apply the settings for systemd

sudo systemctl daemon-reload

Start Mongo

sudo service mongod start

To check if everything works as expected see the log file /var/log/mongodb/mongod.log.
Recommendation: For convenient DB exploration and query execution install Robomongo (Robo3T) UI client.
Optionally set up users with authentication as described here.
For tuning tips go to here.

3. Elastic Search

Install ES (5.6.16)

sudo apt-get update
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.16.deb
sudo dpkg -i elasticsearch-5.6.16.deb
sudo systemctl enable elasticsearch.service

Modify the configuration in /etc/elasticsearch/elasticsearch.yml

  cluster.name: ada-cluster       (if not changed "elasticsearch" is used by default)
  bootstrap.memory_lock: true
  network.host: x.x.x.x           (set to a non-localhost ip address if the db should be accessible within a network)
  thread_pool:
      index:
          queue_size: 8000
      search:
          queue_size: 8000 
      bulk:
          queue_size: 500
  indices.query.bool.max_clause_count: 4096

If you want to store data in a special directory set also

path.data: /your_custom_path

(Note that you need to make your_custom_path writeable for the elasticsearch user)

Create a limits file /etc/security/limits.d/elasticsearch.conf

elasticsearch    soft    nofile          1625538
elasticsearch    hard    nofile          1625538
elasticsearch    soft    memlock         unlimited
elasticsearch    hard    memlock         unlimited

To force your new limits to be loaded log out of all your current sessions and log back in.
Configure open-file and locked memory constraints in /etc/init.d/elasticsearch (and/or /etc/default/elasticsearch)
```
MAX_OPEN_FILES=1625538
MAX_LOCKED_MEMORY=unlimited
```
Depending on the Linux installation configure also /usr/lib/systemd/system/elasticsearch.service
```
LimitNOFILE=1625538
LimitMEMLOCK=infinity
```
Set a reasonable heap size; recommended to 50% of available RAM, but no more than 31g in /etc/elasticsearch/jvm.options, e.g.
```
-Xms5g
-Xmx5g
```
Fix Apache Log4j2 Vulnerability. In /etc/elasticsearch/jvm.options, add property:
```
-Dlog4j2.formatMsgNoLookups=true
```
and finally apply the settings for systemd
```
sudo systemctl daemon-reload
```
Start ES
```
sudo service elasticsearch start
```
To check if everything works as expected see the log file(s) at /var/log/elasticsearch/ and/or curl the server info by curl -XGET localhost:9200.
Recommendation: For convenient DB exploration and query execution you might want to install the cerebro app:

 sudo wget https://github.com/lmenezes/cerebro/releases/download/v0.8.3/cerebro-0.8.3.zip
 sudo unzip cerebro-0.8.3.zip

Open the configuration file conf/application.conf and add the host and name of your Elastic server to the hosts section, e.g.,
```
{
  host = "http://x.x.x.x:9200"
  name = "ada-cluster"
}
```
Run Cerebro, for example at the port 9209
```
 sudo ./bin/cerebro -Dhttp.port=9209
```
(Cerebro web client is then accessible at http://localhost:9209)

Alternatively you can install Kibana, which also allows execution of Elastic queries and many more, as follows (more info here):

echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
sudo apt-get update
sudo apt-get install kibana
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service
sudo service start kibana

4. Application Server (Netty)

Download the version 0.10.1

wget https://webdav-r3lab.uni.lu/public/ada-artifacts/ada-web-0.10.x/ada-web-0.10.1.zip

Unzip the server binaries

sudo apt-get install unzip
unzip ada-web-0.10.1.zip
cd ada-web-0.10.1/bin

Create a temp folder wherever you want, e.g.

mkdir ada_temp

Set the path and memory constraint variables ADA_TEMP and ADA_MEM in runme script (Ada’s bin folder).
Configure DB connections in set_env.sh (bin folder)

1 . Mongo

if not password-protected (default) set ADA_MONGO_DB_URI

export ADA_MONGO_DB_URI="mongodb://localhost:27017/ada?rm.nbChannelsPerNode=20"

if password-protected set the following variables

export ADA_MONGO_DB_HOST=x.x.x.x:27017
export ADA_MONGO_DB_NAME="ada"
export ADA_MONGO_DB_USERNAME="xxx_user"
export ADA_MONGO_DB_PASSWORD="XXX"

2 . Elastic Search

if non-localhost server is used set ADA_ELASTIC_DB_HOST

export ADA_ELASTIC_DB_HOST=x.x.x.x:9200

Also set the custom ES cluster name if it was configured in Section 3 (see /etc/elasticsearch/elasticsearch.yml)

export ADA_ELASTIC_DB_CLUSTER_NAME="ada-cluster"

3 . General Setting

Configure project and datasetimport in custom.conf (Ada conf folder)

project {
  name = "Ultimate"
  url = "https://ada-discovery.github.io"
  logo = "images/logos/ada_logo_v4.png"
}

datasetimport.import.folder = "/custom_path"

Optionally if you want to use external images not shipped by default with Ada you must register your resource folder(s) placed in the Ada root by editing custom.conf

assets.external_paths = ["/folder_in_classpath"]

(Warning: Never set any of the folders to “/” since this would make all the classpath files, including common configuration files, accessible from outside as application assets. An example: https://localhost:8080/assets/application.conf)

Now you can use the logo images placed in your folder_in_classpath by adapting custom.conf for instance as

project {
  ....
  logo = "syscid_large.png"
}

footer.logos = [
  {url: "https://www.aetionomy.eu", logo: "aetionomy.png", height: 150},
  {url: "https://www.efpia.eu", logo: "efpia_logo.png"}
]

(Note an optional height attribute)

Start Ada (Netty)

./runme

(if cannot be executed chmod +x runme might be needed)

To check if everything works as expected see the log file at $ADA_ROOT/bin/logs/application.log
You can now access Ada at

http://localhost:8080

To login as admin go to

http://localhost:8080/loginAdmin

Set your custom Homepage, Contact, and Links in Admin → HTML Snippets → [+]

Homepage example:

<p>
	Ada provides key infrastructure for secured integration, visualization,
	and analysis of heterogeneous clinical and experimental data generated
	during the <a href="http://www.project_url_to_set.com">My TODO project</a>.
</p>
<p>
	My TODO research project focuses on improving ...
</p>
<button type="button" class="btn btn-default btn-sm" data-toggle="collapse" data-target="#more-info">Read More</button>
<div id="more-info" class="collapse">
	<blockquote>
		<h4>The platform currently ...</h4>
	</blockquote>
</div>
<br/>

Contact example:

<strong>Dr. John Snow</strong></br>
Winterfell Team</br>
Westeros Centre For Systems Biomedicine (WCSB)</br>
University of Seven Kingdoms</br></br>
<i class="glyphicon glyphicon-envelope"></i>
<a href="mailto:john.snow@north.edu?Subject=Ada Question">john.snow@north.edu</a><br>
<i class="glyphicon glyphicon-chevron-right"></i>
<a target="_blank" href="http://www.north.edu/wcsb">www.north.edu/wcsb</a><br>

Links example:

<li><a href="https://www.project-redcap.org">RedCap</a></li>
<li><a href="https://www.synapse.org">Synapse</a></li>
<li role="separator" class="divider"></li>
<li><a href="https://uni.lu/lcsb">LCSB Home</a></li>

5. OpenID

Ada is working with OpenID provider. We recommend to use Keycloak and follow related documentation.
For a rapid deployment it’s possible to run Keycloak using docker. Keycloak-postgres template creates a volume for PostgreSQL and starts Keycloak connected to a PostgreSQL instance.
After Keycloak installation and configuration is necessary to set up Ada with following parameters in custom.conf file:

oidc {
  clientId = "ADA_OIDC_CLIENT_ID"
  secret = "ADA_OIDC_SECRET"
  discoveryUrl = "ADA_OIDC_DISCOVERY_URL"
  adaBaseUrl = "ADA_BASE_URL"
  tokenEndPointUrl = "ADA_OIDC_TOKEN_ENDPOINT_URL"
  logoutUrl = "ADA_OIDC_LOGOUT_URL"

  returnAttributeIdName = "sub"
  accessTokenName = "access_token"
  refreshTokenName = "refresh_token"
  rolesAttributeName = "roles"
  realmAccessAttribute = "realm_access"
  resourceAccessAttribute = "resource_access"
  roleAdminName = "ROLE_ADA_ADMIN"
  dataSetGlobalIdPrefix = "DATASET_GLOBAL_ID_PREFIX"
  dataSetGlobalIdRegex = "DATASET_GLOBAL_ID_REGEX"
  clientAuthenticationMethod = "client_secret_post"    // optional
  preferredJwsAlgorithm = "RS256"                      // optional
//  responseType = "code" // id_token token            // optional
//  scope = "openid email profile phone"               // optional
//  useNonce = "true"                                  // optional
//  enableCentralLogout = false                        // optional - true by default
}

where some parameters must be substituted with a proper value:

ADA_OIDC_CLIENT_ID: Client id name;
ADA_OIDC_SECRET: Client secret credential;
ADA_OIDC_DISCOVERY_URL: The url to identify openid configuration end points.
```
  https://<host_name>:<port_number>/auth/realms/<realm_name>/.well-known/openid-configuration 
```
where:
- host_name: The host name of the OpenID provider;
- port_number: The host port number;
- realm_name: The OpenID Connect realm name.
ADA_BASE_URL: The host name of Ada application. Ex: https://ada.parkinson.lu;
ADA_OIDC_TOKEN_ENDPOINT_URL: Token end point url.
```
  https://<host_name>:<port_number>/auth/realms/<realm_name>/protocol/openid-connect/token 
```
- host_name: The host name of the OpenID provider;
- port_number: The host port number;
- realm_name: The OpenID realm name.
ADA_OIDC_LOGOUT_URL: Logout url.
```
https://<host_name>:<port_number>/auth/realms/<realm_name>/protocol/openid-connect/logout
```
- host_name: The host name of the OpenID Connect provider;
- port_number: The host port number;
- realm_name: The OpenID Connect realm name.
ROLE_ADA_ADMIN: Administrator role name (role must be set up in the client within the OIDC provider).
DATASET_GLOBAL_ID_PREFIX: Prefix role to identify dataset and give access to the user.

For instance writing a role on OpenID provider side:
```
ACCESS::datasetName
```
Imply DATASET_GLOBAL_ID_PREFIX is ACCESS::
DATASET_GLOBAL_ID_REGEX: Regex of the role to identify dataset and give access to the user.

For instance writing a role on OpenID provider side:
```
ACCESS::datasetName
```
Imply DATASET_GLOBAL_ID_REGEX is (ACCESS::)(.*)

The rest of the configuration should remain unchanged using Keycloak.