AppStore

The HeLx Appstore is the primary user experience component of the HeLx data science platform. It is a Django based application whose chief responsibilities are authentication and authorization, all visual presentation concerns including transtions between the appstore and apps.

Compute

The system’s underlying computational engine is Kubernetes. HeLx runs in a Kubernetes cluster and apps are launched and managed within the same namespace, or administrative context, it uses. Tycho translates the docker compose representation of systems with one ore more containers into a Kubernetes represenation, coordinates and tracks related processes, provides utilization information on running processes, and manages the coordinated deletion of all components when a running application is stopped.

Storage

HeLx apps, in the near term, will mount a read only file system containing reference data and to a writable file system for user data.

Security

HeLx prefers open standard security protocols where available, and applies standards based best practices, especially from NIST and FISMA, in several areas.

Authentication

Authentication is concerned with verifying the identity of a principal and is distinct from determining what actions tha principal is entitled to take in the system. We use OAuth 2.0 facilitated by django allauth framework to integrate Google and GitHub login into the application. In addition SAML based login is also supported for integration with institutional single sign on as needed.

Authorization

Authorization assumes an authenticated principal and is the determination of the actions permitted for that principal. The first layer of authorization is a list of identities allowed access to the system. Email addresses associated with IdP accounts are included in the list. Only principals whose IdPs present an email on the list on their behalf during authentication are authorized to access the system.

Secrets

Data that serves, for example, as a credential for an authentication, must be secret. Since it may not be added to source code control, these values are private to the deployment organization, and must be dynamically injected. This is handled by using Kubernetes secrets during deployment, and trivial keys for developer desktops, all within a uniform process framework. That framework provides a command line interface (CLI) for creating system super users, data migration, starting the application, and packaging executables, among other tasks.

Management CLI

The appstore management CLI provides uniform commands for using the environment. It provides default values for secrets during local development and ways to provide secrets in proudction.

Command Description

Command Description
bin/appstore tests {product} Run automated unit tests with {product} settings.
bin/appstore run {product} Run the appstore using {product} settings.
bin/appstore createsuperuser Create admin user with environment variable provided values.
bin/appstore image build Build the docker image.
bin/appstore image push Push the docker image to the repository.
bin/appstore help Run automated unit tests with {product} settings.

Testing

Automated testing uses the Python standard unittest and Django testing frameworks. Tests should be fast enough to run conveniently, and maximize coverage. For example, the Django testing framework allows for testing URL routes, middleware and other use interface elements in addition to the logic of components.

Packaging

Appstore is packaged as a Docker image. It is a non-root container, meaning the user is not a superuser. It packages a branch of Tycho cloned within the appstore hierarchy.

App Development

HeLx supports metadata driven app development. Apps are expressed using Docker and Docker Compose. AppStore uses the Tycho engine to discover and manage Apps. The Tycho app metadata format specifies the details of each application, contexts to which applications belong, and inheritance relationships between contexts.

App specificatinos are stored in GitHub, each in an application specific subfolder. Along with the docker compose, a .env file specifies environment variables for the application. If a file called icon.png is provided, that is used as the application’s icon.

To develop a custom app for AppStore, use the guidelines below:

  1. develop one or more docker containers for your app following NIST security best practices
  2. fork dockstore-yaml-proposals
  3. create a docker-compose for your application and publish it to dockstore-yaml-proposals
  4. fork Tycho
  5. add your app specific metadata in Tycho registry metadata, including the URL to your application’s docker-compose from step 3
  6. create pull requests for both Tycho and dockstore-yaml-proposals based off your forks and submit for a code review to us. We’ll run your code through our build and security scan pipeline, rejecting any container with high or critical vulnerabilities, subject to further review and either accept or reject the pull requests.

Deployment

Appstore is deployed to Kubernetes in production using Helm. The main deployment concerns are: Security: Secrets are added to the container via environment variables. Persistence: Storage must be mounted for a datbaase. Services: The chief dependency is on Tycho which must be at the correct version.

During development, environment variables can be set to control execution:

Variable Description
DEV_PHASE=[stub, local, dev, val, prod] In stub, does not require a Tycho service.
ALLOW_DJANGO_LOGIN=[TRUE, FALSE] When true, presents username and password authentication options.
SECRET_KEY Key for securing the application.
OAUTH_PROVIDERS Contains all the providers(google, github).
GOOGLE_CLIENT_ID Contains the client_id of the provider.
GOOGLE_SECRET Contains the secret key for provider.
GOOGLE_NAME Sets the name for the provider.
GOOGLE_KEY Holds the key value for provider.
GOOGLE_SITES Contains the sites for the provider.
GITHUB_CLIENT_ID Contains the client_id of the provider.
GITHUB_SECRET Contains the secret key of the provider.
GITHUB_NAME Sets the name for the provider.
GITHUB_KEY Holds the key value for provider.
GITHUB_SITES Contains the sites for the provider.
APPSTORE_DJANGO_USERNAME Holds superuser username credentials.
APPSTORE_DJANGO_PASSWORD Holds superuser password credentials.
TYCHO_URL Contains the url of the running tycho host.
OAUTH_DB_DIR Contains the path for the database directory.
OAUTH_DB_FILE Contains the path for the database file.
POSTGRES_DB Contains the connection of the database.
POSTGRES_HOST Contains the database host.
DATABASE_USER Contains the database username credentials.
DATABASE_PASSWORD Contains the database password credentials.
APPSTORE_DEFAULT_FROM_EMAIL Default email address for appstore.
APPSTORE_DEFAULT_SUPPORT_EMAIL Default support email for appstore.
ACCOUNT_DEFAULT_HTTP_PROTOCOL Allows to switch between http and https protocol.

Development Environment

The following script outlines the process:

#!/bin/bash

set -ex

# start fresh
rm -rf appstore
#  get a vritualenv
if [ ! -d venv ]; then
    python3 -m venv venv
fi
source venv/bin/activate
# clone appstore
if [ ! -d appstore ]; then
    git clone git@github.com:helxplatform/appstore.git
fi
cd appstore
# use metadata branch and install requirements
git checkout develop
cd appstore
pip install -r requirements.txt

# configure helx product => braini
product=braini
# configure dev mode to stub (run w/o tycho api)
export DEV_PHASE=stub
# create and or migrate the database
bin/appstore updatedb $product
# create the superuser (admin/admin by default)
bin/appstore createsuperuser
# execute automated tests
bin/appstore tests $product
# run the appstore at localhost:8000
bin/appstore run $product

Development Environment with Kubernetes

Prerequisites 1. Have Access to a running k8s cluster. 2. Have kubectl set up. 3. Installing kubectl on Linux: 4. Download the latest release 5. Run:

curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-               release/release/stable.txt)/bin/linux/amd64/kubectl"
  1. Make the kubectl binary executable:: chmod +x ./kubectl
  2. Move the binary into your PATH:: sudo mn ./kubectl /usr/local/bin/kubectl
  3. Check to see if installed:: kubectl version –client

NOTE: Once kubectl has been setup then set the KUBECONFIG env variable to use other kubeconfigs. For example, the one provided to you will be exported into the terminal where tycho api would be run: export KUBECONFIG=path-to-kubeconfig-file.

Step 1: - Clone the Appstore repo (develop branch):

git clone -b develop [https://github.com/helxplatform/appstore.git](https://github.com/helxplatform/appstore.git

  • Activate virtual environment:

    python3 -m venv venv

source venv/bin/activate

  • Install the requirements:

    pip install -r requirements.txt

  • Finally run Appstore by using the management CLI:

    bin/appstore start {product}

NOTE: After running bin/appstore start {product} for the first time, please use bin/appstore run {product} every other time. So that migrations to the data-base will only take place once.

Step 2:

  • Clone the Tycho repo (develop branch):

    git clone -b develop [https://github.com/helxplatform/tycho.git](https://github.com/helxplatform/tycho.git)

  • Activate virtual environment:

    python3 -m venv venv

source venv/bin/activate

  • Install the requirements:

    pip install -r requirements.txt

  • Export the kubeconfig to the terminal where tycho api is to be run:

    export KUBECONFIG=path-to-kubeconfig-file

  • Now run tycho api in the terminal where the kubeconfig was exported:

    bin/tycho api -d

Step 3:

  • Now Appstore is running, navigate to the admin panel by appending /admin to the url http://localhost:8000/admin.
  • Login in to the admin panel using admin/admin for user/password.
  • Nagivate to the application manager: http://localhost:8000/apps. From this endpoint we can launch applications.