Varda¶

Use cases¶

The following are some example use cases which Varda is designed to support.

• Private exome variant database for a sequencing lab

  Installed on the local network, Varda can be used to import and annotate variants from all exome sequencing experiments at a sequencing lab. Additionally, the database could contain public datasets from population studies (e.g., 1000 Genomes, Genome of the Netherlands), such that all exome experiments are also annotated with frequencies in those studies.

• Shared database between several groups

  Several sequencing centers can import their variants in a central Varda installation which can subsequently be used by the same centers for frequency annotation. The system can be setup such that annotation is only possible on previously imported data (to encourage sharing).

  Data from one center can only be accessed anonymized by other groups, since only the frequencies over the entire databased are available. To accomodate even stricter anonymity, samples can be imported after pooling.

• Publicly sharing variant frequencies from a population study

  Variation data from a population study can be imported in a Varda installation accessible over the internet such that others can annotate their data with frequencies in the study.

For contrast, consider the following examples of what Varda is not designed to do.

• Sharing and browsing genomic variants

  Varda is focussed on sharing variant frequencies only, and as such is not designed for direct browsing. Other systems, such as LOVD, are much more suitable for sharing and browsing genomic variants and additionally store phenotypes and other metadata.

• Ad-hoc exploration of genomic variation

  Again, Varda is focussed on sharing variant frequencies only, and does not store additional metadata nor does it allow for effective exploration of variants. If you have variation data from a disease or population study which you want to analyse in a flexible way, have a look at gemini.

Implementation¶

The server is implemented in Python using the Flask framework and directly interfaces the PostgreSQL (or MySQL) database backend using SQLAlchemy. It exposes a RESTful API over HTTP where response payloads are JSON-encoded.

Long-running actions are executed asynchonously through the Celery distributed task queue.

Variant frequencies and genomic coverage¶

Todo.

User roles¶

Todo.

Sample activation¶

Todo.

Duplication of data¶

Todo.

Calculating checksums of all imported data is used as a measure to prevent the same data to be imported twice. Of course, this is quite a weak measure in that it can easily be circumvented, so its main value lies in preventing accidental duplicate imports.

A model for trading data¶

For certain use cases it may be desirable that variant frequencies can only be retrieved from Varda by annotating variants that are imported in Varda (see Shared database between several groups). In this model variant observations are traded for variant frequencies.

Varda facilitates this with the trader role. The trader role gives a user permission to annotate a data source, but only if that data source has been imported as part of an active sample.

See User roles for more information on roles.

Sample anonymity¶

Todo. Only global frequencies, except for public samples. Of course depending on the number of samples in the database. See Pooling samples.

Pooling samples¶

By design, Varda cannot be queried in a way to reconstruct the genotype for a specific sample, unless that sample is explicitely marked as being public (see Sample anonymity). However, the sample genotypes are stored in the Varda database and for various reasons you might not be comfortable with that. This can be addressed by pooling samples before sending them to Varda, a trick that loses individual genotypes at no cost in functionality.

The idea is to mix the data from several samples together and send the result to Varda as one sample. Variant frequency calculations are not affected by this, yet individual genotypes are irrevocably scrambled. There are different ways to mix variant calls from different samples, you can find some examples below.

Besides variant calls, coverage information could also be mixed. However, it is probably not worth the trouble since this is not sensitive data.

$ (grep '^#' 1.vcf; grep -hv '^#' *.vcf | sort -k 1,1 -k 2n,2) > pooled.vcf

$ grep '^#' samples.vcf > shuffled.vcf
$ paste \
    <(grep -v '^#' samples.vcf | cut -f 1-9) \
    <(grep -v '^#' samples.vcf | cut -f 10- \
      | xargs -L 1 bash -c 'shuf -e $* | paste -s' _) \
    >> shuffled.vcf

If you’re in a hurry¶

The impatient can install and run Varda without a database server and more such nonsense with the following steps:

$ pip install -r requirements.txt
$ python -m varda.commands debugserver --setup

Don’t use this for anything serious though.

Database server: PostgreSQL¶

Install PostgreSQL and add a user for Varda. Create a database (e.g. varda) owned by the new user. For example:

$ sudo apt-get install postgresql
$ sudo -u postgres createuser --superuser $USER
$ createuser --pwprompt --encrypted --no-adduser --no-createdb --no-createrole varda
$ createdb --encoding=UNICODE --owner=varda varda

Also install some development libraries needed for building the psycopg2 Python package later and add the package to the list of requirements:

$ sudo apt-get install python-dev libpq-dev
$ echo psycopg2 >> requirements.txt

This will make sure the Python PostgreSQL database adapter gets installed in the Python virtual environment section.

Message broker and task result backend: Redis¶

Varda uses Celery for distributing long-running tasks. A message broker is needed for communication between the server process and worker processes. Simply install Redis and you’re done.

$ sudo apt-get install redis-server

Python virtual environment¶

It is recommended to run Varda from a Python virtual environment, using virtualenv. Installing virtualenv and creating virtual environment is not covered here.

Assuming you created and activated a virtual environment for Varda, install all required Python packages:

$ pip install -r requirements.txt

Now might be a good idea to run the unit tests:

$ nosetests -v

If everything’s okay, install Varda:

$ python setup.py install

Varda setup¶

Varda looks for its configuration in the file specified by the VARDA_SETTINGS environment variable. First create the file with your configuration settings, for example:

$ export VARDA_SETTINGS=~/varda/settings.py
$ cat > $VARDA_SETTINGS
DATA_DIR = '/data/varda'
SQLALCHEMY_DATABASE_URI = 'postgresql://varda:*****@localhost/varda'
BROKER_URL = 'redis://'
CELERY_RESULT_BACKEND = 'redis://'

Make sure DATA_DIR refers to a directory that is writable for Varda. This is where Varda stores uploaded and generated files.

A script is included to setup the database tables and add an administrator user:

$ varda setup

You can now proceed to Running Varda.

Alternative setups¶

The remainder of this page documents some alternatives to the recommended setup documented above.

$ sudo apt-get install mysql-server
$ mysql -h localhost -u root -p
> create database varda;
> grant all privileges on varda.* to varda@localhost identified by '*****';

$ sudo apt-get install python-dev libmysqlclient-dev
$ echo MySQL-python >> requirements.txt

$ sudo apt-get install rabbitmq-server
$ sudo rabbitmqctl add_user varda varda
$ sudo rabbitmqctl add_vhost varda
$ sudo rabbitmqctl set_permissions -p varda varda '.*' '.*' '.*'

Example configuration¶

If you followed the steps in Installation, this is a standard configuration file that will work for you:

DATA_DIR = '/data/varda'
SQLALCHEMY_DATABASE_URI = 'postgresql://varda:*****@localhost/varda'
BROKER_URL = 'redis://'
CELERY_RESULT_BACKEND = 'redis://'

This is not yet a minimal configuration. In fact, you can run Varda without a configuration file since the default configuration works out of the box. The default configuration uses an in-memory database, broker, and task result backend and a temporary directory for file storage, so it is not recommended for anything more than playing around.

The next section describes all available configuration settings.

Configuration settings¶

Note that the configuration file is interpreted as a Python module, so you can use arbitrary Python expressions as configuration values, or even import other modules in it.

Unsetting a configuration setting is done by using the value None. If no default value is mentioned for any configuration setting below it means it is not set by default.

Setting up Varda¶

Follow the installation instructions to install Varda. Configure Varda to use hg19.fa in the tests/data directory as reference genome and enable cross-origin resource sharing (CORS) (this allows Aulë to communicate with Varda). The Varda configuration file may look something like this:

DATA_DIR = 'data'
SQLALCHEMY_DATABASE_URI = 'sqlite:///varda.db'
BROKER_URL = 'redis://'
CELERY_RESULT_BACKEND = 'redis://'
GENOME = 'tests/data/hg19.fa'
CORS_ALLOW_ORIGIN = '*'

Remember to point the VARDA_SETTINGS environment variable to the configuration file before continuing.

Start Varda and a Celery worker node as described in Running Varda:

$ varda debugserver

and:

$ celery worker -A varda.worker.celery -l info

Opening http://127.0.0.1:5000/genome in your browser should now show you a JSON representation of the reference genome configuration.

Setting up Aulë¶

Get the source code for Aulë, configure it to use MyGene.info with GRCh37/hg19, and run it:

$ git clone https://github.com/varda/aule.git
$ cd aule
$ nano config.js
AULE_CONFIG = {
  BASE: '/',
  API_ROOT: 'http://127.0.0.1:5000/',
  PAGE_SIZE: 50,
  MANY_PAGES: 13,
  MY_GENE_INFO: {
    species: 'human',
    exons_field: 'exons_hg19'
  }
  MY_GENE_INFO: null
};
$ npm install
$ npm run dev

You can now open http://localhost:8000/ in your browser, which should show you the Aulë homepage. Login with admin and the password you choose during Varda setup.

Setting up Manwë¶

Manwë authenticates with the Varda API using a token. You can generate a token in the Aulë web interface by choosing API tokens in the menu and clicking Generate API token. Copy the token by clicking Show token.

Install Manwë and create a configuration file with the token you just created:

$ pip install manwe
$ nano manwe.cfg
API_ROOT = 'http://127.0.0.1:5000'
TOKEN = 'c7fa8780025c8efa5077567434e0fcb56274fbb0'

Verify that everything is setup correctly by listing all Varda users:

$ manwe users list -c manwe.cfg
User:   /users/1
Name:   Admin User
Login:  admin
Roles:  admin

Importing exome sequencing data¶

Let’s import an example set of variant calls from an exome sequencing experiment. The file tests/data/exome.vcf contains some variant calls on chromosome 20 for one individual and tests/data/exome.vcf contains regions on chromosome 20 where the sequencing was deep enough (or of high enough quality) to do variant calling:

$ cat tests/data/exome.vcf
##fileformat=VCFv4.1
##samtoolsVersion=0.1.16 (r963:234)
...
#CHROM  POS     ID  REF    ALT  QUAL  FILTER  INFO  FORMAT    -
chr20   76962   .   T      C    173   .       ...   GT:PL:GQ  0/1:203,0,221:99
chr20   126159  .   ACAAA  A    217   .       ...   GT:PL:GQ  0/1:255,0,255:99
chr20   126313  .   CCC    C    126   .       ...   GT:PL:GQ  0/1:164,250,0:99
...
$ cat tests/data/exome.bed
chr206811268631
chr207658177410
chr209002590400
...

Import the data as follows:

$ manwe samples import --vcf tests/data/exome.vcf --bed tests/data/exome.bed \
>     -l -w 'Exome sample'
Added sample: /samples/1
Added data source: /data_sources/1
Started variation import: /variations/1
Added data source: /data_sources/2
Started coverage import: /coverages/1
[################################] 100/100 - 00:00:02
Imported variations and coverages for sample: /samples/1

Since Varda supports importing data for a sample in multiple steps, new samples are inactive by default to prevent using them in frequency calculations until everything is complete. Activate the sample you just imported with:

$ manwe samples activate /samples/1
Activated sample: /samples/1

If you go back to the Aulë web interface and choose Samples in the menu, you should see the exome sample you just imported.

Importing aggregate data from 1000 Genomes¶

Sometimes it makes sense to calculate variant frequencies within a dataset separately, as opposed to global frequencies over all datasets. An example might be a large public population study such as the 1000 Genomes project. Varda allows you to import a dataset like this without providing coverage data (i.e., the BED file).

The tests/data/1kg.vcf file contains a subset of variant calls from the 1000 Genomes project over 1092 individuals. Import it as follows:

$ manwe samples import --vcf ../varda/tests/data/1kg.vcf -s 1092 -p \
>     --no-coverage-profile -w '1000 Genomes'
Added sample: /samples/2
Added data source: /data_sources/3
Started variation import: /variations/2
[################################] 100/100 - 00:00:02
Imported variations and coverages for sample: /samples/2
$ manwe samples activate /samples/2
Activated sample: /samples/2

Querying variant frequencies¶

Aulë allows for some ad-hoc querying of variant frequencies globally and per sample, as well as by variant, by region and by transcript region. Choose By region in the menu and set:

Query:

Global query

Chromosome:

chr20

Region begin:

1

Region end:

200000

This should show you the variants from the exome sequencing example, all with frequency 1.0 and N=1 (since it’s the only sample used in the calculation).

You can run the same query on the 1000 Genomes data by setting:

Query:

Sample query (1000 Genomes)

As an alternative to setting the region manually, you can also choose By transcript in the menu and select a region based on a gene transcript. The exome example has two variants in the DEFB126 gene. You can select it by clicking on Choose a transcript and typing DEFB126.

Annotating variants¶

The ad-hoc frequency queries with Aulë are nice for one-time lookups, but you would presumably also want to automate this on a larger scale. Manwë allows you to annotate local VCF or BED files with variant frequencies by supplying a list of queries:

$ manwe annotate-vcf -q GLOBAL '*' -q 1KG 'sample:/samples/2' -w \
>     tests/data/exome.vcf
Added data source: /data_sources/4
Started annotation: /annotations/1
[################################] 100/100 - 00:00:02
Annotated VCF file: /data_sources/5
$ manwe data-sources download /data_sources/5 > exome.annotated.vcf.gz

The resulting VCF file is annotated with several fields in the INFO column.

Conformance with REST¶

Although Varda tries to follow REST in its API, there are certainly parts of the API that are not completely in the spirit of REST.

Todo: More text here, at least covering the following points:

• JSON is not a hypertext format
• Accepting a request body with GET
• Todo

An example request using curl¶

To get us started, here’s an example of creating a new sample resource named 1000 Genomes using curl:

curl -u user:password -X POST -H 'Content-Type: application/json' \
  -d '{"name": "1000 Genomes"}' https://example.com/samples/

The first thing to observe is that the request is authenticated using HTTP Basic Authentication (the -u user:password argument). See Authentication for more information.

The request body is a JSON document (specified with the Content-Type header) consisting of only a name field with value 1000 Genomes. See Passing data with a request for more ways of sending data.

Finally, the request is done at the collection endpoint for sample resources using the POST method. This is the typical way of creating new resources and you can find more information on specific resources in REST API resources.

What we’ll get back is the following HTTP response:

HTTP/1.1 201 CREATED
Server: gunicorn/18.0
Date: Sat, 16 Nov 2013 10:03:17 GMT
Connection: close
Content-Type: application/json
Content-Length: 282
Location: https://example.com/samples/140
Api-Version: 0.3.0

{
  "sample": {
    "uri": "/samples/140",
    "active": false,
    "added": "2013-11-16T10:47:28.711076",
    "coverage_profile": true,
    "name": "1000 Genomes",
    "notes": null,
    "pool_size": 1,
    "public": false,
    "user": {
      "uri": "/users/1"
    }
  }
}

The response body contains a representation of the created resource as a JSON document. We can follow the Location header to that same resource.

Authentication¶

Many requests require user authentication which can be provided with HTTP Basic Authentication or token authentication.

Authentication state can be checked on the authentication resource.

GET /samples HTTP/1.1
Authorization: Token 5431792000be7601697fb5a4005984ebdd60320c

Passing data with a request¶

Data can be attached to a request in three ways:

1. As query string parameters.
2. As HTTP form data.
3. In a JSON-encoded request body.

Generally, using a JSON-encoded request body is preferred since it offers richer structure. For example, JSON has separate datatypes for strings and numbers, and supports nesting for more complex documents.

JSON-encoded bodies must always be accompanied with a application/json value for the Content-Type header.

[45, 3, 11, 89]

45,3,11,89

{
  "name1": "value1",
  "name2": "value2",
  "name3": "value3"
}

name1:value1,name2:value2,name3:value3

Date and time¶

All date and time values are formatted as strings following ISO 8601.

Queries¶

A query defines a set of samples, used to calculate observation frequencies over when annotationg variants. A query is represented as an object with two fields:

name (string)

Name for this query (alphanumeric).

expression (string)

Search query string.

The expression field is a boolean search query string in which clauses can reference sample resources and group resources. This is the grammar for query expressions:

<expression> ::= <tautology>
               | <clause>
               | "(" <expression> ")"
               | "not" <expression>
               | <expression> "and" <expression>
               | <expression> "or" <expression>

<tautology> ::= "*"

<clause> ::= <resource-type> ":" <uri>

<resource-type> ::= "sample" | "group"

The tautology query * matches all samples. A clause of the form sample:<uri> matches the sample with the given URI. A clause of the form group:<uri> matches samples that are in the group with the given URI.

When creating the set of samples matched by a query expression, only active samples with a coverage profile are considered. The exception to this are expressions of the form sample:<uri>, which can match inactive samples or samples without coverage profile.

As an example, the following is an expression matching the sample with URI /samples/5 and samples that are in the group with URI /groups/3 but not in the group with URI /groups/17:

sample:/samples/5 or (group:/groups/3 and not group:/groups/17)

Linked resources and embeddings¶

Resources can have links to other resources. In the resource representation, such a link is an object with a uri field containing the linked resource URI.

For some links, the complete representation of the linked resource can be embedded instead of just the uri field. This is documented with the resource representation.

For example, sample resources can have the linked user resource embedded:

GET /samples/130?embed=user

HTTP/1.1 200 OK
Content-Type: application/json

{
  "sample": {
    "uri": "/samples/130",
    "active": false,
    "added": "2013-03-30T00:18:48.298526",
    "coverage_profile": false,
    "name": "1KG phase1 integrated call set",
    "notes": null,
    "pool_size": 1092,
    "public": true,
    "user": {
      "uri": "/users/2",
      "added": "2012-11-30T20:28:11.409536",
      "email": null,
      "login": "martijn",
      "name": "Martijn Vermaat",
      "roles": [
        "trader",
        "annotator"
      ]
    }
  }
}

Collection resources¶

A collection resource is a grouping of any number of instance resources. Use a POST request on the collection resource to add an instance resource to it. Listing the instance resources is done with a GET request and comes with a number of utilities as described below.

GET /samples/?public=true&order=-pool_size,name HTTP/1.1
Range: items=0-5

HTTP/1.1 206 PARTIAL CONTENT
Content-Type: application/json
Content-Range: items 0-5/8

{
  "sample_collection": {
    "uri": "/samples/",
    "items": [
      {
        "uri": "/samples/130",
        "name": "1KG phase1 integrated call set",
        "pool_size": 1092,
        "public": true,
        ...
      },
      {
        "uri": "/samples/134",
        "name": "My sample",
        "pool_size": 4,
        "public": true,
        ...
      },
      {
        "uri": "/samples/135",
        "name": "A new sample",
        "pool_size": 3,
        "public": true,
        ...
      },
      {
        "uri": "/samples/129",
        "name": "Another sample",
        "pool_size": 1,
        "public": true,
        ...
      },
      {
        "uri": "/samples/131",
        "name": "Sample 42",
        "pool_size": 1,
        "public": true,
        ...
      },
      {
        "uri": "/samples/128",
        "name": "Some test sample",
        "pool_size": 1,
        "public": true,
        ...
      }
    ]
  }
}

Tasked resources¶

A tasked resource is a type of resource associated with a server task. This task is submitted upon creation of a new resource instance (i.e., via a POST request on the corresponding collection resource).

Information on the server task can be obtained with a GET request on the instance resource. A task can be resubmitted by setting its state field to submitted in a PATCH request (this requires the admin role).

Versioning¶

The API is versioned following Semantic Versioning. Clients can (but are not required to) ask for specific versions of the API with a Semantic Versioning specification in the Accept-Version header.

If the server can match the specification, or Accept-Version is not set, the response will include the API version in the Api-Version header. If the specification cannot be matched, a 406 status is returned with a no_acceptable_version error code.

Example request with Accept-Version header, and corresponding response:

GET /
Accept-Version: >=0.3.1,<1.0.0

HTTP/1.1 200 OK
Api-Version: 0.4.2

Error responses¶

If a request results in the occurrence of an error, the server responds by sending an appropriate HTTP status code and an error document containing:

1. An error code (code).
2. A human readable error message (message).

These fields are wrapped in an object called error.

POST /samples/ HTTP/1.1
Content-Type: application/json

{
  "name": "Test sample",
  "pool_size": "Thirty"
}

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "error":
    {
      "code": "bad_request",
      "message": "Invalid request content: value of field 'pool_size' must be of integer type"
    }
}

Summary of HTTP status codes¶

We give a brief overview of response status codes sent by the server and their meaning. For more information, consult HTTP/1.1: Status Code Definitions.

200

Everything ok, the request has succeeded.

201

The request has been fulfilled and resulted in a new resource being created.

206

The server has fulfilled the partial GET request for the resource.

301

Moved permanently.

400

The request data was malformed.

401

The request requires user authentication.

403

Not allowed to make this request.

404

Nothing was found matching the request URI.

406

The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.

409

The request could not be completed due to a conflict with the current state of the resource.

413

The request entity was too large.

416

Requested range not satisfiable.

500

Internal server error.

501

Not implemented.

API root¶

The API root resource contains links to top-level resources in addition to a server status code.

The root resource representation has the following fields:

uri (uri)

URI for this resource.

status (string)

Currently always ok, but future versions of the API might add other values (e.g. maintanance).

api_version (string)

API version (see Versioning).

authentication (object)

Link to the authentication resource.

genome (object)

Link to the genome resource.

annotation_collection (object)

Link to the annotation collection resource.

coverage_collection (object)

Link to the coverage collection resource.

data_source_collection (object)

Link to the data_source collection resource.

sample_collection (object)

Link to the sample collection resource.

token_collection (object)

Link to the token collection resource.

user_collection (object)

Link to the user collection resource.

variant_collection (object)

Link to the variant collection resource.

variation_collection (object)

Link to the variation collection resource.

GET /¶

Returns the resource representation in the root field.

Example request:

GET / HTTP/1.1

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "root": {
    "uri": "/",
    "api_version": "0.3.0",
    "status": "ok",
    "authentication": {
      "uri": "/authentication"
    },
    "genome": {
      "uri": "/genome"
    },
    "annotation_collection": {
      "uri": "/annotations/"
    },
    "coverage_collection": {
      "uri": "/coverages/"
    },
    "data_source_collection": {
      "uri": "/data_sources/"
    },
    "sample_collection": {
      "uri": "/samples/"
    },
    "token_collection": {
      "uri": "/tokens/"
    },
    "user_collection": {
      "uri": "/users/"
    },
    "variant_collection": {
      "uri": "/variants/"
    },
    "variation_collection": {
      "uri": "/variations/"
    }
  }
}

Authentication¶

This resource reflects the current authentication state.

The authentication resource representation has the following fields:

uri (uri)

URI for this resource.

authenticated (boolean)

Whether or not the request is authenticated.

user (object)

Link to a user resource if the request is authenticated, null otherwise.

GET /¶

Returns the resource representation in the authentication field.

Example request:

GET /authentication HTTP/1.1

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "authentication": {
    "uri": "/authentication",
    "authenticated": true,
    "user": {
      "uri": "/users/1"
      "added": "2012-11-30T20:14:27.954255",
      "email": null,
      "login": "admin",
      "name": "Admin User",
      "roles": [
        "admin"
      ],
    }
  }
}

Genome¶

If the server is configured with a reference genome, this resource lists its chromosomes.

The genome resource representation has the following fields:

uri (uri)

URI for this resource.

chromosomes (list of string)

List of chromosome names.

GET /¶

Returns the resource representation in the genome field.

Annotations¶

Annotation resources model sets of variants annotated with observation frequencies.

An annotation resource is a tasked resource. The associated server task is calculating frequencies on the linked original data source and writing the annotated data to the linked annotated data source.

An annotation is represented as an object with the following fields:

uri (uri)

URI for this resource.

task (object)

Task information, see Tasked resources.

original_data_source (object)

Link to a data source resource (embeddable).

annotated_data_source (object)

Link to a data source resource (embeddable).

Coverages¶

Coverage resources model sets of genomic regions having high enough coverage in sequencing to do variant calling.

A coverage resource is a tasked resource. The associated server task is importing the coverage data from the linked data source in the server database.

A coverage is represented as an object with the following fields:

uri (uri)

URI for this resource.

task (object)

Task information, see Tasked resources.

data_source (object)

Link to a data source resource (embeddable).

sample (object)

Link to a sample resource (embeddable).

Data sources¶

Data source resources model data from files that are either uploaded to the server by the user or generated on the server.

The actual data is modeled by the blob subresource type.

A data source is represented as an object with the following fields:

uri (uri)

URI for this resource.

added (string)

Date and time this data source was added, see Date and time.

gzipped (boolean)

Whether or not the data is compressed using gzip.

name (string)

Human readable data source name.

filetype (string)

Data filetype. Possible values for this field are bed, vcf, and csv.

data (object)

Link to a blob resource.

user (object)

Link to a user resource (embeddable).

Groups¶

Group resources model sample groups (e.g., disease type).

A group is representend as an object with the following fields:

uri (uri)

URI for this resource.

name (string)

Human readable group name.

Samples¶

Sample resources model biological samples which can contain one or more individuals.

A sample is represented as an object with the following fields:

uri (uri)

URI for this resource.

active (boolean)

Whether or not this sample is active.

added (string)

Date and time this sample was added, see Date and time.

coverage_profile (boolean)

Whether or not this sample has a coverage profile.

name (string)

Human readable sample name.

notes (string)

Human readable notes in Markdown format.

pool_size (integer)

Number of individuals in this sample.

public (boolean)

Whether or not this sample is public.

user (object)

Link to a user resource (embeddable).

groups (list of object)

Links to group resources (embeddable).

Tokens¶

Token resources model authentication tokens for API users.

A token is represented as an object with the following fields:

uri (uri)

URI for this resource.

added (string)

Date and time this sample was added, see Date and time.

key (string)

Token key used for authentication.

name (string)

Human readable sample name.

user (object)

Link to a user resource (embeddable).

Users¶

User resources model API users and their permissions.

A user is represented as an object with the following fields:

uri (uri)

URI for this resource.

added (string)

Date and time this user was added, see Date and time.

email (string)

User e-mail address.

login (string)

Login name used for authentication.

name (string)

Human readable user name.

roles (list of string)

Roles for this user. Possible values for this field are admin, importer, annotator, and trader.

Variants¶

Variant resources model genomic variants with their observed frequencies.

A variant is represented as an object with the following fields:

uri (uri)

URI for this resource.

Variations¶

Variation resources model sets of variant observations.

A variation resource is a tasked resource. The associated server task is importing the variation data from the linked data source in the server database.

A variation is represented as an object with the following fields:

uri (uri)

URI for this resource.

task (object)

Task information, see Tasked resources.

data_source (object)

Link to a data source resource (embeddable).

sample (object)

Link to a sample resource (embeddable).

Implementation¶

Varda is implemented on top of the Flask web microframework, the Celery distributed task queue, and the SQLAlchemy object relational mapper.

A typical deployment looks like this:

                         ________
                        /        \
                       |\________/|
          ___________  |          |  ___________
        /              | Database |              \
       |               |          |               |
       |                \________/                |
       |                                          |
       |
  __________                      ________  +------------+
 /          \              ____  /          |  Worker 1  |
|   Varda    |  ________  /    \  ________  +------------+
|____________|           /      \           |  Worker 2  |
|            |          | Broker | _______  +------------+
|  REST API  |           \      /           |  Worker 3  |
 \__________/             \____/            +------------+
                                                  ...
       |
       |
  __________
 /          \
|   Client   |
 \__________/

Sample types¶

Three types of samples:

1. Simple sample (one individual)
2. Pooled sample (multiple individuals)
3. Population study

The following table gives a summary of what is stored for each sample type:

• N = number of individuals
• O = number of observations per individual
• R = number of covered regions per individual

For all sample types, data can be imported from an arbitrary number of data sources. This means you could for example import variant observations per individual, per chromosome, or per variant type.

Pooled samples can have their individuals effectively anonymized by importing variant observations from one big data source in which the order is not related to the individuals. For example, cat the VCF files for all individuals and sort the result by genome position before sending it to the server.

Frequency calculation¶

Only take samples into account with covered regions (to rule out population studies).

Binning of regions and observations¶

Todo. UCSC binning scheme.

Security¶

Todo: Add a page on security to the Managing Varda section.

Authentication via HTTP Basic Authentication, or API tokens, so only use with SSL.

Sample state¶

A sample can be either active or inactive (default). An inactive sample is ignored in any frequency calculations.

Importing data sources is only possible for inactive samples. A sample cannot be made active while any data source is being imported for that sample. Users can only make a sample active, not inactive.

Version 0.1.0¶

Release date to be decided.

First public release.

Authors¶

Varda is written and maintained by Martijn Vermaat, based on a prototype by Jeroen Laros.

• Leiden University Medical Center <humgen@lumc.nl>
• Martijn Vermaat <martijn@vermaat.name>
• Jeroen Laros <j.f.j.laros@vermaat.name>

License¶

Copyright (c) 2011-2013 by Martijn Vermaat and contributors (see AUTHORS for details).

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.