Skip to content

Release notes 2018 08

Jump to bottom
korseby edited this page Aug 8, 2018 · 4 revisions

The 2018-08 release of PhenoMeNal, also known as “Dalcotidine”, was released start of August 2018. The release includes a large number new tools and workflows, including a Galaxy visualization plugin to help users to visualise ISA-Tab datasets within Galaxy. Existing tools and workflows were further improved based on user experience and feedback. All tools and workflows as part of this release have been extensively tested, and documentation and tutorials have been updated and improved. The portal’s interface has been improved thanks to the feedback received from our usability testing sessions. Moreover, we have also implemented the possibility to better customize deployments (e.g., set the number of computing nodes desired) and support for partner cloud providers – which get pre-set connection parameters.

The main user facing components of the release are:

The main deployment components of the release are:

PhenoMeNal Galaxy workflow environment

The Dalcotidine release of the PhenoMeNal Galaxy container includes:

  • >250 Galaxy tools. Large number of new tools in comparison to the last release, including

    • 5 tools for extracting collections of raw data files from an ISA dataset.
    • 30 tools for the Eco-Metabolomics workflow targeting data handling and biostatistics
    • Existing tools supporting Fluxomics workflow of 13C tracer data analysis have been updated to extend the applicability of the tools. It allows the evaluation of mass isotopomer distribution from more complex mass spectrometer data.

  • 3 new pre-assembled Workflows

    • ISA-Create-Validate-Upload workflow: The ISA-Create-Validate-Upload workflow is a demonstration of the Galaxy ISA-tools and ISA-Tab Galaxy datatype, utilising the ISA Create tool study designer that generates prospective ISA-Tab templates, validates the new ISA-Tab and generates a validation report, and uploads to MetaboLights Labs if there are no validation errors. The workflow additionally generates a factors summary and visualizes it as a parallel sets plot. This allows users to pre-register a prospective study based on a study plan.
    • Eco-Metabolomics workflow: The Eco-Metabolomics workflow is an implementation of a resource demanding use case in the field of ecology, where recently metabolomics methods have been used in large field experiments to describe interactions between different species of organisms in remarkable detail. The workflow is based on MTBLS520: Seasonal variation of secondary metabolites in nine bryophytes. Ecological aspects of the workflow are described in: Peters K, Gorzolka K, Bruelheide H, Neumann S. Seasonal variation of secondary metabolites in nine different bryophytes, [Ecol Evol. 2018;00:1–13] (https://doi.org/10.1002/ece3.4361).
    • mzQuality workflow: A set of tools that can be used to assess the quality of targeted mass spectrometry measurements. More information about the tools can be found at mzQuality.nl

This PhenoMeNal Galaxy release provides increased reliability for high loads on Kubernetes clusters by setting resources (CPU/RAM) limits for each tool.

The ISA-Tab Galaxy composite data type that was introduced in the last release has been further enhanced with better preview presentation of the included raw data files.

More details available on the PhenoMeNal Galaxy container README file and on the CI job built.

Documentation

Additional documentation has been added to the project Wiki including:

#Video Tutorials The PhenoMeNal YouTube channel includes recordings of webinars that have taken place as part of the PhenoMeNal webinar series. A video tutorial for the ISAcreate metabo galaxy workflow. (https://youtu.be/0HH1g6Bps4s)

Visualisation plugins

The Dalcotidine release includes a Galaxy visualization plugin that integrates the ISA-Tab Viewer. The [ISA-Tab Viewer plugin] (https://github.com/phnmnl/container-galaxy-k8s-runtime/tree/release/v18.01-dalcotidine/config/plugins/visualizations/isatab-viewer) allows us to view an ISA-Tab dataset in a JavaScript-rendered view, with a summary overview of study sample characteristics, study contacts, protocols, and experimental factors, and a clean view of the sample and assay tables.

PhenoMeNal Gateway Portal

The Dalcotidine release of the gateway portal puts significant emphasis on improving usability and maintainability. To this end, we ran professional usability testing sessions, which have provided valuable feedback that has been integrated in the portal. In addition, the code has been extensively refactored to favour future maintainability by a broader community. Also in this direction, we have implemented the ability to easily add “partner” OpenStack cloud providers, which are partly preconfigured and thus easier to use.

Here is a short summary of the changes included in this release. Numerous user experience improvements throughout gateway portal Improved App Library navigation and query functionality Improved usability of sign-in/sign-out procedure Advanced deployment parameters Deploy previous releases of PhenoMeNal Configure instance type, cluster and storage size Improved deployment progress monitoring Support for partner cloud providers

PhenoMeNal KubeNow Plugin Command Line Deployment

Plugin repo system. KubeNow phenomenal specific settings are now a plugin Kn provision and provisioning block in config-file (together with plugin-repo system allows for allows for simple user specific configuration) Logging and monitoring of cluster and pods are done via Elasticsearch, Fluentd, Kibana (EFK) Stack Updated to Kubernetes 1.9.5 version Added support for EGI-VOMS authentication

Galaxy Helm Chart

We have contributed our work on a Galaxy Helm chart for Kubernetes deployments to the wider Galaxy community (available here). Thus, this release now relies on community-supported Helm charts to which we are actively contributing. The new Helm chart supports the new Galaxy deployment architecture, based as well on the docker-galaxy-stable compose community images, used by PhenoMeNal which introduces several important improvements over previous versions. The main ones are: Multi-process Galaxy deployment that brings improved scalability and easily supports large numbers of concurrent users; Secure FTP service to allow upload and download of large datasets With this setup we have verified that the PhenoMeNal VRE works much better in settings with many simultaneous users, such as hands-on workshops.

PREVIOUS: Release notes 2018 02 (Cerebellin)

The 2018-02 release of PhenoMeNal, also known as “Cerebellin”, was released end of February 2018. It represents a major upgrade to the 2017-08 production release. It has a richer set of tools, depends on improved deployment software, includes improved workflows for MS and NMR, and strengthens massively the resilience deployments under high load. As in the previous release, tools and workflows are all tested.

The main user facing components of the release are:

The main deployment components of the release are:

PhenoMeNal Galaxy workflow environment

The Cerebellin release of the PhenoMeNal Galaxy container includes:

  • >180 Galaxy tools, based on more than >50 containers, including
    • 5 Tools for data transfer.
    • 3 Tools for metadata creation (ISA related).
    • 5 Tools for metadata exploration (ISA related).
    • 2 Tools for metadata conversion (W4M, ISA).
    • 1 Tool for metadata validation.
    • 12 Tools for NMR data analysis.
    • 8 XCMS modules for MS data analysis
    • 97 OpenMS modules for MS data analysis
    • 8 tools for direct infusion MS data processing (DIMSpy)
    • 26 Tools for MS feature annotation (CAMERA, metfrag, W4M, others)
    • 6 Tools for MS based 13C Fluxomics
    • 12 Tools for Statistical analysis/postprocessing.
  • 8 Workflows pre-assembled as example
    • 3 Fluxomics 13C-labeled MS Workflow, with different inputs, with and without final visualization.
    • 1D NMR Workflow
    • LC-MS/MS XCMS-Metfrag Workflow
    • Metabolomics LCMS/MS processing, quantification, annotation, identification and statistics
    • Multi-omics univariate/multivariate analysis Workflow (W4M)
    • W4M Omics generic biosigner feature selection statistics

This PhenoMeNal Galaxy release provides increased reliability for high loads on Kubernetes clusters by setting resources (CPU/RAM) limits for each tool. It includes a new Galaxy composite datatype for the ISA archive standard, and tools that can harness it. Tools are now arranged in an improved thematic layout to facilitate navigation.

More details available on the PhenoMeNal Galaxy container README file and on the CI job built.

PhenoMeNal Gateway Portal

The PhenoMeNal Gateway Portal Cerebellin release includes:

  • General reorganisation of the User Interface (UI):
    • user profile indication through a new user menu
    • new CRE menu to rapidly reach the main functionalities to manage CREs
    • new card layout for creating CREs
    • new CRE dashboard which introduces
      • more details for every CRE:
        • creation/destroy time
        • rapid links to the deployed services (Jupyter, Galaxy, Luigi)
      • direct access to deployment logs
    • new page for analyzing deployment logs
  • Validation of Cloud Provider credentials (only AWS supported at the moment)
  • Fully deployable on Kubernetes through Portal helm charts
  • Support CRE deployment on Amazon Web Services (AWS) and Google Cloud Platform (GCP); OpenStack support will be available in short time

KubeNow

  • A Kubernetes cluster up and running in less than 10 minutes (provisioned with kubeadm)
  • Supports deployment on Amazon Web Services, Google Cloud Platform, Microsoft Azure, OpenStack and KVM.
  • Simple usage via Dockerized command line client kn
  • Flannel networking
  • Traefik HTTP reverse proxy and load balancer
  • Cloudflare dynamic DNS integration
  • GlusterFS distributed file system

PhenoMeNal plugin for KubeNow command line tool

Standalone Command Line Client for rapid and customized deployment of PhenoMeNal CRE

  • Supports deployment on Amazon Web Services, Google Cloud Platform, Microsoft Azure, OpenStack and KVM.
  • Simple usage via Dockerized command line client kn

Cloud deploy KubeNow

Cloud deployment backend of PhenoMeNal Gateway Portal and standalone Command Line Client for rapid and customized deployment of PhenoMeNal CRE

  • Supports deployment on Amazon Web Services, Google Cloud Platform, Microsoft Azure, OpenStack

Galaxy Helm Chart

A Helm chart for deploying Galaxy on Kubernetes clusters and local minikube installations (mostly for development):

  • Support for production and development environments
  • Handle RBACs
  • Automatics Database Schema update
PhenoMeNal logo Funded by the EC Horizon 2020 programme, grant agreement number 654241 EU logo

PhenoMeNal Logo

Popular pages

Clone this wiki locally