Merge pull request galaxyproject#3561 from galaxyproject/post-gcc-gat

Add some post gat updates

_plugins/jekyll-jsonld.rb

@@ -33,7 +33,6 @@ def generate_dublin_core(material, site)
 
       material['last_modified_at'].format = '%s'
       begin
-        puts "#{material['last_modified_at']} #{Time.at(material['last_modified_at'].to_s.to_i)} #{material.fetch('path')}"
         attributes += [
           ["DC.date", Time.at(material['last_modified_at'].to_s.to_i)],
         ]

topics/admin/faqs/git-gat-path.md

@@ -0,0 +1,55 @@
+---
+title: Galaxy Admin Training Path
+area: ansible
+box_type: comment
+layout: faq
+contributors: [hexylena]
+---
+
+The yearly Galaxy Admin Training follows a specific ordering of tutorials. Use this timeline to help keep track of where you are in Galaxy Admin Training.
+
+{% assign tutorials = "ansible-galaxy tus cvmfs singularity tool-management data-library connect-to-compute-cluster job-destinations pulsar gxadmin monitoring tiaas reports ftp" | split: " " %}
+
+{% assign seen_tuto = 0 %}
+<ol id="git-gat-timeline">
+{% for tutorial in tutorials %}
+    <a href="{{ site.baseurl }}/topics/admin/tutorials/{{ tutorial }}/tutorial.html">
+    <li class="{% if include.tutorial == tutorial %}active{% elsif seen_tuto == 0 %}disabled{% endif %}">
+        <span>Step {{ forloop.index }}</span>
+        <span>{{ tutorial }}</span>
+    </li>
+    {% if include.tutorial == tutorial %}{% assign seen_tuto = 1 %}{% endif %}
+    </a>
+    {% unless forloop.last %}
+    <span aria-hidden="true">
+        <i class="fas fa-arrow-right" aria-hidden="true"></i>
+    </span>
+    {% endunless %}
+{% endfor %}
+</ol>
+
+<style type="text/css">
+#git-gat-timeline {
+    display: flex;
+    flex-direction: row;
+    flex-wrap: wrap;
+}
+#git-gat-timeline li  {
+    display: flex;
+    flex-direction: column;
+    border: 1px solid black;
+    border-radius: 5px;
+    padding: 0.5em;
+    margin: 0.5em;
+}
+#git-gat-timeline li.active {
+    background: #86d486;
+    color: black;
+}
+#git-gat-timeline li.disabled {
+    background: #eee;
+}
+#git-gat-timeline span {
+    align-self: center;
+}
+</style>

topics/admin/tutorials/ansible-galaxy/tutorial.md

@@ -44,6 +44,10 @@ requirements:
     title: "A VM with at least 2 vCPUs and 4 GB RAM, preferably running Ubuntu 18.04 - 20.04."
 ---
 
+> ### {% icon warning %} Warning: 22.05 not yet released
+> We updated the tutorials ahead of GCC, preparing for the GCC Galaxy Admin Training. However, 22.05 is not yet released and we still have some bugs in the roles to work our, so, until that time we recommend you follow the [GTN Training Archive](https://training.galaxyproject.org/archive/2022-05-01/topics/admin/tutorials/ansible-galaxy/tutorial.html) materials for this tutorial (and all others in this series.)
+{: .warning}
+
 This tutorial assumes you have some familiarity with [Ansible](https://www.ansible.com/resources/get-started) and are comfortable with writing and running playbooks. Here we'll see how to install a Galaxy server using an Ansible playbook. The Galaxy Project has decided on Ansible for all of its deployment recipes. For our project, Ansible is even more fitting due to its name:
 
 > An ansible is a category of fictional device or technology capable of instantaneous or faster-than-light communication. It can send and receive messages to and from a corresponding device over any distance or obstacle whatsoever with no delay, even between star systems (Source: [Wikipedia](https://en.wikipedia.org/wiki/Ansible))
@@ -58,6 +62,8 @@ We want to give you a comprehensive understanding of how the Galaxy installation
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="ansible-galaxy" %}
+
 {% snippet topics/admin/faqs/admin-testing.md %}
 
 # Playbook Overview
@@ -249,6 +255,10 @@ To proceed from here it is expected that:
 {: .comment}
 
 
+> ### {% icon warning %} Warning: 22.05 not yet released
+> We updated the tutorials ahead of GCC, preparing for the GCC Galaxy Admin Training. However, 22.05 is not yet released and we still have some bugs in the roles to work our, so, until that time we recommend you follow the [GTN Training Archive](https://training.galaxyproject.org/archive/2022-05-01/topics/admin/tutorials/ansible-galaxy/tutorial.html) materials for this tutorial (and all others in this series.)
+{: .warning}
+
 ## Requirements
 
 We have codified all of the dependencies you will need into a YAML file that `ansible-galaxy` can install.
@@ -265,7 +275,7 @@ We have codified all of the dependencies you will need into a YAML file that `an
 >    +++ b/requirements.yml
 >    @@ -0,0 +1,14 @@
 >    +- src: galaxyproject.galaxy
->    +  version: 0.10.3
+>    +  version: 0.10.4
 >    +- src: galaxyproject.nginx
 >    +  version: 0.7.0
 >    +- src: galaxyproject.postgresql

topics/admin/tutorials/connect-to-compute-cluster/tutorial.md

@@ -41,21 +41,21 @@ follow_up_training:
       - job-destinations
 ---
 
-# Running Galaxy Jobs with Slurm
-
-{% snippet faqs/galaxy/analysis_results_may_vary.md %}
-
 The tools that are added to Galaxy can have a wide variance in the compute resources that they require and work efficiently on.
 To account for this, Galaxy's job configuration needs to be tuned to run these tools properly. In addition, site-specific variables must
 be taken into consideration when choosing where to run jobs and what parameters to run them with.
 
+{% snippet faqs/galaxy/analysis_results_may_vary.md %}
+
 > ### Agenda
 >
 > 1. TOC
 > {:toc}
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="connect-to-compute-cluster" %}
+
 ## Installing Slurm
 
 > ### {% icon comment %} Ansible Best Practices

topics/admin/tutorials/cvmfs/tutorial.md

@@ -101,6 +101,8 @@ A slideshow presentation on this subject can be found [here](slides.html). More
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="cvmfs" %}
+
 > The agenda we're going to follow today is: We're going to install and
 > configure Galaxy CVMFS reference data using ansible. We're going to explore
 > the CVMFS installation and then we're going to configure Galaxy to use it.
@@ -609,6 +611,56 @@ Now all we need to do is tell Galaxy how to find it! This tutorial assumes that
 > Yes. Often admins with a cluster keep a smaller cache local to each compute node, and then setup a Squid proxy to hold the most commonly accessed data on a machine with more storage. E.g. each compute node could have 10-50GB of CVMFS storage while you might setup a Squid proxy with 200-300 GB of storage that will store everything your site uses.
 {: .question}
 
+> ### {% icon tip %} Debugging failed mounting
+>
+> Are you having issues mounting your CVMFS mount? Is it giving strange errors like "Endpoint not connected"
+> Try running this command as root:
+>
+> > > ### {% icon code-in %} Input: Bash
+> > > ```console
+> > > /usr/bin/cvmfs2 -d -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint singularity.galaxyproject.org /mnt
+> > > ```
+> > {: .code-in}
+> >
+> > > ### {% icon code-out %} Output: Consolue
+> > > ```console
+> > > Debug: using library /usr/lib/libcvmfs_fuse3_stub.so
+> > > CernVM-FS: running in debug mode
+> > > CernVM-FS: loading Fuse module... (cvmfs) Parsing config file /etc/cvmfs/default.conf    [07-21-2022 11:11:20 UTC]
+> > > (cvmfs) execve'd /bin/sh (PID: 280373)    [07-21-2022 11:11:20 UTC]
+> > > (cvmfs) Parsing config file /etc/cvmfs/default.d/50-cern-debian.conf    [07-21-2022 11:11:20 UTC]
+> > > (cvmfs) execve'd /bin/sh (PID: 280375)    [07-21-2022 11:11:20 UTC]
+> > > (cvmfs) Parsing config file /etc/cvmfs/default.d/80-ansible-galaxyproject-cvmfs.conf    [07-21-2022 11:11:20 UTC]
+> > > (cvmfs) execve'd /bin/sh (PID: 280378)    [07-21-2022 11:11:20 UTC]
+> > > [...]
+> > > (dns) empty hostname    [07-21-2022 11:11:20 UTC]
+> > > (download) installed 1 proxies in 1 load-balance groups    [07-21-2022 11:11:20 UTC]
+> > > (cvmfs) DNS roaming is disabled for this repository.    [07-21-2022 11:11:20 UTC]
+> > > (catalog) constructing client catalog manager    [07-21-2022 11:11:20 UTC]
+> > > (catalog) Initialize catalog    [07-21-2022 11:11:20 UTC]
+> > > (cache) unable to read local checksum    [07-21-2022 11:11:20 UTC]
+> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfspublished to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfspublished    [07-21-2022 11:11:20 UTC]
+> > > (download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 0)    [07-21-2022 11:11:20 UTC]
+> > > (cache) miss ./e2/ab48b0984729d99951cb62c4312f501b3ddc6b (-2)    [07-21-2022 11:11:20 UTC]
+> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX    [07-21-2022 11:11:20 UTC]
+> > > (download) Verify downloaded url /data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX, proxy DIRECT (curl error 0)    [07-21-2022 11:11:20 UTC]
+> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfswhitelist to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfswhitelist    [07-21-2022 11:11:20 UTC]
+> > > [...]
+> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/c7/f1555f421b1868b979291dc23f34a83132eadbC to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/c7/f1555f421b1868b979291dc23f34a83132eadbC    [07-21-2022 11:11:20 UTC]
+> > > (download) Verify downloaded url /data/c7/f1555f421b1868b979291dc23f34a83132eadbC, proxy DIRECT (curl error 0)    [07-21-2022 11:11:25 UTC]
+> > > (cache) finished downloading of /data/c7/f1555f421b1868b979291dc23f34a83132eadbC    [07-21-2022 11:11:25 UTC]
+> > > (cache) commit ./c7/f1555f421b1868b979291dc23f34a83132eadb ./txn/fetchJWcwtt    [07-21-2022 11:11:25 UTC]
+> > > (quota) pin into lru c7f1555f421b1868b979291dc23f34a83132eadb, path file catalog at singularity.galaxyproject.org:/ (c7f1555f421b1868b979291dc23f34a83132eadb)    [07-21-2022 11:11:25 UTC]
+> > > (cache) commit failed: cannot pin c7f1555f421b1868b979291dc23f34a83132eadb    [07-21-2022 11:11:25 UTC]
+> > > (catalog) failed to load catalog '' (2 - not enough space to load catalog)    [07-21-2022 11:11:25 UTC]
+> > > (catalog) failed to initialize root catalog    [07-21-2022 11:11:25 UTC]
+> > > Failed to initialize root file catalog (16 - file catalog failure)
+> > > (cache) unpinning / unloading all catalogs    [07-21-2022 11:11:25 UTC]
+> > > ```
+> > {: .code-out}
+> {: .code-2col}
+{: .tip}
+
 # Other Aspects
 
 > Right, we'll go back to our tutorial. um Yeah. Just finally, just before we

topics/admin/tutorials/data-library/tutorial.md

@@ -46,6 +46,7 @@ of a sequencing run or similar project to a group of users on the servers.
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="data-library" %}
 
 # Setup
 

topics/admin/tutorials/ftp/tutorial.md

@@ -30,9 +30,6 @@ tags:
   - git-gat
 ---
 
-# Overview
-{:.no_toc}
-
 This tutorial will guide you to setup an {FTP} server so galaxy users can use it to upload large files. Indeed, as written on the [galaxy community hub](https://galaxyproject.org/ftp-upload/), uploading data directly from the browser can be unreliable and cumbersome. FTP will allow users to monitor the upload status as well as resume interrupted transfers.
 
 > ### Agenda
@@ -42,6 +39,8 @@ This tutorial will guide you to setup an {FTP} server so galaxy users can use it
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="ftp" %}
+
 # FTP
 
 {FTP} is a very old and reliable communication protocol that has been around since 1971 {% cite rfc114 %}. It requires a server (here our galaxy server) and a client (user's computer). The FTP server requires to have at least 2 ports accessible from outside (one for the commands and one for the transfer). Usually the port for the command is 21.

topics/admin/tutorials/gxadmin/tutorial.md

@@ -23,9 +23,6 @@ tags:
   - git-gat
 ---
 
-# Overview
-{:.no_toc}
-
 We will just briefly cover the features available in `gxadmin`, there are lots of queries that may or may not be useful for your Galaxy instance and you will have to read the documentation before using them.
 
 It started life as a small shell script that Helena wrote because she couldn't remember what [Gravity](https://github.com/galaxyproject/gravity) was called or where it could be found. Some of the functions needed for things like swapping zerglings are still included in gxadmin but are highly specific to UseGalaxy.eu and not generally useful.
@@ -39,6 +36,8 @@ Since then it became the home for "all of the SQL queries we [galaxy admins] run
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="gxadmin" %}
+
 ## Installing gxadmin
 
 It's simple to install gxadmin. Here's how you do it, if you haven't done it already.

topics/admin/tutorials/job-destinations/tutorial.md

@@ -35,21 +35,22 @@ requirements:
       - connect-to-compute-cluster
 ---
 
-# Mapping Jobs
-
-{% snippet faqs/galaxy/analysis_results_may_vary.md %}
 
 This tutorial heavily builds on the [Connecting Galaxy to a compute cluster]({% link topics/admin/tutorials/connect-to-compute-cluster/tutorial.md %}) and it's expected you have completed this tutorial first.
 
 Now that you have a working scheduler, we will start configuring which jobs are sent to which destinations.
 
+{% snippet faqs/galaxy/analysis_results_may_vary.md %}
+
 > ### Agenda
 >
 > 1. TOC
 > {:toc}
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="job-destinations" %}
+
 # Galaxy and Slurm - Statically Mapping a Job
 
 We don't want to overload our training VMs trying to run real tools, so to demonstrate how to map a multicore tool to a multicore destination, we'll create a fake tool.

topics/admin/tutorials/monitoring/tutorial.md

@@ -36,9 +36,6 @@ requirements:
 ---
 
 
-# Overview
-{:.no_toc}
-
 Monitoring is an incredibly important part of server monitoring and maintenance. Being able to observe trends and identify hot spots by collecting metrics gives you a significant ability to respond to any issues that arise in production. Monitoring is quite easy to get started with, it can be as simple as writing a quick shell script in order to start collecting metrics.
 
 
@@ -49,6 +46,8 @@ Monitoring is an incredibly important part of server monitoring and maintenance.
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="monitoring" %}
+
 This tutorial explicitly assumes you are starting with a setup like that created in the *Galaxy installation with Ansible* tutorial
 
 # Data Flow

topics/admin/tutorials/pulsar/tutorial.md

@@ -41,9 +41,6 @@ requirements:
 ---
 
 
-# Overview
-{:.no_toc}
-
 Pulsar is the Galaxy Project's remote job running system. It was written by John Chilton ([@jmchilton](https://github.com/jmchilton)) of the Galaxy Project. It is a python server application that can accept jobs from a Galaxy server, submit them to a local resource and then send the results back to the originating Galaxy server.
 
 More details on Pulsar can be found at:
@@ -65,6 +62,8 @@ At the Galaxy end, it is configured within the `job_conf.yml` file and uses one
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="pulsar" %}
+
 **This tutorial assumes that:**
 
 - You have a VM or machine where you will install Pulsar, and a directory in which the installation will be done. This tutorial assumes it is `/mnt`

topics/admin/tutorials/reports/tutorial.md

@@ -38,6 +38,8 @@ The reports application gives some pre-configured analytics screens. These are v
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="reports" %}
+
 # Setting up Reports
 
 The reports application is included with the Galaxy codebase and this tutorial assumes you've already done all of the setup required for Galaxy, systemd, uWSGI, and NGINX.

topics/admin/tutorials/singularity/tutorial.md

@@ -26,9 +26,6 @@ requirements:
       - ansible-galaxy
 ---
 
-# Overview
-{:.no_toc}
-
 In this tutorial you will learn how to configure Galaxy to run jobs using [Singularity](https://sylabs.io/singularity/) containers provided by the [BioContainers](https://biocontainers.pro/) community.
 
 ## Background
@@ -52,6 +49,8 @@ Singularity is an alternative to Docker that is much friendlier for HPCs
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="singularity" %}
+
 # Installing Singularity
 
 First, we will install Singularity using Ansible. On most operating systems there is no package for singularity yet, so we must use a role which will compile it from source. If you're on CentOS7/8, it is available through the EPEL repository.

topics/admin/tutorials/tiaas/tutorial.md

@@ -32,9 +32,13 @@ requirements:
       - pulsar
 ---
 
+Galaxy is widely used for teaching. In order to facilitate instructors, [Galaxy Europe](https://usegalaxy.eu) has developed Training Infrastructure as a Service (TIaaS).
+Workshop instructors can apply for TIaaS, and on the day of their workshop, their participants will be placed in a special group and use dedicated
+resources, thus reducing queue times on the day of the training.
 
-# Overview
-{:.no_toc}
+![TIaaS concept](../../images/tiaas/tiaas_intro.png "With TIaaS, all of your users visit the same server. In the background, the scheduler recognises which users are training users, and directs their jobs to special resources. In the EU deployment of TIaaS jobs preferentially use private resources, but can spill over to the main queue if there is not enough space available."){: width="70%"}
+
+This tutorial will go cover how to set up such a service on your own Galaxy server.
 
 
 > ### Agenda
@@ -44,15 +48,7 @@ requirements:
 >
 {: .agenda}
 
-# Introduction
-
-Galaxy is widely used for teaching. In order to facilitate instructors, [Galaxy Europe](https://usegalaxy.eu) has developed Training Infrastructure as a Service (TIaaS).
-Workshop instructors can apply for TIaaS, and on the day of their workshop, their participants will be placed in a special group and use dedicated
-resources, thus reducing queue times on the day of the training.
-
-![TIaaS concept](../../images/tiaas/tiaas_intro.png "With TIaaS, all of your users visit the same server. In the background, the scheduler recognises which users are training users, and directs their jobs to special resources. In the EU deployment of TIaaS jobs preferentially use private resources, but can spill over to the main queue if there is not enough space available."){: width="70%"}
-
-This tutorial will go cover how to set up such a service on your own Galaxy server.
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="tiaas" %}
 
 
 # Setting up TIaaS

topics/admin/tutorials/tool-management/tutorial.md

@@ -24,9 +24,6 @@ tags:
   - git-gat
 ---
 
-# Overview
-{:.no_toc}
-
 This tutorial will introduce you to one of Galaxy's associated projects - [Ephemeris](https://ephemeris.readthedocs.io/). Ephemeris is a small Python library and set of scripts for managing the bootstrapping of Galaxy plugins - tools, index data, and workflows. It aims to help automate, and limit the quantity of manual actions admins have to do in order to maintain a Galaxy instance.
 
 > ### Agenda
@@ -36,6 +33,8 @@ This tutorial will introduce you to one of Galaxy's associated projects - [Ephem
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="tool-management" %}
+
 # Background
 
 You are an administrator of a Galaxy server. A colleague has approached you with a request to run a specific [Galaxy workflow]({% link topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga %}) on their data. In order to enable this workflow for your users, you will have to:

topics/admin/tutorials/tus/tutorial.md

@@ -32,6 +32,8 @@ Here you'll learn to setup [TUS](https://tus.io/) an open source resumable file
 >
 {: .agenda}
 
+{% snippet topics/admin/faqs/git-gat-path.md tutorial="tus" %}
+
 # TUS and Galaxy
 
 To allow your user to upload via TUS, you will need to: