ERROR: resolved path doesnt match with open path

[asura117]

Hi My main.nf exits with the following error:

Command exit status:
  255

Command output:
  (empty)

Command error:
  FATAL:   resolved path /containers/sarek-latest.simg doesn't match with opened path /containers/sarek-latest.simg (deleted)
  ERROR  : Child exit with status 255

Work dir:
/work/11/9bf34e8a12305c98e2ec87bfdc1e2e

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
WARN: Killing pending tasks (50)
[b6/480ed9] NOTE: Process `RunFastQC (79887-1)` terminated with an error exit status (255) -- Error is ignored
[c7/c51326] NOTE: Process `RunFastQC (79880-1)` terminated with an error exit status (255) -- Error is ignored
[29/7de678] NOTE: Process `RunFastQC (79890-1)` terminated with an error exit status (255) -- Error is ignored
[11/0cf4a8] NOTE: Process `RunFastQC (79893-1)` terminated with an error exit status (255) -- Error is ignored
[b9/694c31] NOTE: Process `RunFastQC (79894-1)` terminated with an error exit status (255) -- Error is ignored
[94/53d0c5] NOTE: Process `RunFastQC (79882-1)` terminated with an error exit status (255) -- Error is ignored
[ef/d77902] NOTE: Process `RunFastQC (79904-1)` terminated with an error exit status (255) -- Error is ignored
[46/548ef9] NOTE: Process `RunFastQC (79898-1)` terminated with an error exit status (255) -- Error is ignored
[ae/3631f4] NOTE: Process `RunFastQC (79885-1)` terminated with an error exit status (255) -- Error is ignored
[b3/9bc6cc] NOTE: Process `RunFastQC (79895-1)` terminated with an error exit status (255) -- Error is ignored
[1d/d7b428] NOTE: Process `RunFastQC (79919-1)` terminated with an error exit status (255) -- Error is ignored

I am running with singularity images installed locally and using necessary modified config files.
when i go to /work/11/9bf34e8a12305c98e2ec87bfdc1e2e and execute
bash .command.run , everything seems to work fine.

[maxulysse]

Hi thanks a lot for the interest you have in Sarek.
I can see that you're having trouble for the first launch.
Can you give me more information, like:
What is the command line you used to launch Sarek?
What is in your config files?

[asura117]

Command_line.txt
containers.txt
conf.txt

nextflow.config.txt
singularity-path.config.txt
LSF.config.txt
base.config.txt
genomes.config.txt

[asura117]

command.run.txt

[asura117]

Hi, does anyone have any suggestions. Thanks

[maxulysse]

Hi,
Sorry for the late reply, I had tons of meetings recently.
It's quite strange that you managed to execute .command.run but that the process is not working.
Did you try removing scratch = true from your LSF config file?

[asura117]

Hi, No worries and thanks for looking into the issue. I removed and re ran but get same error:

Completed at: Mon Apr 08 13:46:35 BST 2019
Duration    : 16.8s
Success     : false
Exit status : 255
Error report: Error executing process > 'MapReads (79889-1)'

Caused by:
  Process `MapReads (79889-1)` terminated with an error exit status (255)

Command executed:

  bwa mem -R "@RG\tID:1\tPU:1\tSM:79889\tLB:79889\tPL:illumina"  -t 1 -M     human_g1k_v37_decoy.fasta 79889_TAAGGCGA-AAGGAGTA_L001_R1_001.fastq.gz 79889_TAAGGCGA-AAGGAGTA_L001_R2_001.fastq.gz |     samtools sort --threads 1 -m 2G - > 1.bam

Command exit status:
  255

Command output:
  (empty)

Command error:
  FATAL:   resolved path /containers/sarek-latest.simg doesn't match with opened path /containers/sarek-latest.simg (deleted)
  ERROR  : Child exit with status 255


Work dir:
  /work/f5/33d8124fa62b2d8c5bdde5984b293a

Again bash .command.run works again!

[jv@dav111(davros) 33d8124fa62b2d8c5bdde5984b293a]$ bash .command.run
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 99010 sequences (10000010 bp)...
[M::process] read 99010 sequences (10000010 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 43613, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (102, 139, 183)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 345)
[M::mem_pestat] mean and std.dev: (146.08, 60.12)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 426)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 99010 reads in 8.992 CPU sec, 9.282 real sec
[M::process] read 99010 sequences (10000010 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (1, 43608, 2, 2)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (103, 138, 183)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 343)
[M::mem_pestat] mean and std.dev: (145.90, 59.83)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 423)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 99010 reads in 9.122 CPU sec, 9.664 real sec
[M::process] read 99010 sequences (10000010 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 43573, 1, 1)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (102, 138, 183)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 345)
[M::mem_pestat] mean and std.dev: (145.53, 59.82)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 426)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 99010 reads in 9.078 CPU sec, 9.472 real sec
[M::process] read 99010 sequences (10000010 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 43606, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (101, 138, 182)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 344)
[M::mem_pestat] mean and std.dev: (145.19, 59.88)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 425)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 99010 reads in 9.0

EDIT: add triple quotes

[maxulysse]

I'm asking other people to help out on this one.
Currently, we think that it could be related to your Singularity installation.
FastQC is the most used tool in all our pipelines, so it's definitively the most tested process.
So at least, I'm pretty sure it's not directly a Sarek issue, but probably something due either to the container (but you managed to execute the .command.run that launch the container...) or the Singularity install.
By any chance, did you try any nf-core pipeline?

[asura117]

I was going to try other pipelines after I 've given up on sarek!
nf-core has no current germline pipeline at the moment, if I am correct there is one under development.
It has an rna-seq pipeline which is not useful for me but https://github.com/nf-core/exoseq is still not released.
So are you certain that the "resolved path..." error is not generated by Sarek/nextflow but is created by Singulairty?

Best

[maxulysse]

This error is definitively generated by Nextflow, but it's definitively not a Sarek one, I do think it's a Singularity one, but I'm afraid I can't be sure.

nf-core/exoseq development is on hold at the moment cf nf-core/exoseq#40

As far as I'm concerned our plan is to move Sarek to nf-core at some point, but no definitive plan yet.

[asura117]

Ok, thanks Maxime. Please let me know if you think of something for me to try e.t.c.

[maxulysse]

Of course, I asked everyone I could think of to help me with that.
I'm hoping we'll figure something out.
Any chance you can use docker or conda instead of singularity ?

[ewels]

@pditommaso - have you seen this before?

  FATAL:   resolved path /containers/sarek-latest.simg doesn't match with opened path /containers/sarek-latest.simg (deleted)

[asura117]

Any chance you can use docker or conda instead of singularity ?
Docker is not available on our HPC due to security issues. Havent tried conda yet...

[maxulysse]

Docker is not available on our HPC due to security issues.

Of course, that's why we're all using Singularity ;-)

Havent tried conda yet...

Tell me how it goes

[pditommaso]

have you seen this before?

This is a Singularity error message but I've never seen it before. Is something deleting the container image behind the scene?

Update:
Maybe NF, but I don't see why it should enter in the exception branch.

[jamie-alnasir]

Yes, it's definitely an error that is thrown by Sarek. It's in the GO language source files: prepare_linux.go which references a function in syscall_linux.go. It appears that it checks to see the input path provided to a container matches the path that the container is loaded from and uses SYS_READLINKAT. I wonder if it's to do with a symbolic link not matching the actual file?

[maxulysse]

Closing due to moving to nf-core/sarek.
If you still have the issue @asura117 can you please open a new one on the nf-core repo?
All the best,
Maxime