Identity on very large data

[davidmaimoun]

Hi,

Thanks for the tool!

Can I use it on a very large data, like 300K-400K genomes (reads & assemblies)

Best,

[hani-girgis]

Hello.

How long are these genomes? And how many reads are there?

Best,

Hani Z. Girgis, PhD

[davidmaimoun]

Approximately 5Mb, and I think to focus on assemblies first, i.e, if I have ~400k fastas, it would be possible to use Identity on it?

Thank yoy

[hani-girgis]

I believe so. How much RAM do you have available? The -b and -v parameters will help you with controlling the memory consumption. I'd start with -b of 1000 and -v of 1000. Please keep me posted.

Best regards.

[davidmaimoun]

Thank you very much for your help,

I'll let you know when I get the results

Kind Regards

[LauraVP1994]

I also have a quite large dataset and it has already been running a week. I was wondering if there are possibilities to speed this up (I have tried already to reduce the number of sequences as much as possible)? Also, being able to see how much it has done and still to do, would be great to know...

[hani-girgis]

Hi.

Would you please provide some information about your dataset? How many sequences? What is the average length? What parameters are you using to run Identity?

Best regards.

[LauraVP1994]

I have multiple datasets on which I would like to use it as I'm using this tool to select sequences to shrink down my dataset. I have for example a dataset of 553 123 Campylobacter coli sequences for which the length ranges from 20340 to 1822675 bp.

I'm currently using this command:
srun --cpus-per-task 40 --mem=100G identity -d -Campylobacter.coli_concatenated_filter.fasta -o Campylobacter.coli_identity.txt -t 0.9

[hani-girgis]

Hello there.

I would divide the sequences into groups based on length. Plotting the length distribution will help with finding the boundaries. Then I'd cluster each group separately. If you would like me to a take a look at the plot, feel free to email it to me at hzgirgis at buffalo dot edu. Please keep me posted.

Best regards.

[LauraVP1994]

And how big should these groups be then ideally (in numbers and difference in lengths)?

[hani-girgis]

You should use a large number of short sequences and a small number of long sequences. A plot of the length distribution would help with finding the cut-offs.