-
Notifications
You must be signed in to change notification settings - Fork 0
Troubleshooting_everything
Make sure that all required libraries are installed.
Ubuntu 10.04 +
apt-get install mysql-server libssl0.9.8 libgfortran3 liblapack3gf libxml-parser-perl
CentOS 5.6 +
yum install mysql-server perl-XML-Parser.x86_64 libgfortran openssl openssl098e
CentOS 6 +
yum install mysql-server perl-XML-Parser.x86_64 libgfortran compat-libgfortran-41 openssl openssl098e
Make sure the system user executing the script is either the smrtanalysis user, or is superuser.
The user which SMRT Analysis will be running under will need to be created, as well as a MySQL user with elevated privileges for creating the smrtportal schema.
SEE ALSO: [SMRT Analysis Software Installation]
All environment variables used by SMRT Analysis are specified in /opt/smrtanalysis/analysis/etc/smrtpipe.rc. A common error occurs when the TMP and SHARED_DIR variables are set to directories that do not exist on your file system. To correct the error, create the directories in the local file system, and give the directories write permissions.
sudo mkdir /scratch/
sudo chmod +x /scratch/
sudo mkdir /opt/smrtanalysis/common/userdata/shared_dir/
sudo chmod +x opt/smrtanalysis/common/userdata/shared_dir/
Then assign those directories to the environment variables by editing smrtpipe.rc:
TMP=/scratch/
SHARED_DIR=/opt/smrtanalysis/common/userdata/shared_dir/
SMRT Analysis upgrades are very sensitive to ownership and permission settings. During upgrade, you need to make sure that you are the same user as the user who installed the previous version of SMRT Analysis. You can check this by examining the ownership of the $SEYMOUR_HOME directory:
user1@server$ ls -l /opt
lrwxrwxr-x 1 smrtanalysis smrtanalysis 32 2012-12-17 08:02 smrtanalysis -> smrtanalysis-1.4.0
drwxrwxr-x 8 smrtanalysis smrtanalysis 142 2012-05-17 21:38 smrtanalysis-1.3.1
drwxrwxr-x 8 smrtanalysis smrtanalysis 142 2012-05-17 21:38 smrtanalysis-1.3.3
drwxrwxr-x 8 smrtanalysis smrtanalysis 142 2012-05-17 21:38 smrtanalysis-1.4.0
logout
smrtanalysis@server$ /opt/smrtanalysis/etc/scripts/postinstall/upgrade_and_configure_smrtanalysis.sh
In this example, the $SEYMOUR_HOME directory is set to /opt/smrtanalysis/ and this directory is actually a softlink to smrtanalysis-1.4.0, which is owned by the user smrtanalysis belonging to the group smrtanalysis. However, you are currently logged in as user1. In order to proceed, you must log out and log back in as the user smrtanalysis. This user should now run upgrade_and_configure_smrtanalysis.sh
If you lost the credentials for the smrtanalysis user, you need a root user (with sudo permissions) to change the ownership of $SEYMOUR_HOME. say user1, and then run upgrade_and_configure_smrtanalysis.sh as the new user.
user1@server$ sudo chown -R user1:group1 /opt/smrtanalysis/
user1@server$ sudo chown -R user1:group1 /opt/smrtanalysis-1.4.0/
user1@server$ ls -l /opt
lrwxrwxr-x 1 user1 group1 32 2012-12-17 08:02 smrtanalysis -> smrtanalysis-1.4.0
drwxrwxr-x 8 smrtanalysis smrtanalysis 142 2012-05-17 21:38 smrtanalysis-1.3.1
drwxrwxr-x 8 smrtanalysis smrtanalysis 142 2012-05-17 21:38 smrtanalysis-1.3.3
drwxrwxr-x 8 user1 group1 142 2012-05-17 21:38 smrtanalysis-1.4.0
If you DO NOT have sudo and, you ARE NOT the same user who installed the previous version, you cannot upgrade smrtanalysis. You MUST either log in as the same user, or find an administrator with root privileges to reset the ownership and permissions in the $SEYMOUR_HOME directory.
##Administration
Cause 1: The hostname and/or port is incorrect. The hostname and port is designated when you installed smrtanalysis. For example, if your want your SMRT Portal URI to be http://server1:8080/smrtportal/, the hostname should be set as server1 and the port should be set as 8080. To reset these variables, re-run configure_smrtanalysis.sh and enter server1 and 8080 when prompted by the script.
Cause 2: Networking issues The client computer/laptop cannot see the server. SMRT Portal is a client-server application. This means that the client computer (your laptop) must know that the server computer (server1) exists somewhere in its network. Some institutions require vpn to login to their network. You must ask your network administrator for what "hostname" to assign to SMRT Portal such that client computers can recognize it.
Cause 3: Tomcat is not on Tomcat is the webserver that hosts SMRT Portal. It must be turned on for SMRT Portal to function. When it is off, the ps command will below returns only a single line. When it is on, the ps command below will return an additional line detailing the path to the tomcatd process.
user@server1$ ps -ef | grep tomcat
71063 15603 23660 0 16:43 pts/15 00:00:00 grep tomcat
user@server1$ /opt/smrtanalysis/etc/scripts/tomcatd start
user@server1$ ps -ef | grep tomcat
71063 15603 23660 0 16:43 pts/15 00:00:00 grep tomcat
71109 16203 1 0 Dec05 ? 00:55:17 /opt/smrtanalysis-1.3.3//redist/java/bin/java -Djava.util.logging.config.file=/opt/smrtanalysis-1.3.3//redist/tomcat/conf/logging.properties -d64 -server -Xmx8g -Djava.library.path=/opt/smrtanalysis-1.3.3//common/lib -Djava.security.auth.login.config=/opt/smrtanalysis-1.3.3//redist/tomcat/conf/kerb5.conf -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/smrtanalysis-1.3.3//redist/tomcat/endorsed -classpath /opt/smrtanalysis-1.3.3//redist/tomcat/bin/bootstrap.jar:/opt/smrtanalysis-1.3.3//redist/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/smrtanalysis-1.3.3//redist/tomcat -Dcatalina.home=/opt/smrtanalysis-1.3.3//redist/tomcat -Djava.io.tmpdir=/opt/smrtanalysis-1.3.3//redist/tomcat/temp org.apache.catalina.startup.Bootstrap start
##Creating/Running Jobs
When a SMRT Portal job fails, you can generally troubleshoot the errors by looking at the smrtpipe.log file. Sometimes a job fails immediately and the smrtpipe.log file is never created. Then this happens, you can look for errors written to $SEYMOUR_HOME/common/log/smrtportal/smrtportal.0.log.
We recommend that SMRT Analysis be installed on a compute cluster with the following HW specs:
1 head node:
• Minimum 16 GB RAM. Larger references such as human may require 32 GB RAM.
• Minimum 250 GB of disk space
3 compute nodes:
• 8 cores per node, with 2 GB RAM per core
• Minimum 250 GB of disk space per node
If SMRT Analysis is only running on a single-server, the software makes no attempt to load-balance or queue any jobs on the single server. All jobs are submitted and executed, which simultaneously slows down all other processes running on the server. You must advise your users to submit SMRT Portal jobs with restraint, preferably one-at-a-time.
If SMRT Analysis is configured for distributed computing, but the jobs are still running slowly, you may want to consider the following edits to the appropriate template file for your job management system (JMS). These files are $SEYMOUR_HOME/analysis/etc/cluster/<JMS>/start.tmpl and $SEYMOUR_HOME/analysis/etc/cluster/<JMS>/interactive.tmpl.
If there are other, perhaps larger jobs, being submitted to the same queue, change the designated queue (-q option) to an exclusive environment and monitor the resource usage. For example, the start.tmpl file for Sun Grid Engine (SGE) looks like this:
qsub -pe <your_parallel_environment> ${NPROC} -S /bin/bash -V -q <your new_queue> -N ${JOB_ID} -o ${STDOUT_FILE} -e ${STDERR_FILE} ${EXTRAS} ${CMD}
You can also add options to limit the memory usage, for example, using the -M option for the bsub command in LSF:
bsub -q pacbio -g /pacbio/smrtanalysis -J ${JOB_ID} -o ${STDOUT_FILE} -e ${STDERR_FILE} -M 33000000 -n 4 ${CMD}
If you have a heterogeneous cluster, you can also use the -R option to specify compute nodes that meet certain resource requrements:
bsub -q pacbio -g /pacbio/smrtanalysis -J ${JOB_ID} -o ${STDOUT_FILE} -e ${STDERR_FILE} -M 33000000 -R 'select[type==LINUX64 && mem>=32000 && tmp>=300000] rusage[mem=32000, tmp=250000] span[hosts=1] -n 4 ${CMD} '
Finally, you can add any arbitrary number of operations to the job submission by adding lines to the tmpl files. In the following example, additional environment variables are being defined in a profile script, instead of being managed by the parallel environment (-pe) option:
. /path/to/profile
qsub ${NPROC} -S /bin/bash -V -q <your new_queue> -N ${JOB_ID} -o ${STDOUT_FILE} -e ${STDERR_FILE} ${EXTRAS} ${CMD}
The assumed file structure needed to import SMRT Cells is a top-level-directory that contains the metadata.xml file and an "Analysis_Results" directory that contains the bas.h5 file.
top-level-directory
*.metadata.xml
Analysis_Results
*.bas.h5
Make sure the directory is in this file structure, and execute ls -l to make sure that the smrtanalysis user has read permissions to the files and execute permissions to the directories.
#SMRT View
SMRT View is a client-server application. Underlying problems can generally be traced to issues with the server, client (your laptop), or the network connecting the two.
First check the client. Your laptop needs Java 6 to execute the .jnlp file. Check your java version by running jave -version on a terminal or command-line prompt. SMRT View has problems with Java 7 in Macs, but works in Java 6. If you are running Java 7, please follow the instructions on this apple support page to revert Java Webstart back to 6.
Second, check the network. Open your .jnlp file in a text editor (e.g. notepad) and check to make sure the hostname is defined correctly. The first line of your jnlp file should look like this:
<jnlp spec="6.0+" version="1.3.3" codebase="http://localhost:8080/smrtview/axis2-web/app/bin" >
In the above example, the hostname is incorrectly set to localhost, which means that only the server can see and open the jnlp file. In order for the client to also open the jnlp file and run SMRT View, it must be an externally facing name or ip address. You can reset the hostname by rerunning $SEYMOUR_HOME/etc/scripts/postinstall/configure_smrtanalysis.sh and typing it in when you are prompted.
Finally, to troubleshoot server-side issues, start by looking for errors in the $SEYMOUR_HOME/common/log/smrtview/ directory.
Go to the SMRT View homepage in http://:8080/smrtview/, and click on the Web Services Validation link to check that all web services are on and all library dependencies are installed.
#Staging, TBD - mj
- an old reference entry with headers only contain refNNNNNN| patterns, e.g.:
>ref000001|ref000001
or
>ref000001
smrtanalysis@server$ qstat -u \*
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
4298052 0.47166 S52657 root qw 12/09/2012 22:12:57 1
You should go back to change /etc/init.d/tomcatd and change from
sh $CATALINA_HOME/bin/startup.sh
to
su -c "sh $CATALINA_HOME/bin/startup.sh" smrtanalysis
(Don't forget to use root to do /etc/init.d/tomcatd stop, qdel the jobs and /etc/init.d/tomcatd start as well.)