[apollo] Webapollo 2 slow

[Jacques Dainat]

Hi all,

I’m using webapollo 2.0.4, and I find that modifying gene models is taking forever compare to older version (as 1.0.5). Like to stretch an exon I have to wait between 20 seconds and 1min that the modification is taken into account and displayed. I have 12 genomes, and I set up tomcat as advised within the documentation:
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?


Webapollo config:
Version: 2.0.4-SNAPSHOT <https://github.com/GMOD/Apollo/releases>
Grails version: 2.5.5
Groovy version: 2.4.4
JVM version: 1.7.0_111

Best regards,

Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service

[Nathan Dunn]

Jacques,

Sorry that things are taking too long. Thanks for posting the details.

A few things:

1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4 <https://github.com/GMOD/Apollo/releases/tag/2.0.4>). This is unlikely to be the issue, but its hard to say.
2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory? (ps -ef | grep java . . . as well as top)
3 - The min-memory is pretty large. I would probably use 1GB (Xms1024), but I don’t think that would be a problem.
4 - What version of Java are you using?
5 - Are you seeing spikes in database (again using top) during this?
6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory

Post by Jacques Dainat
Hi all,
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?
Version: 2.0.4-SNAPSHOT <https://github.com/GMOD/Apollo/releases>
Grails version: 2.5.5
Groovy version: 2.4.4
JVM version: 1.7.0_111
Best regards,
Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/

[Jacques Dainat]

Hello,

Post by Nathan Dunn
1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4 <https://github.com/GMOD/Apollo/releases/tag/2.0.4>). This is unlikely to be the issue, but its hard to say.

I will change it soon.

Post by Nathan Dunn
2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory? (ps -ef | grep java . . . as well as top)

the ps command gives the result I pasted in the previous email.
the top show: 2018 tomcat 20 0 15.9g 9.0g 7988 S 0.3 28.6 64:36.86 java

Post by Nathan Dunn
3 - The min-memory is pretty large. I would probably use 1GB (Xms1024), but I don’t think that would be a problem.

Ok, I will give a try.

Post by Nathan Dunn
4 - What version of Java are you using?

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)

Post by Nathan Dunn
5 - Are you seeing spikes in database (again using top) during this?

Yes postgres is going up to 100% CPU.

Post by Nathan Dunn
6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

The user load is quite low. But no matter how many people are using it. It is slow even if I’m alone to use it.


Best regards,

Jacques

Post by Nathan Dunn

Post by Jacques Dainat
Hi all,
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?
Version: 2.0.4-SNAPSHOT <https://github.com/GMOD/Apollo/releases>
Grails version: 2.5.5
Groovy version: 2.4.4
JVM version: 1.7.0_111
Best regards,
Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/ <http://genomearchitect.org/>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/

[Nathan Dunn]

Jacques,

It is probably a bug in one of the queries (PostgreSQL being pegged), though you should update to the current release (2.0.4) in case we have already fixed it.

Just so I can reproduce it (it it is unfixed with the release), I had a few questions:

1 - do you see the same effect for ALL organisms?
2 - roughly how many annotated features (genes, etc.) do you have in the organism in question?
3 - if you create a new test organism with no annotated features, do you see the same effect?
4 - how many organisms do you have?
5 - how many users do you have registered on the system?
6 - is it any worse / better in regions with lots of isoforms (screenshot would be helpful if possible) and if so, how many isoforms are we looking at?

Thanks,

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory

Post by Jacques Dainat
Hello,

Post by Nathan Dunn
1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4 <https://github.com/GMOD/Apollo/releases/tag/2.0.4>). This is unlikely to be the issue, but its hard to say.

I will change it soon.

Post by Nathan Dunn
2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory? (ps -ef | grep java . . . as well as top)

the ps command gives the result I pasted in the previous email.
the top show: 2018 tomcat 20 0 15.9g 9.0g 7988 S 0.3 28.6 64:36.86 java

Post by Nathan Dunn
3 - The min-memory is pretty large. I would probably use 1GB (Xms1024), but I don’t think that would be a problem.

Ok, I will give a try.

Post by Nathan Dunn
4 - What version of Java are you using?

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)

Post by Nathan Dunn
5 - Are you seeing spikes in database (again using top) during this?

Yes postgres is going up to 100% CPU.

Post by Nathan Dunn
6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

The user load is quite low. But no matter how many people are using it. It is slow even if I’m alone to use it.
Best regards,
Jacques

Post by Nathan Dunn

Post by Jacques Dainat
Hi all,
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?
Version: 2.0.4-SNAPSHOT <https://github.com/GMOD/Apollo/releases>
Grails version: 2.5.5
Groovy version: 2.4.4
JVM version: 1.7.0_111
Best regards,
Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/ <http://genomearchitect.org/>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/ <http://genomearchitect.org/>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/

[Nathan Dunn]

Also forgot to ask, how many scaffolds / sequences per organism are we looking at and what is the approximate size of these genomes?

If any of this is non-public data, please feel free to reply to me directly.

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory

Post by Nathan Dunn
Jacques,
It is probably a bug in one of the queries (PostgreSQL being pegged), though you should update to the current release (2.0.4) in case we have already fixed it.
1 - do you see the same effect for ALL organisms?
2 - roughly how many annotated features (genes, etc.) do you have in the organism in question?
3 - if you create a new test organism with no annotated features, do you see the same effect?
4 - how many organisms do you have?
5 - how many users do you have registered on the system?
6 - is it any worse / better in regions with lots of isoforms (screenshot would be helpful if possible) and if so, how many isoforms are we looking at?
Thanks,
Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory

Post by Jacques Dainat
Hello,

Post by Nathan Dunn
1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4 <https://github.com/GMOD/Apollo/releases/tag/2.0.4>). This is unlikely to be the issue, but its hard to say.

I will change it soon.

Post by Nathan Dunn
2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory? (ps -ef | grep java . . . as well as top)

the ps command gives the result I pasted in the previous email.
the top show: 2018 tomcat 20 0 15.9g 9.0g 7988 S 0.3 28.6 64:36.86 java

Post by Nathan Dunn
3 - The min-memory is pretty large. I would probably use 1GB (Xms1024), but I don’t think that would be a problem.

Ok, I will give a try.

Post by Nathan Dunn
4 - What version of Java are you using?

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)

Post by Nathan Dunn
5 - Are you seeing spikes in database (again using top) during this?

Yes postgres is going up to 100% CPU.

Post by Nathan Dunn
6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

The user load is quite low. But no matter how many people are using it. It is slow even if I’m alone to use it.
Best regards,
Jacques

Post by Nathan Dunn

Post by Jacques Dainat
Hi all,
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?
Version: 2.0.4-SNAPSHOT <https://github.com/GMOD/Apollo/releases>
Grails version: 2.5.5
Groovy version: 2.4.4
JVM version: 1.7.0_111
Best regards,
Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/ <http://genomearchitect.org/>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/ <http://genomearchitect.org/>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/ <http://genomearchitect.org/>

[Nathan Dunn]

Just a quick update on this if you are experiencing a slowdown (operations that take >3 seconds to complete):

- I’ve tested this with mysql (but it should be the same) and 35K annotated genes and I have good performance (0.3-2 seconds) for all operations I’ve tried (setting exon boundaries, adding transcripts, merging, etc.). I’ll test with postgresql and see if I notice anything different.

- I’ve moved most of the conversation to https://github.com/GMOD/Apollo/issues/1276 <https://github.com/GMOD/Apollo/issues/1276> in case there is a coding fix

- Loading features using the add-transcript-from-gff3.pl may clog the preferences table: https://github.com/GMOD/Apollo/issues/1277 <https://github.com/GMOD/Apollo/issues/1277> . . I can’t see why that would be a problem, but feel free to delete the contents of that table to see if it helps (and let me know if it does).

- You should have an index on unique_name on the feature_event table. This only seems to affect mysql (postgresql does magic), so make sure that is there. By default this is added, so you should not need to do anything.

- I’ll let you know if there are any system-wide recommendations.


Nathan

Post by Jacques Dainat
Hello,

Post by Nathan Dunn
1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4 <https://github.com/GMOD/Apollo/releases/tag/2.0.4>). This is unlikely to be the issue, but its hard to say.

I will change it soon.

Post by Nathan Dunn
2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory? (ps -ef | grep java . . . as well as top)

the ps command gives the result I pasted in the previous email.
the top show: 2018 tomcat 20 0 15.9g 9.0g 7988 S 0.3 28.6 64:36.86 java

Post by Nathan Dunn
3 - The min-memory is pretty large. I would probably use 1GB (Xms1024), but I don’t think that would be a problem.

Ok, I will give a try.

Post by Nathan Dunn
4 - What version of Java are you using?

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)

Post by Nathan Dunn
5 - Are you seeing spikes in database (again using top) during this?

Yes postgres is going up to 100% CPU.

Post by Nathan Dunn
6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

The user load is quite low. But no matter how many people are using it. It is slow even if I’m alone to use it.
Best regards,
Jacques

Post by Nathan Dunn

Post by Jacques Dainat
Hi all,
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?
Version: 2.0.4-SNAPSHOT <https://github.com/GMOD/Apollo/releases>
Grails version: 2.5.5
Groovy version: 2.4.4
JVM version: 1.7.0_111
Best regards,
Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/ <http://genomearchitect.org/>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/ <http://genomearchitect.org/>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/