Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 13 Nov 12:27 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Dannes,


On Nov 13, 2012, at 4:52 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

Yes I guess study is required. I would have assumed that if the eXist-DB starts some threads it is responsible for terminating them in response to an HttpServlet.destroy(). These threads are operating within the control of the eXist-DB as servlet not as separate servlets that conform to the Servlet 3 spec. At least this is my meager understanding. I would think that whatever approach is used for on servlet container should work when deployed in another Servlet 3 compliant container?


from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

Yes the only issues that I've run across thus far are related to the new package / repo system and the orphan threads - which as I say seem to have just appeared since sometime after rev 17457. I've fixed a few aspects of the interface between the war deploy and the package / repo system but am stuck on the missing linkage when the autodeploy is triggered in the war deploy.

Thanks,
Chris









D.


On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Wolfgang Meier | 13 Nov 20:43 2012

Re: war file issue - hanging threads keep tomcat from terminating

> We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should
be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 07:49 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello,

I've looked a bit more at the hanging threads and it appears that the QuartzScheduler threads in fact terminate. Even though the quartz scheduler reports that it is shutdown (BrokerPool line 1840), they are reported by tomcat as still running before tomcat actually shuts down ports and so forth:

Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-51173"]
Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-51176"]

jstack does not show the DefaultQuartzScheduler_Workers at this point. Only the betterform threads:

xfTestConfigOneElementInMemory.data

and

net.sf.ehcache.CacheManager 

are still hanging.

I look forward to seeing the solution.

This is the remaining issue I know of that is specific to the war files.

Chris




On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:

We need to study how to stop these separate threads (ehcache and quartz).

The ehcache thread is used by betterform and I think Joern said he found a solution. The quartz thread should be stopped by eXist. At least it did so in the past.

Wolfgang

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 10:44 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

the ehcache problem has been solved in betterFORM itself and that
version has been updated in trunk but it seems that the necessary
config in web.xml did not make it into the repo yet.

The following markup needs to be added to web.xml to stop the ehcache thread:
    <listener>
        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
    </listener>

We'll fix that in trunk asap.

Joern

On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
<chris.j.tomlinson <at> gmail.com> wrote:
> Hello,
>
> I've looked a bit more at the hanging threads and it appears that the
> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
> reports that it is shutdown (BrokerPool line 1840), they are reported by
> tomcat as still running before tomcat actually shuts down ports and so
> forth:
>
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
> INFO: A valid shutdown command was received via the shutdown port. Stopping
> the Server instance.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
> very likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
> likely to create a memory leak.
> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/exist] appears to have started a thread named
> [Thread-9] but has failed to stop it. This is very likely to create a memory
> leak.
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-51173"]
> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>
>
> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
> the betterform threads:
>
> xfTestConfigOneElementInMemory.data
>
> and
>
> net.sf.ehcache.CacheManager
>
> are still hanging.
>
> I look forward to seeing the solution.
>
> This is the remaining issue I know of that is specific to the war files.
>
> Chris
>
>
>
>
> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>
> We need to study how to stop these separate threads (ehcache and quartz).
>
>
> The ehcache thread is used by betterform and I think Joern said he found a
> solution. The quartz thread should be stopped by eXist. At least it did so
> in the past.
>
> Wolfgang
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 14 Nov 16:34 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Hello Joern,

Great! I tried the patch and it works fine.

Thank you,
Chris

On Nov 14, 2012, at 3:29 PM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Hi,
> 
> the ehcache problem has been solved in betterFORM itself and that
> version has been updated in trunk but it seems that the necessary
> config in web.xml did not make it into the repo yet.
> 
> The following markup needs to be added to web.xml to stop the ehcache thread:
>    <listener>
>        <listener-class>de.betterform.agent.web.servlet.BfServletContextListener</listener-class>
>    </listener>
> 
> We'll fix that in trunk asap.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 7:49 AM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>> Hello,
>> 
>> I've looked a bit more at the hanging threads and it appears that the
>> QuartzScheduler threads in fact terminate. Even though the quartz scheduler
>> reports that it is shutdown (BrokerPool line 1840), they are reported by
>> tomcat as still running before tomcat actually shuts down ports and so
>> forth:
>> 
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardServer await
>> INFO: A valid shutdown command was received via the shutdown port. Stopping
>> the Server instance.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol pause
>> INFO: Pausing ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.core.StandardService
>> stopInternal
>> INFO: Stopping service Catalina
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [net.sf.ehcache.CacheManager <at> 528f2588] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is
>> very likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Nov 14, 2012 12:09:50 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/exist] appears to have started a thread named
>> [Thread-9] but has failed to stop it. This is very likely to create a memory
>> leak.
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol stop
>> INFO: Stopping ProtocolHandler ["ajp-bio-51176"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["http-bio-51173"]
>> Nov 14, 2012 12:09:50 PM org.apache.coyote.AbstractProtocol destroy
>> INFO: Destroying ProtocolHandler ["ajp-bio-51176"]
>> 
>> 
>> jstack does not show the DefaultQuartzScheduler_Workers at this point. Only
>> the betterform threads:
>> 
>> xfTestConfigOneElementInMemory.data
>> 
>> and
>> 
>> net.sf.ehcache.CacheManager
>> 
>> are still hanging.
>> 
>> I look forward to seeing the solution.
>> 
>> This is the remaining issue I know of that is specific to the war files.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> On Nov 14, 2012, at 1:28 AM, Wolfgang Meier <wolfgang <at> exist-db.org> wrote:
>> 
>> We need to study how to stop these separate threads (ehcache and quartz).
>> 
>> 
>> The ehcache thread is used by betterform and I think Joern said he found a
>> solution. The quartz thread should be stopped by eXist. At least it did so
>> in the past.
>> 
>> Wolfgang
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov
>> _______________________________________________
>> Exist-open mailing list
>> Exist-open <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> 

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Dannes Wessels | 14 Nov 16:41 2012

Re: war file issue - hanging threads keep tomcat from terminating

Another step closer to 2.0 final release!!!!



On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson <chris.j.tomlinson <at> gmail.com> wrote:

Great! I tried the patch and it works fine.

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Joern Turner | 14 Nov 21:51 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Tobi has applied the necessary change to fix the web.xml to include
the context listener. So it should be all fine now.

Joern

On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
> Another step closer to 2.0 final release!!!!
>
>
> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
> <chris.j.tomlinson <at> gmail.com> wrote:
>>
>>
>> Great! I tried the patch and it works fine.
>
>
> --
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 15 Nov 07:37 2012
Picon

Re: war file issue - hanging threads keep tomcat from terminating

Joern,

Excellent! I have verified that it is working. Tomcat now shuts down properly.

As of now I am not aware of any war file related issues with trunk (rev 17590) and Tomcat 7.0.11 and Tomcat
7.0.32. 

I have not verified deploying the war in Jetty, but I think Wolfgang did so yesterday as far as the expathrepo issues.

Regards,
Chris

On Nov 15, 2012, at 2:36 AM, Joern Turner <joern.turner <at> gmail.com> wrote:

> Tobi has applied the necessary change to fix the web.xml to include
> the context listener. So it should be all fine now.
> 
> Joern
> 
> On Wed, Nov 14, 2012 at 4:41 PM, Dannes Wessels <dannes <at> exist-db.org> wrote:
>> Another step closer to 2.0 final release!!!!
>> 
>> 
>> On Wed, Nov 14, 2012 at 4:34 PM, Chris Tomlinson
>> <chris.j.tomlinson <at> gmail.com> wrote:
>>> 
>>> 
>>> Great! I tried the patch and it works fine.
>> 
>> 
>> --
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
Chris Tomlinson | 13 Nov 11:28 2012

war file issue - hanging threads keep tomcat from terminating

Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris



------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dannes Wessels | 13 Nov 12:07 2012

Re: war file issue - hanging threads keep tomcat from terminating

Hi,

We need to study how to stop these separate threads (ehcache and quartz). Actually for log4j there is a separate hook to stop this function, for jetty we have a similar trick uisng a WebapContext doing this.

A solution might be complex, but maybe it is not that bad.... It might a tomcat-specific-change in web.xml, which I do not like. Servlet3 would make a solution more simple..... but lets study it first.

from your wording I can conclude that build.sh dist-war results into a more or less working WAR file?

D.



On Tue, Nov 13, 2012 at 11:28 AM, Chris Tomlinson <ct <at> moonvine.org> wrote:
Dannes,

I've identified an issue with war files on trunk rev 17576 and earlier - appearing sometime after rev 17457.

I've tried both tomcat 7.0.11 and 7.0.32.

The issue is that there are some threads that are not being terminated by eXist when the HttpServlet.destroy() is called:

Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [net.sf.ehcache.CacheManager <at> 1b2dd1b8] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [xfTestConfigOneElementInMemory.data] but has failed to stop it. This is very likely to create a memory leak.
Nov 13, 2012 3:24:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/exist] appears to have started a thread named [Thread-9] but has failed to stop it. This is very likely to create a memory leak.

The BrokerPool.shutdown is executing normally but the above orphan threads - and sometimes QuartzScheduler threads - keep tomcat from exiting normally.

This seems to have appeared sometime after rev 17457 - I have a build of 17457 with which tomcat gracefully terminates. I haven't seen this issue on builds prior to and including rev 17457.

I don't know how these sorts of non-BrokerPool threads were being terminated and am hoping that someone who is more familiar with this area of the system can point in a helpful direction.

Thanks,
Chris






--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.