Load testing on single server

Application: w3wp.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.AccessViolationException

   at System.Threading.Thread.InternalCrossContextCallback(System.Runtime.Remoting.Contexts.Context, IntPtr, Int32, System.Threading.InternalCrossContextDelegate, System.Object[])

   at System.Runtime.Remoting.Channels.CrossContextChannel.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage)

   at System.Runtime.Remoting.Proxies.RemotingProxy.CallProcessMessage(System.Runtime.Remoting.Messaging.IMessageSink, System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Contexts.ArrayWithSize, System.Threading.Thread, System.Runtime.Remoting.Contexts.Context, Boolean)

   at System.Runtime.Remoting.Proxies.RemotingProxy.InternalInvoke(System.Runtime.Remoting.Messaging.IMethodCallMessage, Boolean, Int32)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)

   at .....followed by application exception 

When I was running my load tests using JMeter against a single server (usually against a load balancer but for some reason wanted to know the)I happened to get this weird error repeatedly with 100% error rate in Jmeter. The application I was testing was a WCF service hosted on IIS.

My first instinct was to check if the error had occurred because of my recent changes, I checked the event viewer which told me that this error was occurring very rarely.

It took a lot of googling and ended up with nothing, so I decided to check on the IIS 7 configuration.

I found the root cause of the failure, it was due to few configurations that had to be changed in IIS.


Configuration:

IIS Queue Length: The default value is 1000, so after 1000 active connections any new connections will be served a 503 Error.

IIS rapid fail protection: By default the value is set as true. So after 5 application errors, the app pool goes down.

To sum it up, since I ran the load test for 'n' Users. The default IIS Queue size is overloaded with many requests crossing the default 1000 queue size which returns application errors our and the other config “Rapid Fail Protection” pulls down the app pool.

This error has occurred only because I load tested on a single server with a connection limit of 1000 in the IIS.

I increased the queue size to the max value of 9000 and disabled the Rapid Fail Protection and Ran the same LOAD TEST. It FIXED the problem and worked well as I saw a error rate of less than 0.3% for a very large sample and the above error never occurred again. Turns out it only occurred because I ran the load test against a single server. But it did point out to me that all the servers had a default value of 1000 Queue size and in case of an actual load all the servers would have gone crashing down. 

So to conclude the above error was a memory error I believe. If you faced the same error please post below on what you did to fix it.

Here are the configurations for IIS 7.0 as suggested by Microsoft. Be careful in whatever you need for your application, not all is good for you.

https://technet.microsoft.com/en-us/library/cc745955.aspx

SoapUI hosting on Tomcat error

06-Nov-2016 14:18:34.624 SEVERE [http-nio-9092-exec-2] org.apache.catalina.core.ApplicationContext.log StandardWrapper.Throwable
 java.lang.NoClassDefFoundError: Could not initialize class com.eviware.soapui.impl.wsdl.support.http.HttpClientSupport
at com.eviware.soapui.DefaultSoapUICore.initSettings(DefaultSoapUICore.java:361)
at com.eviware.soapui.DefaultSoapUICore.init(DefaultSoapUICore.java:129)
at com.eviware.soapui.DefaultSoapUICore.<init>(DefaultSoapUICore.java:114)
at com.eviware.soapui.mockaswar.MockAsWarServlet$MockServletSoapUICore.<init>(MockAsWarServlet.java:317)
at com.eviware.soapui.mockaswar.MockAsWarServlet.init(MockAsWarServlet.java:71)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1183)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1099)
at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:779)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:133)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)

This is the error I happened to get on SOAPUI 5.2.1 on trying to deploy a war file containing mock service on Apache Tomcat 8.5.

This issue is caused because the hosting service is unable to find some jar files required to host the service. This can be FIXed by ensuring soapUI has internet access. Once I ran this on a system having internet access, it worked fine.

If you have a system without internet access, you can use the SOAPUI itself to host the mock service and disable logging, it is good enough to run a Load Test on it. Response time I saw was 0.1ms . 

JMeter Memory error

As I was running a load test on an WCF service for the first time with JMeter, once the samples reached 30k the JMeter UI froze up and I had to close JMeter abruptly. I happened to check on the JMeter logs and found that this was the last entry
jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: Java heap space

As usual I turned to google and found out that this error can be forgone by increasing the Java heap space.

This line in  jmeter.bat or jmeter.sh script which tells the launching instance of JMeter how much heap size it should use. 

JVM_ARGS="-Xms512m -Xmx512m" jmeter.sh

I changed it to -Xms1024m at both places and relaunched JMeter.  (Please Note that your system that has JMeter setup has sufficient RAM )

It FIXed the problem. 

I am able to test the application now, crossing the previous 30k sample and going beyond till I stop the test manually.

Also I found the below article with a couple of more fixes that you might need in case you are facing the above error still.

https://www.blazemeter.com/blog/9-easy-solutions-jmeter-load-test-%E2%80%9Cout-memory%E2%80%9D-failure