8
responses

I am having problems with the nxlog manager. The application takes forever to load the login page and when it does load once it finally logs in it just hangs and will not load any of the dashboards. When checking the systemctl status I have seen a couple different errors.

Exception in thread "agentServerTaskExecutor-145" java.lang.OutOfMemoryError: Java heap space

Jun 09, 2020 8:55:23 AM org.apache.jasper.compiler.JDTJavaCompiler setSourceVM
WARNING: Unknown source VM 1.7 ignored.

: ERROR: 'XML document structures must start and end within the same entity.'
Jun 09, 2020 9:51:11 AM com.sun.xml.internal.messaging.saaj.soap.EnvelopeFactory createEnvelope
SEVERE: SAAJ0511: Unable to create envelope from given source

service.sh[1395]: 2020-06-09 10:55:39.820:WARN:oejh.HttpParser:qtp1329552164-8630: BadMessage: 400 Illegal character for HttpChannelOverHttp@76b3d195{r=0,a=IDLE,uri=-}

Exception in thread "C3P0PooledConnectionPoolManager[identityToken->31989haa16jlcxi18f6us8|6366ebe0]-HelperThread-#1" java.lang.OutOfMemoryError: Java heap space

AskedJune 9, 2020 - 4:43pm

Answer (1)

Hey Anthony...

In nxlog-manager.conf, increase these two values and restart the service.

manuel@portatil:/opt/nxlog-manager/conf$ sudo grep JVM *
nxlog-manager.conf:JVM_OPTS="-Xms1g -Xmx2g...

Comments (7)

  • anthonyweller's picture

    Increase the value of Xms1g and Xmx2g? increase them to what? I'm sorry maybe I don't understand. Below is pasted what that entire block in the .conf file read as now.

    JVM_OPTS="-Xms1g -Xmx2g -XX:PermSize=64m -XX:MaxPermSize=256m -Xss2m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dorg.apache.commons.collections.enableUnsafeSerialization=true -Dorg.eclipse.jetty.annotations.maxWait=120 -Dnxlog.manager.jrobin.path=$JROBIN_PATH -Dlogs.root=$LOGS_PATH -Dnxlog.manager.path=$NXLOG_MANAGER_PATH -Dnxlog.manager.contextpath=$CONTEXT_PATH -Dnxlog.manager.webapp.path=$WEBAPP_PATH -Dspring.profiles.active=$INSTANCE_MODE"

  • manuel.munoz's picture
    (NXLog)

    You could try with...

    JVM_OPTS="-Xms2g -Xmx4g -XX:PermSize=64m -XX:MaxPermSize=256m -Xss2m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dorg.apache.commons.collections.enableUnsafeSerialization=true -Dorg.eclipse.jetty.annotations.maxWait=120 -Dnxlog.manager.jrobin.path=$JROBIN_PATH -Dlogs.root=$LOGS_PATH -Dnxlog.manager.path=$NXLOG_MANAGER_PATH -Dnxlog.manager.contextpath=$CONTEXT_PATH -Dnxlog.manager.webapp.path=$WEBAPP_PATH -Dspring.profiles.active=$INSTANCE_MODE"
    

  • anthonyweller's picture

    Hi Manual,

    Thank you that is clear to me now. I adjusted the size on those and reloaded and the GUI is responsive again. I do however have many agents showing as offline that were online before this issue, maybe it will take time for all of them to get caught up on reporting in? Thank you for any further insight or help on the manager, it is greatly appreciated.

  • manuel.munoz's picture
    (NXLog)

    Anthony, you are right, if you have hundreds/thousands of agents it will take a good few minutes for all of the to reconnect.

  • anthonyweller's picture

    Hello,

    After changing the settings as you suggested the manager has been working pretty well although still a bit slow. However we have begun to have issues with the performance again and even have problems getting the login page to load. I do continue to see java heap memory errors however when increasing this setting again it does not resolve the issue. I also continue to see the this warning. Thank you.

    Jun 17 10:07:01 service.sh[12323]: Jun 17, 2020 10:07:01 AM org.apache.jasper.compiler.JDTJavaCompile...rceVM
    Jun 17 10:07:01 service.sh[12323]: WARNING: Unknown source VM 1.7 ignored.

  • anthonyweller's picture

    We have not increased the number of agents significantly however I do think the issue is on 2 of our agents and the amount of load that they create. From investigating the logs the heap memory errors are only happening when the manager is trying to connect with these 2 agents. They handle a very high EPS and a fair amount of filtering/processing as well. I have the config set at -Xms4g -Xmx8g and the manager is now operating fairly well. I am going to increase the overall memory on this server as setting it this way is eating up the majority of available resources on this virtual server (4 CPUs, 10GB RAM). Do you have any other recommendations on how I could improve the performance on the manager? Thank you.