New Authentications Denied due to Low System Memory
Situation
Periodically, new connections to Identity Server or Access Gateway (proxy) services were failing with the error.
New authentications are being denied due to low system memory. Threshold 10 Current: 6.109713
Restarting the Identity Server or Access Gateway would temporarily resolve the problem
The Access Manager 4.4.4 Appliance each had 8 GB of memory. As the error was regarding authentications rather than proxy connections, it was suspected that this was java memory issue used by the embedded service provider.
Default Java memory is 1GB and had already been increased to 2 GB but the problem persisted.
Environment
- Micro Focus Access Manager 4.4.4 Appliance(s)
Resolution
Enabling Statistics Logging on the IDP Cluster will also enable statistics in the catalina log for the ESP on the Access Gateway. A recommended logging interval value for a production system would be 600 (seconds).
These statistics are then printed to the log every ten minutes and looks like
NIDPMonitor: Tick: 598
System Status
Initialization State: Started
Total Sessions: 26497
Total Subjects: 1766
Total Principals: 3532
System Memory
Free Memory: 3.2926752E8
Total Memory: 2.11759923E9
Percent Free: 15.549095
We can see that this is a busy Access Manager system.
Total memory is 2.11 exp 9 which equates to the -Xmx2048m (max Heap Memory) value configured in server.xml.
In the above statistic we only have 15% of memory available.
The error mentioned above states "Threshold 10 Current: 6.109713"
The Threshold of 10 is defined in server.xml
JAVA_OPTS="${JAVA_OPTS} -Dnids.freemem.threshold=10"
and the ESP will limit further authentication when free memory goes below 10 % and throttling will begin as we see in this example.
Free Memory: 1.36996032E8
Total Memory: 2.11759923E9
Percent Free: 6.4694033
System Throttle:
Due to Low Memory: (Request Blocked)
The solution here is to allocate more than 2 GB memory to the java heap. As the server has 8 GB RAM and is a dedicated Access Manager Appliance, 4 GB is sufficient for the operating system and we can allocate 4 GB to the java heap for the Identity Server.
As we know from our baseline that we will use at least 2 GB for java, we should allocate at least this value at startup (-Xms) to improve performance.
The following line was added to /opt/novell/nam/idp/conf/tomcat.conf to resolve the problem.
JAVA_OPTS="-server -Xmx4096m -Xms2048m -Xss128k"
