Qlik Sense Repository or Proxy unable to perform operation due to buffer space TCP Port Exhaustion

[Andre_Sostizzo]

When the Repository is unable to perform an operation but is not able to assign an internal port to handle the operation, then it will output this error in the Repository and/or Proxy > Trace > System_Repository log:

Unexpected error in ExecuteGet    An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
Unable to read data from the transport connection: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.
The underlying provider failed on Open

At the same time, the following warning will be found in Windows Event System Log: 

Type of entry: Warning
Source: Tcpip
Message: A request to allocate an ephemeral port number from the global TCP port space has failed due to all such ports being in use.

Type of entry: Warning
Source: Tcpip
Message: TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint. 


Environment:

• Qlik Sense Enterprise - All versions

Cause:

This signifies that the Repository was unable to perform an operation due to the lack of an internal port on the server, which it uses to handle routing of different requests.

By adjusting MaxUserPort registry key (https://technet.microsoft.com/en-us/library/cc938196.aspx), we are allocating more internal ports available for the OS.

If further issues present, then the system may need to have an adjusted value for the TcpTimedWaitDelay registry key (https://technet.microsoft.com/en-us/library/cc938217.aspx), which may also be effective in dropping old connections which are taking up internal TCP ports.

Resolution:

1. Open regedit
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters
3. Either:
   1. Edit the existing MaxUserPort entry to a higher value. The default value is 5,000 and the maximum value is 65,534.
   2. Create a DWORD Entry, edit the entry, and enter a decimal value:

In some cases, adding the following values may also be needed to solve the issue:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"TcpTimedWaitDelay"=dword:0000001e
"TcpNumConnections"=dword:00fffffe
"TcpMaxDataRetransmissions"=dword:00000005

Server must be rebooted for registry change to take affect and this must be applied to all nodes.
Verify the port range increase.
Open a command prompt.
Enter: 
c:\netsh int ipv4 show dynamicport tcp
c:\netsh int ipv6 show dynamicport tcp

How many ports are actively established:
c:\netstat -an | find /c "ESTABLISHED"
 
-------------------------------------------------
 

If you suspect this is the problem, you can measure the port consumption using this batch file. The example is set to capture once a minute and runs until terminated.
The text output can be loaded into Qlik Sense app for quick analysis
 

:loop

set /p=%date%, %time%, %computername%, <nul >>"<your file system >\ports.log"

netstat -an |find /c "ESTABLISHED" >> "<your file system >\ports.log"

timeout 60

goto loop