Windows, Virtual Memory, Working Sets, and the Page File
Qlik products run on the Windows family of operating systems which run on one of two main families of chipsets: x86 (32bit) and x64 (64bit). This combination is very well understood and documented.
Windows and the chipset collaborate to present a memory model to process called “Virtual Memory”. In this model a process does not address physical memory directly, but instead uses IDs to address chunks of “Virtual Memory” called “Pages”. When an application wants to read or write to a Page of memory it uses the ID of the page and the address of memory within the Page rather than an address in physical memory.
This separation of “Virtual Memory” from physical memory allows Windows to move Pages from physical memory to the disk and back from the disk to physical memory, without the application being aware of where its memory is at any one time. The effect is that the set of applications running on Windows can address much more memory than physically exists on the machine. The downside of this system is that the “Page File” is on disk and is very slow compared to physical memory. It is typically up to Windows to reduce this lost time by intelligently deciding which Pages stay in memory and which are moved to the “Page File”.
The “Working Set” is a term invented by Microsoft that refers to the set of Pages an application has in physical memory at any one time. The Working Set varies over time because Windows can swap Pages onto the disk or load Pages into memory.
Page File Settings
The Qlik engine can use the Page File, but due to the impact on performance, this usage should be reduced.
We suggest to allow Windows to manage the page file size, though this can be modified. If memory settings for the QlikView or Qlik Sense system are correct, and the memory is sufficient for the requirements, then the Page File should rarely be used by qvs.exe or the Sense engine.
The Page File should not be removed or reduced to 0 because this can introduce instabilities into Windows itself and also into qvs.exe if it does need to use the Page File.
Microsoft also suggests that a Page File be moved to a dedicated disk that does not have redundancy checks (RAID for example is not needed).
Allocated, Committed, and Free Virtual Memory
In the Performance log of qvs.exe there are 4 fields that relate to Virtual Memory:
A page is allocated to a process by Windows when the process requests the memory. However, it is only Committed when the process begins to use it (first write). This allows certain Page IDs to be valid but to not actually have the Page at all – neither in physical memory nor in the Page File – because the Page contains nothing. As a result Committed <= Allocated.
Free only functions on 32bit systems. On 64bit systems Windows returns the theoretical amount of memory it can address rather than the theoretical maximum given the real amount of physical RAM and Page File. So on 64bit it almost always returns 8912GB.
The Largest Free Block is typically not very useful. In 32bit mode QlikView may ask for one set of contiguous pages of memory that is larger than what can be returned by Windows. This can cause problems for QlikView. In 64bit architectures though, this situation rarely occurs and this information should simply be ignored.
In short, the key measure is the Allocated field. Other information that can be useful but must be obtained from Windows monitoring tools include the amount of physical memory allocated to the process and the amount of Page File allocated to the process. Unfortunately, how to do this is out of the scope of this document.
- Allocated – Amount of memory in Pages that have been requested by qvs.exe.
- Committed – Amount of memory in Pages that have been used by qvs.exe.
- Free – Amount of Virtual Memory theoretically available.
- Largest Free Block – The largest block of contiguous memory that is free.
QlikView Memory Model
QlikView’s memory usage can be broken up into two main categories:
Necessary memory – internal data structures, each document’s database, definition of GUI, every user’s session state etc.
Cache – results of calculations previously carried out to avoid recalculating. This memory can theoretically be lost and recalculated each time.
By default, each QlikView/Sense Server is set to a Low Working Set Limit of 70% and a High limit of 90%. The QlikView/Sense Server Service will try to use physical memory exclusively and not use the Page File at all up until the Low value is reached.
Once the Low value is reached, QlikView will begin discarding old cache data and replace it with new where possible. QlikView will also signal Windows that it can begin using the pagefile if necessary. When the High working set limit is reached, QlikView will try to stop caching data altogether. Note that at this point issues with the services will begin to arise (performance/stability problems). The QlikView Event log will usually shed light on that and help identify whether or not you might require more memory or not.
Because usage of the Page File will slow QlikView or Qlik Sense Server down dramatically, it is important to monitor memory usage and adjust the Low setting correctly. If memory grows beyond the Low setting and this setting can’t be increased without impacting the rest of the server, then more memory is needed. Memory can grow beyond the High setting, but this will dramatically impact the performance of the platform. In short, there is no way to fix a hard limit on how much memory QlikView Server can use and have it refuse to load documents or connect users for example.