Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
May 24, 2023 6:11:38 AM
Jul 23, 2015 3:34:17 AM
The max concurrent reloads can be configured in the Qlik Sense Management Console.
- Open the Qlik Sense Management Console
- Navigate to Schedulers
- Enable the Advanced section (if not already on by default)
- Set the required max concurrent reloads
IMPORTANT TO REMEMBER
- A reload node could run NumberOfCores - 2
- If for example, the reload node has 8 CPU Cores then it could possibly run 6 concurrent reloads at its maximum.
- A reload task typically consumes at least one core, sometimes more
- With more concurrent tasks running than there are number of cores, the total time to complete a number of tasks that is larger than the number of cores does not improve with more parallel tasks running due to starvation of the server
- Each reload running is loaded into RAM. This means that the average RAM footprint grows with more parallel reloads
- With more parallel reloads than cores the probability for total starvation increases since the QIX engine is CPU hungry and other processes (for example, the operating system) might not get enough CPU cycles to perform their duties
- Use the guideline, NumberOfCPUCores - 2 in performing capacity planning on the possible amount of concurrent scheduled reloads.
- Scheduled reloads are queued, however, if not enough resources in the system
- To validate that environment needs proper capacity planning, go to C:\ProgramData\Qlik\Sense\Log\Scheduler\Trace\<Server>_System_Scheduler.txt, an example log shows below:
<ServerName>_System_Scheduler.txt
Domain\qvservice Engine connection released. 5 of 4 used
Domain\qvservice Engine connection 6 of 4 established
Domain\qvservice Request for engine-connection dequeued. Total in queue: 25
Concurrent settings
Use the "Max concurrent reloads" to limit the maximal concurrent tasks can be run at same time on current node. By default, it's set to 4, which means only 4 tasks can be run at same time on this node.
When the 5th task comes in:
- It will be queued by sequence.
- The queue has a time setting, which will eliminate (erase) a queued task if the timeout limit reaches. By default, it's set to 30 mins, in this case, if none of the first 4 tasks finishes in 30 minutes after the 5th task comes in, then the 5th task will be cancelled due to timeout. It has to be triggered again (manually or next scheduled time slot).
- Once one of the 4 running tasks finishes, the 5th task will be executed on this node if it's not timed out.
- If you set timeout to 0, this will make the queue never timeout.
- It's suggested not to run more than 10 concurrent reloads at once, unless the schedular node has an extreme amount of resources dedicated to it.
- If there is low memory or cores available on the node, then some tasks may kick off however take several hours to complete. Adding resources to the node running the reloads is suggested.
Multi-node deployment
On a multi-node deployment, tasks will be balanced from the manager node to any node(s) designated as workers.
It's highly advised to check if the central node configured is set to Manager and Worker or Manager. When set to Manager, it will send all reload jobs to the reload/scheduler nodes, as it should. However if a central node is set to Manager and Worker, this means the Central node will also be involved in performing reloads. This is not recommended.
The work flow looks as follows:
- Manager node receives a new task execution request.
- Manager node checks the resource availability on each of the worker nodes.
- Manager node assigns this task to the node with the lowest number of running tasks per "Max concurrent reloads" setting.
The improvement to track the Max concurrent reloads can, if desired, be disabled. This reverts Sense to an older load balancing method that relies only on CPU usage.
To disable the setting:
- Open the Scheduler.exe.config, which by default is located in: C:\Program Files\Qlik\Sense\Scheduler\Scheduler.exe.config
- Set DisableLegacyLoadBalancingBehavior setting to false
- Restart Qlik Sense Scheduler Service
- Repeat these actions on each node of the cluster running the Qlik Sense Scheduler Service
Example
In our example, we allow one concurrent reload, but we assume that two reloads are executed at the same time.
- If Task A is executed first, Task B is queued.
- If Task A is executed first and then fails, Task B is executed after the failure.
- After Task A's failure, Task A is queued and will execute after Task B has finished.
