Incremental Reloads from QVDs are slow after upgrade
Article Number: 000053581 | Last Modified: 2018/09/07
An incremental load that used to finish in a few minutes is now taking 20+ minutes after upgrading. Using the Max() function directly on a field from a QVD is extremely slow.
There have been many underlying changes between versions which has resulted in performing aggregations on QVDs to behave the way they do currently. It will always be beneficial to make sure that the load from the QVD is an Optimized load.
Changes occurred from Qlikview 11.20 > 12.00 and Qlik Sense 3.1 > 3.2
A better way to handle Incremental Reloads when they require an aggregation performed such as max/min is to first load the full QVD (or just the fields needed) without any field modifications so that the load gets the benefit of the optimized load.
Once the table has been loaded from the QVD it will be in memory. At this point aggregations can be performed efficiently, resident load the field, and store the Max/Min value. The Max/Min value can be used as a variable when pulling the newer data from the original data source.
Alternatively, by using the 'Order by' functionality, it is possible to sort the data that will be stored in a QVD based on a field in the dataset so that when it comes time to add newer data the Max value will be first and the Min value will be last.
A Script is attached that was provided by an Implementation Consultant to handle this use case.