Performance impact of loading a large number of data files in reload script

In certain cases, you need to load from a large number of data files with the same metadata, each file containing a small set of rows. For example: a legacy system that can only export one transaction to a text file each time (number of data files = number of transactions). How does this affect performance when these files are loaded in to QlikView or Qlik Sense?

Generally, reading from a large number of files would be much more time-consuming than a few files due to the following reasons:

• When reading data from a file, like any other file-reading process, Qlik Engine has to create a file stream, read the file line by line and close the stream after the last line. Therefore, reading from multiple files means Qlik Engine needs to do a lot of stream open - close steps
• Reading from a large number of file also requires a lot of disk operations - at least one operation per file. This is significantly slower than loading a single file with many lines into memory in a single disk operation, and process that chunk of data in-memory. If the data files are located remotely, more files also means more network operations.

Therefore, for better performance, it is recommended to have a pre-processing step where multiple small data files of the same metadata are combined into a larger file before loading into QlikView or Qlik Sense.