Enterprise features of archiving in SCADA and MES systems

The second part of our blog introduces additional enterprise features of archiving in SCADA and MES systems

Archive cache

The archive cache was implemented as early as 2004, however for a long time, it only served to store the values needed to calculate statistical archives. A few MB of memory was enough for such a cache, which it was able to use efficiently. For each object, only such an interval of values was cached that was needed by the depending statistical archive with the largest period. This cache was therefore short-lived and was not used when reading from the archive.

It was not until 2014 that the archive cache was expanded - the so-called isochronous cache mode, in which values with the same time depth are stored for all archive objects, was implemented. The depth depends on the configured cache size (usually hundreds of MB and, in the case of 64-bit archives, even several GB) and it is dynamically adjusted to the number of archived values per second so that it does not exceed the configured size. The isochronous cache is also used for reading.

Figure 6: Information about isochronous cache size (4 GB) covering 1 day and 9 hours. The cache contains about 75 million records, which corresponds to 620 archived values every second.

If the average user needs to display data for graphs for the last work shift, 24 hours, and the like, it is possible to handle most requests by reading from the cache, without the need to burden the SQL database (and disks) by reading. Thus, such reading is significantly faster (real-world measurements show the availability of data within a few tens of ms, so most of the user's waiting is caused by waiting for the graph to be displayed) and less demanding on system resources - and the throughput of the archive reading is multiplied. If the user also needs older data, it is first read from the cache and then the older data is read from the database - so even such reading is accelerated (in the ratio of cached data to older data).

The D2000 Archive allows the isochronous cache to be initialized after startup (by artificially generated reading). Although it creates a load on the disks (usually for a few minutes), on the other hand, afterward the archive has the data available in a cache. An alternative is the gradual spontaneous filling of the cache with data being archived and data being read as a result of user requests.

Figure 7: A part of the archive log about the beginning and end of filling the cache at startup. Cache with a size of 2 GB was filled with values of 65,000 archive objects in 9 minutes.

Analysis and diagnostics

It is important that configurators and system administrators have the means to perform diagnostics and analysis of the archive in SCADA and MES systems. Questions such as "what is the archive doing right now", "what burdens it the most", or "why this particular archive object has just such a value" are probably the most common questions that people responsible for archiving need to answer. Let's take a look at what the D2000 Archive has to offer in this regard.

• Diagnostic structure (see Figure 2) showing basic information about the status of archives: number of values inserted into the database, number of requests in queues, size of archive databases, free space in them, cache usage, and others.

• Possibility to enable diagnostic logging for one/multiple objects according to the mask. The archive lists information on writing and reading the values of specified archive objects

Figure 8: activation of monitoring for the structured archive H._SysProf.Archiv.PendingDbRequest and log about archiving of two new values for the 1st and 3rd lines.

• Display of an information log about a specific archive object - configuration and runtime information, information about the source object, data from the cache, etc.

Figure 9: Part of the information log.

• Writing the request queue to a disk - at any time it is possible to save the queue of accumulated requests in text format to disk. It is intended for the analysis of what objects and with which timestamps are in the queue and why the queue has grown.

• Generation of statistics for the last N hours - on request, the archive will generate a CSV file containing a list of all objects and the number of values stored in the archive for a specified period. It is used to detect archive objects with the largest number of changes and thus occupying the most disk space.

• Detection of objects with the largest number of values in the archive cache - similar functionality as in the previous point (if the isochronous cache is enabled), but it only applies to the values in the cache, so it is not possible to specify a time period. The advantage over statistics is the already sorted list of objects and the response within a few seconds. To generate statistics may take up to several minutes and the data must be analyzed with a spreadsheet.

Figure 10: Listing of objects from the archive cache - the name and ID of the object as well as the number of values and the time interval for which the data is cached are displayed.

• Measuring the performance of writes in the archive - it is possible to pause the writing of values (“freeze”) for a selected time interval. The archive then processes the requests accumulated in the queue. From the processing speed, it is possible to judge the performance reserve of particular hardware. If this test is performed on a passive archive, users are not affected in any way.

Figure 11: A graph of the number of values inserted in the archive (ten-second averages) showing the period of the four-minute “freeze” command (10:30) and the subsequent processing of the accumulated values (10:34). The ratio of the maximum of the graph (16123) to the weighted average (1313) shows a satisfactory performance reserve.

These were probably the most important enterprise features that the D2000 Archive offers to configurators and administrators in SCADA and MES systems. Some of them can also be used in smaller solutions. In any case, it's good to know that D2000 technology is scalable, can handle significant data growth, and can take advantage of the resources that today's servers offer.

The D2000 archive also has other interesting features - we will mention the automatic reorganization or support of time slices, which could be a subject of an independent article.

Ing. Peter Humaj, www.ipesoft.com