Approach to improving the performance of software processes for processing and storing large volumes of geomagnetic data

The issues of increasing the computational speed of software processes for the analytical processing of large volumes of geomagnetic data, which are the result of continuous monitoring of the parameters of the geomagnetic field by a great number of distributed ground magnetic stations and observatories, are discussed. A comparative review of the existing geomagnetic data architecture (presented in the framework of the specified IAGA-2002 format provided by International Association of Geomagnetism and Aeronomy), as well as popular data formats is given, and arguments are presented in favor of the need to improve the approach to organizing the results of geomagnetic observations. To solve this problem, a new hybrid format for long-term storage of geomagnetic data is presented, represented by a set of three interrelated components and characterized in that it uses the rules of referential integrity to combine relational, hierarchical and columnar data models used to describe metadata and geomagnetic data, and also sets POSIX-component addressing structure and implements a combination of textual and binary formats for presenting information. The main purpose of the proposed architecture is to increase the reactivity of software tools for analytic processing of geomagnetic data, on the one hand, and reducing the cost of the required amount of physical memory, on the other hand. The results of the comparison of the proposed hybrid format for presenting geomagnetic data with the existing approach to describing geomagnetic observation data (IAGA-2002), as well as other common formats for presenting large volumes of structured and semi-structured data (XML, JSON, Avro, etc.) are presented. In this case, the criteria for evaluating the effectiveness of a hybrid format for storing geomagnetic data determined the reactivity of software data processing and the amount of required disk space for their placement. The results of the experiment showed that the proposed format provides a significant increase in computing performance (about 4 times), conducted in relation to sets of heterogeneous geomagnetic data, and also significantly reduces the computational costs associated with their physical storage (approximately 5 times).

Keywords

Authors

References