Data warehouse process

First, standardize the data warehouse process.

Standardized operation flow is an effective means to avoid operation errors. On this basis, the contents and methods of data quality check in the process of storing airborne geophysical data are analyzed, and 9 system checks and 5 topology checks are summarized (Table 5-5). Considering that in the process of data storage, the data collector needs to be given the right to edit and delete the database data (so as to edit the number of errors entered and delete the incorrect data imported), when editing or deleting the database data, the archive data may be edited or deleted incorrectly, which may destroy the integrity and correctness of the archive data. The standardization process of checking the data quality stored in the aviation geophysical database is proposed (Figure 5-2).

Table 5-5 Incoming Data System Check and Topology Check

1) Create a project, that is, create a project before data storage, and import or enter data according to the project.

2) System check before warehousing. The imported or entered warehousing data must pass the pre-warehousing inspection (data uniqueness, data type and omission inspection) of the system before being saved in the collection library.

3) After the data enters the library, it must be checked by the system. If the spatial data must be topologically checked, it will be compared with the original data file byte by byte, and then manually checked.

4) Manual inspection and manual recheck: Manual inspection and manual recheck are carried out on engineering profile data, spatial element data (graphics and attributes), text data, map data and object data that can be made into maps. The inspection method is manual comparison. This method is labor-intensive, and inspectors should have a strong sense of responsibility to find mistakes. Manual inspection and manual audit work content is the same. The system requires that manual inspection and manual audit must be completed by different personnel, strengthen data inspection, and try to eliminate errors caused by human factors.

Figure 5-2 Normalized Data Warehouse Process

5) System archive check, which is used to check the non-empty fields of receipt data. After the system archive inspection passes, the warehousing data can be archived and saved in the database.

After testing, the data warehousing work is carried out in strict accordance with the data warehousing process. The consistency between the data in the aviation geophysical database and the original data file before storage can reach 100%.

This process separates the warehousing data from the database data, establishes a data acquisition database (referred to as "acquisition database"), and temporarily stores the data to be warehousing in the acquisition database. All kinds of quality inspection, editing or deleting operations are carried out on the incoming data in the collection library until it meets the requirements of data warehousing quality, and it is filed into the database (other users except the database administrator have no right to edit or delete the data entered into the database), which ensures the consistency and integrity of the database data and provides a guarantee for improving the overall quality of the aviation geophysical database.

Second, regularize the data checking method.

In the past 50 years, aerial geophysical exploration has obtained a large number of basic data and achievement data, which has played an increasingly important role in basic geoscience research, oil and gas resources evaluation and other fields. People pay more and more attention to solving geological problems by using airborne geophysical data, and also want to know the source and quality of the data used (such as measurement age, measurement method, instrument accuracy, flying height, positioning accuracy, data processing method, etc.). ) to evaluate the credibility of problem solving. This is exactly what the builders of this information system want to provide to users. History has become a fait accompli, and a lot of information related to data quality, such as the accuracy of measuring instruments before digital recording, flying altitude and the positioning accuracy of many items, is now available.

The deficiency of the past proves the progress of the present, and the purpose of this information system construction is to respect history and try our best to adapt to the future technological development. Therefore, according to the actual situation of the data, the regularization method of the validity check of the incoming data is put forward, which solves the quality check problem of the data with incomplete information in different years.

Conventionally, the validity check code of the field to be checked in each database table is written directly in the software code.

The Construction of Aerogeophysical Information System

This system uses regularization method to check the incoming data. After the database structure design is completed, the checking rules for the correctness of the incoming data are formulated for each field in each database table, and the dynamic checking rule table is established. The checking function is written for different checking rules, and the checking rules for the database fields to be checked are obtained from the database to check the incoming data. The code implementation example of regularization method is as follows:

The Construction of Aerogeophysical Information System

The traditional verification method is adopted for system verification, and the code amount is about 15345 lines (Table 5-6). The workload of code development is large and the flexibility is poor, which is not conducive to the later code maintenance and expansion. For example, after adding a table or table check field, you need to modify and compile the code. However, the code of regularization method in this system is only 495 lines (Table 5-6), which is only 3.22% of the code of traditional inspection method, and there is no need to modify the code after adding tables or adding inspection fields to tables. When storing data, users can directly modify the inspection rule table according to actual needs.

Table 5-6 Comparison Table of Code Quantity of Two Implementation Methods of System Check