Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop
American Journal of Applied Scientific Research
Volume 5, Issue 1, March 2019, Pages: 6-16
Received: Jan. 24, 2019;
Accepted: Mar. 4, 2019;
Published: Mar. 28, 2019
Views 97 Downloads 10
Li Chaokui, National-Local Joint Engineering Laboratory of Geospatial Information Technology, Hunan University of Science and Technology, Xiangtan, China
Zhao Yanan, National-Local Joint Engineering Laboratory of Geospatial Information Technology, Hunan University of Science and Technology, Xiangtan, China
Xiao Keyan, Institute of Mineral Resources, Chinese Academy of Geological Sciences, Beijing, China
Chen Jianhui, National-Local Joint Engineering Laboratory of Geospatial Information Technology, Hunan University of Science and Technology, Xiangtan, China
With the emergence of big data of TB and PB geological and mineral resources, the storage of large geological data has become a worldwide problem puzzling geologists. The traditional storage and service model of geological data is facing a great challenge. For example, when the scale of data increases dramatically, general relational database can not solve the problem of insufficient scalability, stability and efficiency of database system. In response to the above problems, this paper proposes a new method of geological and mineral data storage based on cloud computing environment combined with hadoop. Taking the mineral resources potential evaluation data of Chongqing as the research object, The proposed method in this paper is compared with the traditional Oracle database storage method in data storage experiments: (1) Small file optimization comparative experiment; (2) Hadoop and Oracle comparative experiment. The performance of writing operation, memory occupancy, data import and data export are tested in different way, and the comparison chart of performance is given. The experimental results show that the new storage method proposed in this paper is more efficient than the traditional method. At the same time, it effectively overcomes the problem of small file storage in Hadoop storage. The research results provide a new technical for the storage and management of geological and mineral data all over the country.
Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop, American Journal of Applied Scientific Research.
Vol. 5, No. 1,
2019, pp. 6-16.
Zhu, Y. X. and Hou, J. G., 2014. Research on geological data storage and management based on large data// Jiangsu province surveying and mapping geographic information society Proceedings of the 2014 academic annual meeting.
Chen, J. P., Li, J., Cui, N., and Yu, P. P., 2015. The construction and application of geological cloud under the big data background. Geological Bulletin of China, 34(7):1260-1265.
Qian, X., and Zhang, D. Y., 2014. Application of geological exploration in geological prospecting. Luoyang: Journal of Henan Science and Technology, 2014(4):56-56.
Zhang, B., 2009. Based on ArcGIS Fugu county geological disaster database established and the evaluation of vulnerable areas research. Xi'an Chang'an University.
Pu, K., 2009. Design and implementation of multi-source geological spatial database storage management system. Chengdu: University of Electronic Science and Technology of China.
Liu, C. J., 2011. The research and development of the comprehensive geological database management system. Changsha: Central South University.
Zhai, Y. D., 2011. The reliability of the Hadoop distributed file system (HDFS) research and optimization. Wuhan: Huazhong University of Science and Technology.
Li, J., Chen, J. P. and Wang, X., 2015. A study of the storage technology of geological big data. Geological Bulletin of China, 2015, 34(8): 1589-1594.
Hao, S. K., 2012. Brief Analysis of the Architecture of Hadoop HDFS and MapReduce. Designing Techniques of Posts and Telecommuni, 2012(7):37-42.
Chen, K. and Zheng, W. M., 2009. Cloud computing: System instances and current research. Journal of Software, 20(5):1337-1348.
Armbrust, M., Fox, A., Griffith, R., et al. 2010. A view of cloud computing. Communications of the ACM, 53(4):50-58.
Ma, H. T., 2015. Based on the nested HBase data storage system design and implementation. Hangzhou: Zhejiang University.
Li, X. A., 2005. Future development of the database technology. Journal of Shangdong College of Electric Power, 8(2):40-43.
Cui, J. S., 2014. The Present Situation and Tendency in the Integration of Remote Sensing and Geographic Information System. TopCapital, 2014(10):247-247.
Li, C. K., Yan, W. Y. and Xiao, K. Y. 2015. The construction of geological database with MapGIS and Oracle. Geological Bulletin of China, 34(7):1359-1364.
Shvachko, K., Kuang, H. and Radia, S., 2010. The hadoop distributed filesystem. In Mass Storage Systems and Technologies (MSST), IEEE 26th Symposium. 2010:1-10.
Li, M., Cao, S. and Qin, Z. G., 2016. Storage Optimization Method of Small Files Based on Hadoop. Chengdu: Journal of University of Electronic Science and Technology of China, 2016(1):141-145.
Henzinger, T., Jhala, R., Majumdar, R., et al. 2002. Lazy Abstraction/ /POPL’02 Proc. of the 29th ACM SIGPLAN-SIGACT symposium onPrinciples of programming languages, New York, NY, USA: ACM Press.
Godefroid P., 1997. Model Checking for Programming Languages using VeriSoft//Proc. of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, New York, NY, USA: ACM Press.
Chang, F., Dean, J., Ghemawat, S., et al. 2008. Bigtable: A Distributed Storage System for Structured Data. Acm Transactions on Computer Systems, 26(2):205-218.