Transformers with knowledge integration for Earth system forecasting
Project descriptionTransformer architectures implementing self-attention have been very successful in time series data, like NLP and forecasting for renewable energy [1,2]. They are currently also going to dominate the area of computer vision, like object detection, video classification, image classification and image generation. Recently, they have been applied to space-time problems in Earth System Science, like weather forecasting, by introducing “cuboid attention”  or hierarchical aggregation . The results show that transformer architecture offers a way to generate self-supervised models of complex dynamic, natural systems. It is still open whether such models are appropriate for medium-sized training data and how one can integrate domain knowledge. Domain knowledge can be either integrated by the loss function, hybrid modelling (see ), or task-dependent attention mechanisms that might be guided by (known or estimated) causal relationships. Another challenge is to make such models applicable to domains that show a variety of modalities, like time-series data, together with remote sensing and SAR data . Finally, one can study such models considering interpretability to generate insights from such models into the dynamic system itself.
This project aims at studying such models in the context of forecasting soil moisture and its effects on ecosystems [cf. e.g. 6-8], where sparsity, representativeness of data, and distribution shifts play a role. The PhD student, jointly supervised and associated with the Computer Vision Group, Friedrich-Schiller-University Jena, and the MPI for Biogeochemistry, Department of Biochemical Integration, is expected to study transformer architectures in combination with domain knowledge for the mentioned forecasting challenges.
RequirementsApplications to the IMPRS-gBGC are open to well-motivated and highly-qualified students from all countries. Prerequisites for this PhD project are:
References: Padilha et al.: Transformer-Based Hybrid Forecasting Model for Multivariate Renewable Energy. Applied Sciences. 2022, 12, 10985. https://doi.org/10.3390/app122110985
 Rao et al.: Transformer-based power system energy prediction model. 2020 IEEE 5th Information Technology and Mechatronics Engineering Conference (ITOEC).
 Gao et al.: Earthformer: Exploring Space-Time Transformers for Earth System Forecasting. NeurIPS 2022.
 Bi et al.: Pangu-Weather: A 3D High-Resolution System for Fast and Accurate Global Weather Forecast. https://doi.org/10.48550/arXiv.2211.02556
 Rahaman et al.: A General-Purpose Neural Architecture for Geospatial Systems. AI + HADR Workshop at 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
 C. Requena-Mesa, V. Benson, M. Reichstein, J. Runge, J. Denzler, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2021), pp. 1132-1142.
 S. O, R. Orth, Global soil moisture data derived through machine learning trained with in-situ measurements. Sci. Data 8, 1-14 (2021).
 B. Kraft et al., Identifying Dynamic Memory Effects on Vegetation State Using Recurrent Neural Networks. Front. Big Data 2 (2019).
Cuboid attention strategy (from )
>> more information about the IMPRS-gBGC + application