PhD project offered by the IMPRS-gBGC in Jan 2023

Transformers with knowledge integration for Earth system forecasting

Joachim Denzler , René Orth , Markus Reichstein

Project description

Transformer architectures implementing self-attention have been very successful in time series data, like NLP and forecasting for renewable energy [1,2]. They are currently also going to dominate the area of computer vision, like object detection, video classification, image classification and image generation. Recently, they have been applied to space-time problems in Earth System Science, like weather forecasting, by introducing “cuboid attention” [3] or hierarchical aggregation [4]. The results show that transformer architecture offers a way to generate self-supervised models of complex dynamic, natural systems. It is still open whether such models are appropriate for medium-sized training data and how one can integrate domain knowledge. Domain knowledge can be either integrated by the loss function, hybrid modelling (see [1]), or task-dependent attention mechanisms that might be guided by (known or estimated) causal relationships. Another challenge is to make such models applicable to domains that show a variety of modalities, like time-series data, together with remote sensing and SAR data [5]. Finally, one can study such models considering interpretability to generate insights from such models into the dynamic system itself.
This project aims at studying such models in the context of forecasting soil moisture and its effects on ecosystems [cf. e.g. 6-8], where sparsity, representativeness of data, and distribution shifts play a role. The PhD student, jointly supervised and associated with the Computer Vision Group, Friedrich-Schiller-University Jena, and the MPI for Biogeochemistry, Department of Biochemical Integration, is expected to study transformer architectures in combination with domain knowledge for the mentioned forecasting challenges.

Requirements

Applications to the IMPRS-gBGC are open to well-motivated and highly-qualified students from all countries. Prerequisites for this PhD project are:

either Master in Computer Science or related areas with a minor/experience in Ecology or Earth System Science, or Master in Ecology or related areas with a minor/experience in computer science
programming experience in Python
background in machine learning
experience with deep learning models is beneficial but not mandatory
interest in working in an interdisciplinary team of ecologists and computer scientists
Very good oral and written communication skills in English

The Max Planck Society seeks to increase the number of women in those areas where they are underrepresented and, therefore, explicitly encourages women to apply. The Max Planck Society is committed to increasing the number of individuals with disabilities in its workforce and therefore encourages applications from such qualified individuals.

References:

[1] Padilha et al.: Transformer-Based Hybrid Forecasting Model for Multivariate Renewable Energy. Applied Sciences. 2022, 12, 10985. https://doi.org/10.3390/app122110985
[2] Rao et al.: Transformer-based power system energy prediction model. 2020 IEEE 5th Information Technology and Mechatronics Engineering Conference (ITOEC).
[3] Gao et al.: Earthformer: Exploring Space-Time Transformers for Earth System Forecasting. NeurIPS 2022.
[4] Bi et al.: Pangu-Weather: A 3D High-Resolution System for Fast and Accurate Global Weather Forecast. https://doi.org/10.48550/arXiv.2211.02556
[5] Rahaman et al.: A General-Purpose Neural Architecture for Geospatial Systems. AI + HADR Workshop at 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
[6] C. Requena-Mesa, V. Benson, M. Reichstein, J. Runge, J. Denzler, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2021), pp. 1132-1142.
[7] S. O, R. Orth, Global soil moisture data derived through machine learning trained with in-situ measurements. Sci. Data 8, 1-14 (2021).
[8] B. Kraft et al., Identifying Dynamic Memory Effects on Vegetation State Using Recurrent Neural Networks. Front. Big Data 2 (2019).

Cuboid attention strategy (from [3])