Modality-Resilient Multimodal Earth-Observation Foundation Models under Missing and Corrupted Sensors

Anna Kalaitzis Pollan

doi:10.58776/ijitcsa.v4i2.259

Authors

Anna Kalaitzis Pollan Istanbul Okan University

DOI:

https://doi.org/10.58776/ijitcsa.v4i2.259

Keywords:

Earth observation foundation models, multimodal learning, missing modality, corrupted modalities, SAR-optical fusion, uncertainty estimation, robustness benchmarking, remote sensing

Abstract

Multimodal Earth-observation (EO) foundation models increasingly rely on the joint use of optical imagery, synthetic aperture radar (SAR), elevation, and auxiliary geospatial signals to support land-cover mapping, biomass estimation, damage assessment, and change detection. Yet the empirical regime in which such models are usually trained and reported remains overly optimistic: modalities are commonly assumed to be synchronised, complete, and clean. This assumption is not defensible for real deployments. Optical observations are routinely obscured by clouds, SAR coherence and interferometric products can be unavailable or decorrelated, spatial alignment across sources is imperfect, and large geospatial archives inevitably contain broken tiles, missing channels, or temporally inconsistent acquisitions. Starting from the released M3LEO dataset and framework, which already expose multimodal EO learning at continental scale, this paper reconstructs the underlying research direction toward a more technically urgent problem: robust multimodal foundation learning under missing or corrupted modalities. We formalise a modality-resilient framework, RAMEO, that combines modality-specific token encoders, reliability-aware gated fusion, masked cross-modal reconstruction, corruption-aware consistency learning, and calibrated uncertainty estimation. We also define a rigorous evaluation protocol over geographically disjoint splits, structured missingness patterns, and modality-specific corruptions. This paper done a dataset-and-benchmark paper and reconstruction distinguishes carefully between source-anchored evidence reported for M3LEO and new robustness analyses that are specified as executable experiments. The result is therefore technically coherent, and reproducible, while remaining honest about what has and has not yet been empirically established.

References

. M. Allen, F. Dorr, J. A. Gallego-Mejia, L. Martínez-Ferrer, A. Jungbluth, F. Kalaitzis, and R. Ramos-Pollán, “M3LEO: A multi-modal, multi-label Earth observation dataset integrating interferometric SAR and multispectral data,” in NeurIPS 2024 Track on Datasets and Benchmarks, 2024, doi: 10.52202/079017-3324.

. spaceml-org, “M3LEO repository,” GitHub, 2026. [Online]. Available: https://github.com/spaceml-org/M3LEO

. Y. Wang et al., “SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation,” IEEE Geosci. Remote Sens. Mag., vol. 11, no. 3, pp. 98–106, 2023, doi: 10.1109/MGRS.2023.3281651.

. J. Stewart et al., “SSL4EO-L: Datasets and foundation models for Landsat imagery,” arXiv:2310.00705, 2023.

. M. Schmitt, L. H. Hughes, C. Qiu, and X. X. Zhu, “SEN12MS – a curated dataset of georeferenced multi-spectral Sentinel-1/2 imagery for deep learning and data fusion,” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci., vol. IV-2/W7, pp. 153–160, 2019, doi: 10.5194/isprs-annals-IV-2-W7-153-2019.

. M. Schmitt, L. H. Hughes, and X. X. Zhu, “The SEN1-2 dataset for deep learning in SAR-optical data fusion,” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci., vol. IV-1, pp. 141–146, 2018, doi: 10.5194/isprs-annals-IV-1-141-2018.

. F. Bastani et al., “SatlasPretrain: A large-scale dataset for remote sensing image understanding,” arXiv:2211.15660, 2023.

. J. Jakubik et al., “Foundation models for generalist geospatial artificial intelligence,” arXiv:2310.18660, 2023.

. Y. Wang, C. M. Albrecht, N. A. A. Braham, L. Mou, and X. X. Zhu, “Self-supervised learning in remote sensing: A review,” IEEE Geosci. Remote Sens. Mag., vol. 10, no. 4, pp. 213–247, 2022, doi: 10.1109/MGRS.2022.3198244.

. Tao, J. Qi, M. Guo, Q. Zhu, and H. Li, “Self-supervised remote sensing feature learning: Learning paradigms, challenges, and future works,” IEEE Trans. Geosci. Remote Sens., vol. 61, pp. 1–26, 2023, doi: 10.1109/TGRS.2023.3276853.

. X. Guo et al., “SkySense: A multi-modal remote sensing foundation model towards universal interpretation for Earth observation imagery,” in Proc. CVPR, 2024, pp. 27672–27682.

. G. Tseng et al., “Galileo: Learning global and local features of many remote sensing modalities,” in Proc. ICML, 2025. [Online]. Available: https://openreview.net/forum?id=gqZO3eSZRy

. X. Sun et al., “Neural plasticity-inspired multimodal foundation model for Earth observation,” arXiv:2403.15356, 2024.

. Phiri et al., “Sentinel-2 data for land cover/use mapping: A review,” Remote Sens., vol. 12, no. 14, p. 2291, 2020, doi: 10.3390/rs12142291.

. L. Dong et al., “Application of convolutional neural network on lei bamboo above-ground-biomass estimation using WorldView-2,” Remote Sens., vol. 12, no. 6, p. 958, 2020, doi: 10.3390/rs12060958.

. Ayala, R. Sesma, C. Aranda, and M. Galar, “A deep learning approach to an enhanced building footprint and road detection in high-resolution satellite imagery,” Remote Sens., vol. 13, no. 16, p. 3135, 2021, doi: 10.3390/rs13163135.

. C. Cavallo et al., “Continuous monitoring of the flooding dynamics in the Albufera wetland (Spain) by Landsat-8 and Sentinel-2 datasets,” Remote Sens., vol. 13, no. 17, p. 3525, 2021, doi: 10.3390/rs13173525.

. G. Nico, M. Pappalepore, G. Pasquariello, A. Refice, and S. Samarelli, “Comparison of SAR amplitude vs. coherence flood detection methods – a GIS application,” Int. J. Remote Sens., vol. 21, no. 8, pp. 1619–1631, 2000, doi: 10.1080/014311600209931.

. M. Chini et al., “SAR coherence and polarimetric information for improving flood mapping,” in Proc. IGARSS, 2016, pp. 7577–7580, doi: 10.1109/IGARSS.2016.7730976.

. M. Watanabe et al., “Detection of damaged urban areas using interferometric SAR coherence change with PALSAR-2,” Earth Planets Space, vol. 68, p. 131, 2016, doi: 10.1186/s40623-016-0513-2.

. Olesk et al., “Interferometric SAR coherence models for characterization of hemiboreal forests using TanDEM-X data,” Remote Sens., vol. 8, no. 9, p. 700, 2016, doi: 10.3390/rs8090700.

. J. Fielding et al., “Surface deformation related to the 2019 Mw 7.1 and 6.4 Ridgecrest earthquakes in California from GPS, SAR interferometry, and SAR pixel offsets,” Seismol. Res. Lett., vol. 91, no. 4, pp. 2035–2046, 2020, doi: 10.1785/0220190302.

. T. Strozzi et al., “Satellite SAR interferometry for the improved assessment of the state of activity of landslides: A case study from the Cordilleras of Peru,” Remote Sens. Environ., vol. 217, pp. 111–125, 2018, doi: 10.1016/j.rse.2018.08.014.

. M. A. Richards, “A beginner’s guide to interferometric SAR concepts and signal processing,” IEEE Aerosp. Electron. Syst. Mag., vol. 22, no. 9, pp. 5–29, 2007, doi: 10.1109/MAES.2007.4350281.

. Moreira et al., “A tutorial on synthetic aperture radar,” IEEE Geosci. Remote Sens. Mag., vol. 1, no. 1, pp. 6–43, 2013, doi: 10.1109/MGRS.2013.2248301.

. Y. Yamaguchi, Polarimetric SAR Imaging: Theory and Applications. Boca Raton, FL, USA: CRC Press, 2020, doi: 10.1201/9781003049753.

. Y. Chen and L. Bruzzone, “Self-supervised SAR-optical data fusion of Sentinel-1/-2 images,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–11, 2022, doi: 10.1109/TGRS.2021.3128072.

. M. Kampffmeyer, A.-B. Salberg, and R. Jenssen, “Urban land cover classification with missing data modalities using deep convolutional neural networks,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 11, no. 6, pp. 1758–1768, 2018, doi: 10.1109/JSTARS.2018.2834961.

. Z. Zheng et al., “Deep multisensor learning for missing-modality all-weather mapping,” ISPRS J. Photogramm. Remote Sens., vol. 174, pp. 11–23, 2021, doi: 10.1016/j.isprsjprs.2021.01.015.

. M. Zhao et al., “Towards robust classification of multi-view remote sensing images with partial data availability,” Remote Sens. Environ., vol. 313, p. 114112, 2024, doi: 10.1016/j.rse.2024.114112.

. Y. Zhou et al., “Remote sensing meta modal representation for missing modality land cover mapping: From EarthMiss dataset to MetaRS method,” Remote Sens. Environ., 2026, doi: 10.1016/j.rse.2025.115132.

. L. Huang et al., “Language-guided visual prompt compensation for multi-modal remote sensing image classification with modality absence,” in Proc. ACM Multimedia, 2024, pp. 4936–4945, doi: 10.1145/3664647.3681563.

. J. Kang et al., “DisOptNet: Distilling semantic knowledge from optical images for weather-independent building segmentation,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–14, 2022, doi: 10.1109/TGRS.2022.3141769.

. Y. Liu et al., “Segmentation of remote sensing data with missing modalities through prototype knowledge distillation,” in Proc. IGARSS, 2024.

. L. Zhang, X. Liu, J. Sun, and Q. Wu, “Deep multimodal learning with missing modality: A survey,” Trans. Mach. Learn. Res., 2026. [Online]. Available: https://openreview.net/pdf?id=tc7RFcx4hT

. K. He et al., “Masked autoencoders are scalable vision learners,” in Proc. CVPR, 2022, pp. 16000–16009.

. Radford et al., “Learning transferable visual models from natural language supervision,” in Proc. ICML, 2021, pp. 8748–8763.

. Dosovitskiy et al., “An image is worth 16×16 words: Transformers for image recognition at scale,” in Proc. ICLR, 2021.

. O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Proc. MICCAI, 2015, pp. 234–241, doi: 10.1007/978-3-319-24574-4_28.

. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.

. L. McInnes, J. Healy, N. Saul, and L. Großberger, “UMAP: Uniform manifold approximation and projection,” J. Open Source Softw., vol. 3, no. 29, p. 861, 2018, doi: 10.21105/joss.00861.

. J. Kellndorfer et al., “Global seasonal Sentinel-1 interferometric coherence and backscatter data set,” Sci. Data, vol. 9, p. 73, 2022, doi: 10.1038/s41597-022-01189-6.

. D. Zanaga et al., “ESA WorldCover 10 m 2020 v100,” 2021, doi: 10.5281/zenodo.5571936.

. M. Santoro and O. Cartus, “ESA Biomass Climate Change Initiative (Biomass_cci): Global datasets of forest above-ground biomass,” 2023, doi: 10.5285/95913ffb6467447ca72c4e9d8cf30501.

. DiMiceli et al., “MOD44B MODIS/Terra vegetation continuous fields yearly L3 global 250m SIN grid V006,” NASA EOSDIS LP DAAC, 2015, doi: 10.5067/MODIS/MOD44B.006.

. T. G. Farr et al., “The Shuttle Radar Topography Mission,” Rev. Geophys., vol. 45, no. 2, 2007, doi: 10.1029/2005RG000183.

. M. Allen et al., “Large scale masked autoencoding for reducing label requirements on SAR data,” arXiv preprint, 2023.

. M. Allen et al., “Few-shot learning on global multimodal embeddings for earth observation tasks,” arXiv preprint, 2023.

. J. A. Gallego-Mejia et al., “Exploring DINO: Emergent properties and limitations for synthetic aperture radar imagery,” arXiv preprint, 2023.

. L. Martínez-Ferrer et al., “Exploring generalisability of self-distillation with no labels for SAR-based vegetation prediction,” arXiv preprint, 2023.