Quality-Assured Predictive Threat Intelligence in Cloud–Lakehouse Systems: Bayesian Statistical Learning and AI-Augmented Risk Monitoring for Data-Limited Environments

Abstract

Ensuring reliable and trustworthy cyber-threat detection in modern cloud–lakehouse infrastructures is increasingly challenging due to sparse labeling, heterogeneous telemetry, and rapidly shifting behavioral patterns. This work introduces Quality-Assured Predictive Threat Intelligence, a Bayesian–AI hybrid framework designed to deliver calibrated, explainable, and continuously validated risk monitoring in data-limited environments. The framework integrates hierarchical Bayesian statistical learning with foundation-model–based feature enrichment to infer threat probabilities under uncertainty, while generative imputations and contextual embeddings strengthen signal fidelity when data is incomplete or noisy. A built-in quality assurance (QA) layer provides systematic validation of data quality, model stability, drift resilience, and risk-score calibration through probabilistic diagnostics, posterior predictive checks, and automated performance gates. Streaming lakehouse pipelines support real-time inference, enabling early identification of cloud-native threats such as identity misuse, lateral movement, and anomalous service interactions. Empirical evaluations across hybrid synthetic–real telemetry show that the QA-enhanced Bayesian–AI approach improves reliability, reduces false positives, and maintains predictive robustness under distributional drift and label scarcity. This framework offers a transparent, auditable, and scalable pathway for operationalizing high-assurance threat intelligence within cloud–lakehouse ecosystems