Plot-and-Ask: Multimodal Local LLMs in Smart Environments for Visual IoT Analytics

Abstract

Smart environments generate continuous heterogeneous sensor data that require real-time processing to support occupant services and decision-making. Existing approaches either compromise data sovereignty through cloud dependencies or employ rigid rule-based systems that lack natural-language interaction. This paper presents a local multimodal analytics architecture for smart indoor environments that combines edge-resident Multimodal Large Language Models (MLLMs) with a Knowledge Graph (KG) policy layer. The proposed plot-and-ask workflow couples automatic time-series visualization with vision-enabled language interpretation, intended to let non-expert users explore indoor sensing data without programming expertise. The system supports natural-language-to-SQL querying, plot generation with KG-derived comfort bands, and multimodal plot interpretation while keeping all processing on local hardware. We evaluate the framework in a two-week indoor air quality deployment across office, hallway, and kitchen locations using four local models: Gemma 3 4B, Ministral 3 3B, Qwen 3 4B, and Granite 3 2B. All four achieved 100% accuracy for structured query synthesis and plot generation. For plot interpretation, Ministral 3 3B and Gemma 3 4B best balanced accuracy and latency, reaching 100% and 96.67% accuracy with 5.84 s and 5.34 s end-to-end latency, respectively. A KG-grounding ablation showed the largest gain for Qwen 3 4B (+25.0 pp), with latency reduced from 27.98 s to 15.21 s, and a supplementary indoor-scene safety check confirms that the same models support direct image understanding. Together, these results show that plot-based summarization and lightweight KG grounding improve local multimodal IoT analytics under edge constraints. Overall, hybrid KG-MLLM architectures can support privacy-preserving visual IoT analytics at the edge while being designed for interactive exploration by non-expert users.