Updated:
Take a layer even when the lakeshore feels warm. Inside the St. Beatus Caves, the temperature stays around 8-10°C, and the damp air makes the change immediate. Limestone formations and the underground stream give you plenty to look at without needing a mountain panorama.
The developed visitor route still involves 297 steps, so this is not a step-free rainy-day substitute. Wear shoes with grip and check current opening or weather restrictions before setting off. Allow roughly an hour inside, plus the uphill approach. If the weather clears afterwards, the view back over Lake Thun is a lovely finish.