CLS-Guided Multimodal Fusion for Image Captioning with LSTM and T5 Decoders

Turning visual evidence into coherent sentences depends on how visual and linguistic representations are combined before decoding. We present two decoder-specific generation strategies, CLS-LSTM and CLS-T5, built on a common CLS-guided Swin–BERT multimodal backbone. Swin-Base extracts a 7×7 grid of 49 spatial feature vectors, each with 1024 dimensions, which are mapped to 512 dimensions through a trainable linear projection initialized from principal components fitted exclusively on the training split. Under a leave-one-out training strategy, one caption serves as the generation target, while the remaining four captions are concatenated and encoded by BERT. The final-layer BERT [CLS] representation is used as the query in multihead cross-attention over the projected visual features, producing a global multimodal conditioning representation. Batch-level modality dropout exposes both decoder configurations to the image-only condition used during validation and testing. CLS-LSTM uses the fused representation for global conditioning, while the previous hidden state of its first LSTM layer queries coverage-aware Bahdanau attention at each decoding step. CLS-T5 projects the fused representation to the visual feature dimension, injects it into the projected visual feature sequence, refines the conditioned sequence using a single Transformer encoder layer, and supplies the resulting visual memory directly to the T5-small decoder while bypassing the original T5 encoder. When auxiliary text is unavailable, CLS-LSTM replaces the BERT [CLS] query with a learned null-text query, whereas CLS-T5 bypasses the auxiliary text fusion branch and conditions the visual features using a learned fusion bias. We evaluate both decoder configurations on the Karpathy splits of Flickr30K and MS COCO 2014 under image-only inference. On Flickr30K, across three random seeds, CLS-LSTM achieves mean BLEU-4 and CIDEr scores of 23.97±1.05 and 50.92±1.08, respectively, whereas CLS-T5 achieves 27.00±1.09 and 64.84±2.86. On MS COCO 2014 for seed 0, CLS-LSTM obtains BLEU-4 and CIDEr scores of 32.16 and 105.18, while CLS-T5 obtains 35.38 and 117.15. These results show that the shared CLS-guided framework supports both recurrent and Transformer-based caption generation while preserving image-only inference.