Actually now that I think about it, LLM’s are decoder only these days. But decoders and encoders are architecturally very similar. You could probably cut off the “head” of the decoder, make a few fully connected layers, and fine tune them to provide a score.