Confidence 0 after finetuning for empty recognition

[hanshupe007]

I fine tuned parseq and other models to be able to recognize empty "" text, like a blank box. The prediction works as expected but for some reason the confidence score is always 0. Does anyone know why?

I know the usual way would be to use the detection model, but I do the detection separately.

[felixdittrich92]

Hi @hanshupe007 👋

If it's "real" empty then it will be 0 if word else 0.0

-->

  word_values = [
      "".join(self._embedding[idx] for idx in encoded_seq).split("<eos>")[0]
      for encoded_seq in out_idxs.cpu().numpy()
  ]
  # compute probabilties for each word up to the EOS token
  probs = [
      preds_prob[i, : len(word)].clip(0, 1).mean().item() if word else 0.0 for i, word in enumerate(word_values)
  ]

[hanshupe007]

Okay thanks, is this the recommended way to detect real empty images using the recognition model, or any alternative that would also return a probability?

[felixdittrich92]

Mh depends a bit is this a region of interest or actually only used to "skip" ?

[hanshupe007]

The background is that I use another approach (MLLM) to detect cells in a table, and feed the cropped cells to doctr. Some cells are empty for which a confidence would be nice to have.

[felixdittrich92]

Got it 👍

You can try to replace the if word else 0.0 with if len(word) > 0 else 0.0
here:

doctr/doctr/models/recognition/parseq/pytorch.py

Line 429 in 1d7b9b4

preds_prob[i, : len(word)].clip(0, 1).mean().item() if word else 0.0 for i, word in enumerate(word_values)