Compute transition scores on CPU

[benjaminking]

This PR moves the computation of transition scores in SilTranslationPipeline to the CPU. The scores form a large tuple whose size is num_generated_tokens × batch_size × num_beams × vocab_size that can occupy a lot of the GPU's RAM. The tradeoff is that inference now takes 1.5-2x longer, since we're using the CPU.

We should discuss further whether this is a good tradeoff to make.

---

This change is 

[benjaminking]

One upside to the performance hit that I just realized: the EITL team has already been running inference with a batch size of 8 instead of 16, so computing transition scores on the CPU with a batch size of 16 will have comparable performance to computing transition scores on the GPU with a batch size of 8. So, this fix is at least no slower than the status quo.

Also, it looks like we compute transition scores whether or not we're calculating confidence. We may want to change that so to avoid an unnecessary performance hit calculating something that we're just throwing away.

@benjaminking made 1 comment.
Reviewable status: 0 of 1 files reviewed, all discussions resolved.

[benjaminking]

I have updated this branch and changed the approach from always computing transition scores on the CPU to instead attempting to perform the calculations on the GPU and failing over to the CPU on any exception. I have verified that this allows an experiment that consistently produces OOM errors to complete with no issues. This also results in an order-of-magnitude speed up compared to the CPU-only approach.

@benjaminking made 1 comment.
Reviewable status: 0 of 1 files reviewed, all discussions resolved.

[mshannon-sil]

@mshannon-sil reviewed 1 file and all commit messages, and made 1 comment.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on benjaminking).