Three ways to spend compute at answer time: parallel, sequential, search

Three ways to spend compute at answer time: parallel, sequential, search A workflow diagram generated by Archify. 01 / Parallel: best-of-N, self-consistency 02 / Sequential: one long chain, self-correction 03 / Search: beam, tree with a PRM Sample N · independent, T > 0 · Parallel: best-of-N, self-consistency Sample N independent, T > 0 Vote or score · majority, ORM, verifier · Parallel: best-of-N, self-consistency Vote or score majority, ORM, verifier Needs · extractable answers · Parallel: best-of-N, self-consistency Needs extractable answers Cost · × N, runs in parallel · Parallel: best-of-N, self-consistency Cost × N, runs in parallel Limit · selection saturates · Parallel: best-of-N, self-consistency Limit selection saturates One long chain · thinking tokens · Sequential: one long chain, self-correction One long chain thinking tokens Check and fix · wait, try again · Sequential: one long chain, self-correction Check and fix wait, try again Needs · an RL-trained model · Sequential: one long chain, self-correction Needs an RL-trained model Cost · latency grows with length · Sequential: one long chain, self-correction Cost latency grows with length Limit · overthinking, context · Sequential: one long chain, self-correction Limit overthinking, context Expand each step · several branches · Search: beam, tree with a PRM Expand each step several branches PRM scores steps · keep the best branches · Search: beam, tree with a PRM PRM scores steps keep the best branches Needs · a PRM, stepwise answers · Search: beam, tree with a PRM Needs a PRM, stepwise answers Cost · a PRM call per step · Search: beam, tree with a PRM Cost a PRM call per step Limit · PRMs get exploited · Search: beam, tree with a PRM Limit PRMs get exploited continue next step Legend cost grows with generate tokens select or score what it needs limits