TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
Paper
• 2404.11912 • Published
• 17
we recognize two memory bottlenecks: model weights and KV cache, and the latter gradually bottleneck(s) as context length increases