FishSpeech S2 Pro streaming code (380ms TTFA, tested on RTX 5090)

So... uh... yes I did a lot of debugging and learning and I'm your average webdev, not ML engineer so my apologies for cursed code 🤣

https://github.com/fishaudio/fish-speech/pull/1193/changes

Streaming should work end-to-end with low TTFA (~400ms until first audio chunk on Arch Linux, RTX 5090, NVIDIA driver 595.45.04, 9950x3D); there’s still work to do on memory, TTFA, and longer prompts.

Here's some ideas:

1. Figure out how to properly torch.compile, right now it just recompiles after warmup on smoke e2e test; and every recompile takes like 6 minutes.
2. Stream tokens into vocoder with a schedule (per lengyue), not one big chunk.
3. Cut memory use more and improve TTFA (profile, smaller first chunk, CUDA graphs).
4. Support longer prompts (~30–50 words) without OOM, possibly #1 should fix it.

I got a tiny bit of help from the maintainer, and so my solution while not really that impressive, should enable others to plumb into this direction.

This is an approximate diagram what is actually happening:

https://preview.redd.it/hgwrc6azb5pg1.png?width=845&format=png&auto=webp&s=29995a0a8ee8a25f2ba2410e1544ac15d9d85ef3

This could be improved. As far as I'm getting DAC can just process tokens on its own with some clever scheduling, and not hold LLM until it actually finishes making PCM chunk 🤷

Anyway, here's my tests.

Without torch.compile TTFA is around 800ms

https://preview.redd.it/1t1en4c0f5pg1.png?width=1622&format=png&auto=webp&s=8199dfc7ff4393ca06144df9a30a801101c1a2fa

With torch.compile (380ms) + some logs / instrumentation

https://preview.redd.it/b7rkejvan5pg1.png?width=2547&format=png&auto=webp&s=3dedb4f7745102b5b1aa77c06da897cfab6d0a73

I'm testing my own branch and found some issues but the main streaming code should be working. There's also a lot of unrelated things, kinda QoL updates for adding reference voices, Makefile, tests, etc.