Kwaipilot

/

KAT-Dev

like 196

Follow

Kwaipilot 375

hey hi @shunxing1234 and @arieldeng

Congrats on the impressive KAT-Coder results!
I'm trying to replicate your HumanEval evaluation to benchmark my own models. Could you clarify:

1. What temperature did you use for HumanEval (96.3% in Table 1)
2. Is this pass@1 or pass@k? If pass@k, what's k and n?
3. Did you use the Python HumanEval or MultiPL-E (Rust/other language)?