Remark: I am NOT a lawyer and following are opinions.
Collective punishment is, to my opinion, lowest anyone can go. The event you describe certainly reminds this:
https://www.youtube.com/watch?v=rbzJTTDO9f4&t=1s
There are few very inappropriate things here.
1) There must be a case, i.e. disciplinary hearing, for each individual (or for small groups of people if there is evidence that they collabrated in this act)
2)Presumption of guilt. They accuse you of cheating and you are asked to prove you are innocent. This is absurd. Presumption of innocence, i.e. innocent until proven otherwise, is a fundamental legal principle. It is the core of modern legal system.
3) "My lecturer said each student have a chance to write a letter or affidavit to share their side of the story but he said they are a few chances to win this case." They seem to decide what proportion of letters are going to "win" before recieving any. Again, very inappropriate.
What I would suggest, can't stress this enough I am not a lawyer:
1) Try to talk with higher administration about this. Don't skip levels. Communicate your worries including above that you agree. You may say it is unjust. You may say it is unmethodical. You may say this is defamation. Start with mathematics department head. Their answers or actions does not satisfy you try to contact the faculty head. Even that does not do it try to get in touch with the respective vice-president of the university. That probably would be the one in charge of student affairs or education.
2) Consider your legal options. I don't know your country's legal system. There seem to be enough to ask a lawyer if this would count as defamation or damages. If they don't have any evidence against you, they have no justifiable reason to defame you or damage your academic standing. Legal advice should apply to the letter you are considering to submit. If you have access to a lawyer or a law student, you might want to explain what is going on and possibly, at the very least, run the letter by them. Ideally, you would have a lawyer throught this phase and act within his/her advices.
edit: I forgot to point out. They are the sole judge on their accusations. That also is quite absurd.