The generators for qudit clifford group are give here https://arxiv.org/abs/1911.08162 This is more concise than the paper in the comment and takes care of subtleties better.
Here is a short GAP program that defines these generators for any dimension:
TestA:=function(p)local Ig,Zg,Xg,Hg,Pg,grp,cen,sgrp,scen,C;
Ig:=IdentityMat(p);
Zg:=DiagonalMat(List([0..p-1],x->E(p)^x));
Xg:=IdentityMat(p){\mod([0..p-1]+1,p)+1};
Hg:=List([0..p-1],x->List([0..p-1],y->E(p)^(x*y)))/ER(p);
Pg:=DiagonalMat(List([0..p-1],x->E(2*p)^(x*(x-\mod(p,2)))));
grp:=Group([Hg,Pg,Zg]);cen:=Center(grp);sgrp:=Size(grp);scen:=Size(cen);
Print(" |G| = ",String(sgrp,-5));Print(" |Cen(G)| = ",String(scen,-5));Print(" |G/Cen(G)| = ",String(sgrp/scen,-5));Print("\n");
if(p=4)then
Print("Z=\n");PrintArray(Zg);
Print("X=\n");PrintArray(Xg);
Print("H=\n");PrintArray(Hg);
Print("P=\n");PrintArray(Pg);
C:=KroneckerProduct([[1,0],[0,1]],[[1,1],[1,-1]]/ER(2));C:=C{[1,3,2,4]}{[1,3,2,4]};
Zg:=C^-1*Zg*C;
Xg:=C^-1*Xg*C;
Hg:=C^-1*Hg*C;
Pg:=C^-1*Pg*C;
grp:=Group([Hg,Pg,Zg]);
Print(" all elements are monomial? ",ForAll(Elements(grp),IsMonomialMatrix),"\n");
fi;
return grp;end;
Running in GAP gives : (takes seconds)
gap> g2:=TestA(2);;
|G| = 192 |Cen(G)| = 8 |G/Cen(G)| = 24
gap> g3:=TestA(3);;
|G| = 2592 |Cen(G)| = 12 |G/Cen(G)| = 216
gap> g4:=TestA(4);;
|G| = 6144 |Cen(G)| = 8 |G/Cen(G)| = 768
Z=
[ [ 1, 0, 0, 0 ],
[ 0, E(4), 0, 0 ],
[ 0, 0, -1, 0 ],
[ 0, 0, 0, -E(4) ] ]
X=
[ [ 0, 1, 0, 0 ],
[ 0, 0, 1, 0 ],
[ 0, 0, 0, 1 ],
[ 1, 0, 0, 0 ] ]
H=
[ [ 1/2, 1/2, 1/2, 1/2 ],
[ 1/2, 1/2*E(4), -1/2, -1/2*E(4) ],
[ 1/2, -1/2, 1/2, -1/2 ],
[ 1/2, -1/2*E(4), -1/2, 1/2*E(4) ] ]
P=
[ [ 1, 0, 0, 0 ],
[ 0, E(8), 0, 0 ],
[ 0, 0, -1, 0 ],
[ 0, 0, 0, E(8) ] ]
all elements are monomial? true
gap> g5:=TestA(5);;
|G| = 30000 |Cen(G)| = 10 |G/Cen(G)| = 3000
As a check, the calculated $G/Cen(G)$ match the results in this paper https://arxiv.org/abs/1810.10259
The GAP code also calculates the generators in an alternate basis. This is based on this
https://arxiv.org/abs/1202.3559 where the entire clifford group consists of monomial matrices (normally only the pauli's are). This can be done when $p$ is a square.
As a final note, when $p$ is prime $H$ and $P$ seem enough (so $Z$ is redundant in the generators).