A symmetric informationally complete positive-operator-valued measure (SIC POVM) is a special POVM that is composed of d^2 subnormalized pure projectors with equal pairwise fidelity. It may be considered a fiducial POVM for reasons of its high symmetry and high tomographic efficiency. Most known SIC POVMs are covariant with respect to the Heisenberg-Weyl (HW) group. We show that in prime dimensions the HW group is the unique group that may generate a SIC POVM. In particular, in prime dimensions not equal to three, each group covariant SIC POVM is covariant with respect to a unique HW group. In addition, the symmetry group of the SIC POVM is a subgroup of the Clifford group, which is the normalizer of the HW group. Hence, two SIC POVMs covariant with respect to the HW group are unitarily equivalent if and only if they are on the same orbit of the Clifford group. In dimension three, each group covariant SIC POVM may be covariant with respect to three or nine HW groups, and the symmetry group of the SIC POVM is a subgroup of at least one of the Clifford groups of these HW groups respectively. There may exist two or three orbits of equivalent SIC POVMs for each group covariant SIC POVM, depending on the order of its symmetry group.