Firstly, there is a requirement for the dark matter to be non-baryonic.
There are manifold reasons for this. The most important are the comparison between the universe's matter density implied by measurements of the cosmic microwave background (CMB) compared with the baryonic matter density in the universe implied by measurements of the primordial abundances of helium and deuterium. The former suggests that $\Omega_M \sim 0.3$ (30 per cent of the total energy density of the universe), whilst that latter suggests that $\Omega_b \sim 0.05$. i.e. There seems to be six times as much gravitating matter in the universe than is required to produce the abundances of chemical elements synthesised during the big bang.
Secondly, non-baryonic matter that does not interact strongly with itself or with "normal" matter is required in order to form the galaxies and clusters of galaxies that we see in the universe today. Without that dark matter, it is difficult to understand how the very tiny fluctuations that are seen in the cosmic microwave background can grow into the structures we see in the universe today. If this extra matter were baryonic it would have been evident in the CMB.
Now both these bits of evidence apply to the universe as a whole and it is in any case true that the "luminous matter" that we can see and count-up may still fall short of the amount of baryonic matter that is supposed to be present (though only by a factor of 2 or so). For that reason it is still possible to hypothesise that the dark, gravitating matter in our own Galaxy, that is required to explain the flat rotation curve, could be in the form of something baryonic that does not emit much light. The candidates would be very low-mass stars and brown dwarfs, old white dwarfs, neutron stars, black holes, bricks, lost golf balls etc.
These possibilities (well maybe not the lost golf balls) were taken very seriously in the 1980s and 1990s. One of the problems with these hypotheses is that the gravitating Galactic dark matter needs to be roughly spherically distributed and have a radial distribution that is much larger than the distribution of the luminous matter. This immediately gives an "origins problem". Why would these dark objects be distributed very differently to the normal matter? After all, we think low mass stars and brown dwarfs are produced like other stars and that white dwarfs, black holes and neutron stars are the endpoints in the lives of stars more massive than the Sun that are distributed mostly in a disc.
Nevertheless, people looked for them. The main tool was the microlensing surveys. By staring at fields towards both the Galactic bulge and Magellanic clouds, the idea was to constrain what populations of "massive compact halo objects" (MACHOS) there could be, by looking for the magnification events that occur when such objects pass directly in front of a background star. These experiments (and there were several) basically ruled out MACHOS as providing anything but a small proportion of the Milky Way dark matter (e.g. Tisserand et al.'s 2008 headline result was that MACHOS in the range $6\times 10^{-8} < M/M_\odot <15$ were ruled out).
Thus a population of white dwarfs, neutron stars, black holes, brown dwarfs or other stars in this mass range are ruled out as contributing significantly to dark matter in the Milky Way and the the other evidences that non-baryonic dark matter is required rule them out as contributing more than a minority of dark matter from that point of view too.
Primordial black holes are a different thing altogether. These would be classed as "non-baryonic" dark matter, since they formed before the epoch of nucleosynthesis. They could in principle lie outside the mass ranges probed by the microlensing surveys and so long as they were not too small, would not have evaporated. The status of this idea is unclear. You can follow the summary and references at https://en.wikipedia.org/wiki/Primordial_black_hole and look at the review by Carr & Kuhnel (2020). However, the Bullet Cluster observations actually have little to say about the nature of the dark matter, only that it is difficult to describe such observations with modified gravity theories.