In my opinion, it should be both, because it will affect the way in which a molecule will interact with plane polarized light (physical), but also affect chemical interactions such as receptor binding.
On the other hand, consider $\ce{CHDBrCH3}$, which is chiral, as there 4 distinct groups attached to the central carbon. This would insinuate that chirality is a purely physical property, as despite deuterium having the same chemical properties as hydrogen, we are able to form a chiral molecule.
I would not consider chirality as a property--either physical or chemical--it is part of the definition of a molecule's structure, which in turn gives rise to the properties it exhibits (rotation of polarized light, melting point, reactivity with X, etc.). In your example, the fact that hydrogen and deuterium are almost (but not entirely) chemically equivalent will make the substance behave almost the same as its dihydrogen analog, and the two mirror image isomers nearly indistinguishable by physical or chemical means.
