Justification for Marriage without State
The first marriage in the Bible involved a man, a woman, and God.
Genesis 2:23 And Adam said, This is now bone of my bones, and flesh
of my flesh: she shall be called Woman, because she was taken out of
Man.
This might be seen as the minimum requirement. Various cultures at various times have added layers of family, friends, church, village, wedding planners, caterer, photographers, musicians, etc.
The government started to get involved when things that might be actionable in court began to come into play like property rights, inheritance, child custody, etc.
There are many people who not only do not have the government involved, but also do not want God involved. For people like this, governments often pass "common law" legislation so that rights can be adjudicated in court in case of death, divorce, or other changes in legal status.
The precept of "obey the law" could be applied to marriage unless the law extended into areas such as forcing you to marry someone or telling you you could not marry someone.
Romans 13:1 Let every soul be subject unto the higher powers. For
there is no power but of God: the powers that be are ordained of God.
I can understand the desirability of being free of government influence. However, I am not aware of any Biblical basis to reject legal requirements such as taxing through licensing or registering a marriage.