There are two common ways of indicating restricting conditions. One way is to use a subordinate clause initiated with the pronomial adverb wobei:
Definition: Sei P(A,B) = A - B, wobei A ⊂ U und A ⊄ B gelten soll.
(Some language economists among the mathematicians might even drop gelten soll.)
A more succinct way is to use the preposition mit followed by the conditions:
Definition: Sei P(A,B) = A - B mit A ⊂ U und A ⊄ B.
Using the subjunction wo is grammatically valid, but it is bad style. I cannot remember to have seen it anywhere in a mathematical text:
Definition: Sei P(A,B) = A - B, wo A ⊂ U und A ⊄ B.