The mathFunction1 and mathFunction2 must not be unknown. Either
- no known algorithm exists for one or both of them (and it is incumbent on you to provide at least a strawman feeble/naïve attempt at that algorithm) or
- one or more algorithms exist for both of them.
If the latter situation, then inventory all (not-unreasonable) known algorithms for each of mathFunction1 and mathFunction2, then obtain what a scientific paper for the lowest-asymptotic growth algorithm states is its big-O growth. (Or if for some reason that algorithm is unsuitable [e.g., has good asymptotic time growth but unacceptable asymptotic space growth], then obtain what a scientific paper for the 2nd-lowest-asymptotic growth algorithm states as its big-O growth.) Then compute your problem algorithm's big-O by utilizing the obtained big-O for both mathFunction1 and mathFunction2, then draw attention to the fact that additional work needs to be done on perhaps finding better algorithms for mathFunction1 and mathFunction2. Especially useful would be to find unproven conjectures about mathProblem1 and/or mathProblem2 that would permit a better big-O growth for one or both of them, then draw attention to these as possible future work that improves your problem algorithm's asymptotic growth.
Also, sometimes big-O (i.e., big omicron) is neither easily theoretically determinable nor easily empirically measurable in practice given worst-case inputs; in this situation small omicron, theta, big omega, and/or small omega might be calculable or empirically measurable for revealing growth-sets of inputs, which also provides a rule-of-thumb idea of mathFunction1's and mathFunction2's growth characteristics which can be utilized to calculate the corresponding small omicron, theta, big omega, or small omega for your problem's algorithm.
Without doing any of this answer in original problem's situation, stating a big-O notation for the original problem's asymptotic growth other than O(n max(O(mathProblem1),O(mathProblem2))) is little better than saying “then a miracle occurs”. It is your job to dig deeper as very very very much as you can into characterizing mathFunction1 and mathFunction2 for your research to be any more than trivially useful (if not: entirely unhelpful, depending on the situation).