How would you determine big O notation for $\log^b x$? I don't think you can simply say $O(\log^b x)$, can you?
If you can, then here is a better question: $x^3 + \log^b x$. How would you know if it's $O(x^3)$ or something else depending on the $b$ value?
How would you determine big o notation for this? I don't think you can simply do O(log(x)^b) can you?
$\mathcal O\left(\log^b x\right)$ or $\mathcal O\left(\left(\log x\right)^b\right)$ is correct.
x^3 + log(x)^b
Assuming $b$ is a constant. You always take the fastest growing term in a polynomial. $T(x)=\mathcal O\left(\log^b x\right)$ is called polylogarithmic time. In this case, $\mathcal O\left(x^3\right)$ grows faster than $\mathcal O\left(\log^b x\right)$.
You can see a list of different complexities (sorted from lowest to highest) here.
To answer any question about $O(-)$ notation, you need to check the definition, which is that, for functions $f$ and $g$, $f=O(g)$ if, and only if, there are constants $x_0$ and $k$ such that $|f(x)|\leq k|g(x)|$ for all $x\geq x_0$.
$O(-)$ is often treated as some fixed hierarchy of functions, "logarithms are $O$(polynomials), polynomials are $O$(exponentials)" and so on, leading students to believe that you can only write $O(g)$ for some very special or very nice functions $g$. This is quite simply not true: the definition allows you to write $O(g)$ for any function of one variable that you want to. For constant $b$, you can write $O(\log^b x)$. Heck, you can write $O\big(\sin\,(\cos\tfrac{1}{x^2})+3\big)$, if you want to, though you probably never will want that.
Usually in O(f(n)) notation, we well try to find a simple function f, and we will try to find one that isn't growing much faster than the left hand side. (So we wouldn't say that n + 3 = O(n^2) usually even though it is completely true). We would probably write that (2 + sin (n^2)) n = O (n), because it's true, it's a simple function, and it is quite close - the left side is not little-O(n); it's always between n and 3n.
And it's relatively easy to show that x^3 is NOT O (log^b n) for any b.
In your case, O (log^b n) is perfectly fine. There's nothing that is both simpler and close to log^b n.
A good "rule of thumb" to use with asymptotic notation is: $O(n^n) > O(k^n) > O(n^k \log^l n) > O(n^k) > O(log^k n) > O(1)$
So if you have $n^3 + log^b n$, the $n^3$ term will dominate, giving you $O(n^3)$.
