The question is: We assume a uniform (0,1) prior for the (unknown) probability of a head. A coin is tossed 100 times with 65 of the tosses turning out heads. What is the probability that the next toss will be head?
Well, the most obvious answer is of course prob = 0.65, but I am afraid this is too simple. However, I really don't know what is wrong with this answer? I think I need to use the fact that we assume a uniform [0,1] before we begin tossing the coin, but I am not sure how to proceed.
$0.65$ is the maximum-likelihood estimate, but for the problem you describe, it is too simple. For example, if you toss the coin just once and you get a head, then that same rule would say "prob = 1".
Here's one way to get the answer. The prior density is $f(p) = 1$ for $0\le p\le 1$ (that's the density for the uniform distribution). The likelihood function is $L(p) = \binom{100}{65} p^{65}(1-p)^{35}$. Bayes' theorem says you multiply the prior density by the likelihood and then normalize, to get the posterior density. That tells you the posterior density is $$ g(p) = \text{constant}\cdot p^{65}(1-p)^{35}. $$ The "constant" can be found by looking at this. We get $$ \int_0^1 p^{65} (1-p)^{35} \; dp = \frac{1}{101\binom{100}{65}}, $$ and therefore $$g(p)=101\binom{100}{65} p^{65}(1-p)^{35}. $$ The expected value of a random variable with this distribution is the probability that the next outcome is a head. That is $$ \int_0^1 p\cdot 101\binom{100}{65} p^{65}(1-p)^{35}\;dp. $$ This can be evaluated by the same method: $$ 101\binom{100}{65} \int_0^1 p\cdot p^{65}(1-p)^{35}\;dp = 101\binom{100}{65} \int_0^1 p^{66}(1-p)^{35}\;dp $$ $$ = 101\binom{100}{65} \cdot \frac{1}{\binom{101}{66}\cdot 102} = \frac{66}{102} = \frac{11}{17}. $$
This is an instance of Laplace's rule of succession (Google that term!). Laplace used it to find the probability that the sun will rise tomorrow, given that it's risen every day for the 6000-or-so years the universe has existed.
