You can find a closed form as follow :
$$ F(a,b)=\left(\sum_{k=0}^n \int_0^1(-4t)^{ak+b-1}\binom{n+k}{2k}dt \right) $$
Finite sum
$$ F(a,b)=\left( \int_0^1\sum_{k=0}^n(-4t)^{ak+b-1}\binom{n+k}{n-k}dt \right) $$
Then using the coefficient extraction function $[X^i]$ for a given $i$.
\begin{align*}
\sum_{k=0}^n(-4t)^{ak+b-1}\binom{n+k}{n-k}
& = \sum_{k=0}^n(-4t)^{ak+b-1}[X^{n-k}](1+X)^{n+k} \\
& = \sum_{k=0}^n(-4t)^{ak+b-1}[X^n]z^k(1+X)^{n+k} \\
& = [X^n](1+X)^n(-t)^{a+b-1}\sum_{k=0}^n(-4^at^a)^{k}X^k(1+X)^{k} \\
& = [X^n](1+X)^n(-t)^{a+b-1}\dfrac{1-(-4^at^aX(1+X))^{n+1}}{1+4^at^aX(1+X)} \\
&= [X^n](1+X)^n(-t)^{a+b-1}\dfrac{1}{1+4^at^aX(1+X)} \\
&=[X^n](-t)^{a+b-1}\dfrac{(1+X)^{n-1}}{1+4^at^aX} \\
& = [X^n](-t)^{a+b-1}(1+X)^{n-1}\sum_{i=0}^\infty(-1)^i(4^at^aX)^i \\
& = (-t)^{a+b-1}\sum_{i=0}^\infty(-1)^i[X^{n-i}](1+X)^{n-1}(4^at^a)\\
& = (-t)^{a+b-1}\sum_{i=0}^\infty(-1)^i\binom{n-1}{i}(4^at^a)^i\\
& = (-t)^{a+b-1}(1-4^at^a)^{n-1}\\
\end{align*}
From here the exercise can be ended.
Credits
This answer is inspired from the work in the answer here : sum of binomial series with alternate terms.