The equation of circle $C$ is $x^2 + (y − 8)^2 = 25$. The eye is located at $E = (0, −5)$. The maximal circular arc visible to the eye is $AB$, which is then being projected on to the one-dimensional "screen" as $A'B'$.
What are the co-ordinates of points $A$ and $B$?
I came this far: point $P$ on circle $C$ has the coordinates $x = 5 \cos\theta$, $y = 8 + 5 \sin \theta$. Now I should use this to find points $A$ and $B$, but I don't know how to proceed.
I will use your parametric representation. Observe that the center of this circle is at $C(0,8)$. The distance $|CE|=8+5=13$. The $\triangle CAE$ is a right angled triangle with $|CA|=5$ (radius) and $|CE|=13$, so $EA=12$ (Pythagoras) (same will be true for $EB$).
Now $$|EA|=12 \implies \sqrt{(0-5 \cos \theta)^2+(-5-8-5\sin \theta)^2}=12.$$ This gives us $$194+130 \sin \theta=144. \implies \sin \theta=\frac{-5}{13}.$$ Thus we also get $\cos \theta=\pm \frac{12}{13}$.
So $A,B=(5 \cos \theta, 8+5\sin \theta)=\left(\pm \frac{60}{13}, \frac{79}{13}\right)$
Note that $O(0,8)$ is the center, and EAO and EBO are right triangles with AB $\perp$ EO, giving
$$EA=EB= \sqrt{EO^2-r^2} =\sqrt{(8+5)^2 -5^2}=12$$
and $AB =2 \frac{ EA \cdot OA }{EO}= \frac{2\cdot 12\cdot 5}{13}=\frac{120}{13}$. So, let $A(\frac{60}{13},b)$, $B(-\frac{60}{13},b)$ and substitute them into $x^2+(y-8)^2=25$ to obtain $b=\frac{79}{13}$.
Thus, the endpoints are $(\pm \frac{60}{13},\frac{79}{13})$.
Note - the solution relies on using AM$\geqslant$ GM inequality without proving it , check this link out if you are not familiar with this inequality Proofs of AM-GM inequality
Let's first initialise parametric coordinates for the circle $x^2 + (y-8)^2 = 25$ as $P= (5 cos \theta , 8 + 5 \sin \theta )$
Let the given point be $Q(0,-5)$
Then find slope of line segment PQ M($ \theta $)=$ \frac {13 + 5 \sin ( \theta)}{5 \cos ( \theta )}$ Which simplifies to$ \frac {13}{5} \sec( \theta ) + tan(\theta)$
We now find range of M Let $\sec( \theta) + \tan( \theta) = t$ Then $\sec(\theta) - tan(\theta) = \frac{1}{t} $(from trigonometric identites)
t lies on ( -$ \infty , + \infty$ )
Now , $\sec(\theta) = (t + \frac{1}{t} )/2$ And $tan(\theta) = (t - \frac{1}{t} )/2$
Putting it in the slope function M and simplifying M = $\frac{18t}{10} + \frac{8}{10t}$ On positive range of t , applying AM $\geqslant$ GM $$\frac{18t}{10} + \frac{8}{10t} \geq \sqrt{\frac{18t}{10} *\frac{8}{10t}}$$ $$M \geq \frac{12}{5} $$ Similarly , $M \leq -\frac{12}{5} $( to prove this assume t is -ve , then $-t$ will be positive , apply $AM \geq GM$ for this )
equality holds good for$ \frac{18t}{10} = \frac{8}{10t}) $ (in both the cases (namely t>0 and t<0 )
Hence t = +12/5 or -12/5
Using these to get $ \sec(\theta)$ and hence find the end points by putting the values of \theta in the circle's parametric coordinates .
This is easier to solve, I think, using the original Cartesian equation. The key observation for any solution is that the endpoints of the visible arc are also the endpoints of the chord of contact of the two tangents to $C$ through $E$.
The nice layout of $E$ and $C$ with respect to the coordinate axes makes for a relatively simple solution using similar triangles, but I’ll continue with a general method. The chord of contact is the polar of $E$ with respect to $C$. If the equation of a circle is $(x-h)^2+(y-k)^2=r^2$, then the polar of the point $(x_0,y_0)$ has the equation $(x-h)(x_0-h)+(y-k)(y_0-k)=r^2$. Plugging in the values from this problem, the polar of $E$ is $(y-8)(-5-8)=25$, or $y=\frac{79}{13}$. Intersect this line with the circle to find $A$ and $B$, which for this problem is a simple matter of substituting into the circle’s equation and solving for $x$.
Another way (non calculus) using only the theory of quadratics is to simultaneously solve the equation of the circle and the lines of tangency.
The circle has equation $x^2 + (y-8)^2 = 25$, while the lines of tangency have the form $y =mx - 5$, where $m$ is the slope (to be found).
We get a single variable quadratic in $x$, whose discriminant is $100m^2 - 576$. For tangency, the discriminant must vanish (so you get a single repeated root at point of contact). Then you get $m = \pm \frac{12}5$ from which you can get the points of tangency as $(\pm \frac{60}{13}, \frac{79}{13})$.
