It sounds like what you're groping your way towards here is computability theory, a richly developed area on the boundary between mathematics and computer science.
Its fundamental result is: Yes, there are known well-defined problem types that are unsolvable in a very strong sense.
What "unsolvable" here means it that
- The problem has an infinite number of well-defined instances.
- Each of the instances has a single correct answer.
- It is provably impossible for any single procedure to produce the correct answer for every instance.
(Here, "procedure" may sound fuzzy, but it is another main result of computability theory that this concept can be formalized and that all halfway resonable attempts to formalize it turn out to be equivalent. This is Church's thesis).
This is a strong kind of unsolvability, because it says that not only can't we always prove that such-and-such is the correct answer; there is not even a crazy procedure that always happens to give the right answer even though we can't prove that it always does.
The most famous such undecidable problem is the halting problem, which is quite technical to define, and there are many known similar problems that concern idealized computing machines or formal logic.
Examples of some more approachable undecidable problems are
These problems have the common theme that there are infinitely many cases we may need to check before we can be sure of the answer, so simply checking all cases is not a solution procedure.
You also mention problems where you can in principle solve them by checking all cases, because there are finitely many of them -- though the number of cases may be astronomically vast. The question then arises whether there is a smarter solution than checking all cases. This leads into the neighboring area of computational complexity. It turns out that even though we have a fairly good idea of how to formalize that question, the answer is disappointingly often "nobody knows". There is a group of problem types -- the NP-complete problems -- where it is strongly suspected that there is no procedure that completes significantly faster than checking all cases, but nobody has been able to prove this. That is the famous P=NP question.
But I am aware that that still is an problematic formulation.
– a.t. Mar 08 '17 at 13:29