There's going to be some confusion in any answer, because the strictness of interpretation of Genesis (especially 5) is precisely what divides a young earth creationist from an old earth creationist.
By definition, the old earth creationist appeals to extra-biblical "evidence," and hence there cannot be a "biblical" old earth creationist stance. The YEC Creationist believes in the young earth because the genealogies of Genesis 5 point to only about 2000 years. This combined with other references fixes the earth at about 6000 years. The OEC Creationist looks at the geological record and the apparent "age" of things, and uses that to contradict this literal reading of Genesis 5. The YEC counters (like William Paley) say God created things with age. The OEC says, no - Genesis 5 is either incomplete or figurative.
(Please note - I am not saying one is right or wrong - just that the rules of interpretation are such that an OEC isn't going to have the 'biblical' basis of a YEC.)
So, how would you make a case for Cavemen in Scripture?
First off, because Adam had language (he was naming the animals after all), we can rule out a non-intelligible caveman grunting society stereotype. So, Neanderthals they weren't.
That said, there is some interesting evidence from Genesis 4 that may pertain to the "caveman" question. Specifically, Genesis 4:19 -22 says:
19 Lamech married two women, one named Adah and the other Zillah. 20 Adah gave birth to Jabal; he was the father of those who live in tents and raise livestock. 21 His brother’s name was Jubal; he was the father of all who play stringed instruments and pipes. 22 Zillah also had a son, Tubal-Cain, who forged all kinds of tools out of[g] bronze and iron
The implications are as follows:
If Tubal-Cain was the first forger of bronze and iron tools, then all characters mentioned prior (i.e. Lamech and those before him), were by definition stone age societies. Bronze and metalwork is the technological divide. (Interestingly, the big gap between the Israelites and the Philistines was that the Israelites were a Bronze Age society that didn't have the stronger Iron Age tools of the Philistines.)
To the point of the "caveman," however, a stone age society could construct stone and adobe shelters, or use caves, but certain types of shelter would have eluded them.
Jabal, if taken to be the first bedoiun, would imply that societies had a different arrangement than that of the bedoiun.
Because Cain was a farmer, we can't necessarily conclude that had to be a hunter-gatherer society, but we also can't rule that out. Who knows, maybe Cain was just a gatherer rather than a cultivator.
So, is this a flimsy case for cavemen in the bible? Yes, it is flimsy. I don't think you really can make the case - but if you were going to, these would be the points I'd go with.