There are several questions already dealing with this topic of using a Mac of some type (MacBook, iMac, iPad, etc.) as an additional display to a computer whether it be another Mac or even a Windows machine.
Target Display Mode
(Just putting this here to address the question)
iMac computers prior to 2014 supported Target Display Mode, Potentially your iMac might support it (you mentioned it was "too old" for AirPlay) but Mac computers running High Sierra or later couldn't connect to it. This makes an Apple-centric hardware solution untenable.
Software Solutions
In the links of similar question linked above, it comes down to about two different solutions:
- Luna Display. This uses a hardware based dongle to connect a computer to a Mac or iPad as a secondary monitor
- AirParrot 3. This allows you to turn any device including AppleTV and Chromecast into another monitor you can stream/mirror to.
The problem with these solutions is the limited amount of bandwidth available to send high definition video with a high refresh rate reliably. This is why you're not getting the resolutions you're looking for.
Final thoughts
There are good to excellent apps out there to stream and extend a desktop, but they are not designed for "production use." They are for setting things up in an ad-hoc manner. I've personally used SpaceDesk on a Windows machine to extend the display of a laptop to my iPad. It was great when I was in the local Panera working on financials, but when I got back to my desk, I didn't give it a second thought since I had my multi-monitor setup.
Don't try and bodge this. Restore that iMac back to it's glory and sell it or get into retro computing. For the secondary display, get a quality monitor.