No, there's no setting to fully close an application when you press the home. It depends on how the program is built. Some stay running, some stay in memory but are inactive (not eating CPU but wasting memory in case you restart the program), and some just close as desired. Additionally, there are plenty of Samsung or carrier applications that run in the background full time and don't have an onscreen program to open or close.
There are several applications on the Google Play Marketplace that will automatically close all applications for you either on a schedule or on demand, but that's as close as it gets unfortunately.
Perhaps there are more technical answers that involve custom ROM's, but that's not applicable within the confines of a SU question/answer.