I tried doing squats about 3 years ago for the first time and experienced the same problem as you - couldn't balance, knee position, etc. I posted a couple questions with exceptional answers on the topic:
I avoided them for another year and did a routine very similar to your first one with minimal results - especially my squat, which almost didn't budge at all on the smith machine (made it to about 25kg).
Eventually I gave into the advice given to me in the above questions, and corrected some flexibility issues that were really holding me back. After that, I realized that I actually lacked almost all strength and mobility in my hips and glutes - I physically couldn't move my legs away from my body (out to the left and right). I did what your trainer is saying now and worked on the abductors / adductors around my hips and glutes and almost immediately was able to perform a squat (within 2 weeks). It has been about 4 months since then and I have seen considerable improvement in both the size and strength of my legs, starting on 30kg and now able to squat 75kg (both including bar).
As for my current routine, I work my legs in the same day as shoulders (which I'll leave out here):
- Barbell Squat
- 1x8 warmup set
- 3x5 working sets
- 1x3 where I go as heavy as I can
- Leg Press Machine
- 3x8 working sets
- 1x set to failure on as heavy as I can for 3-5 reps
- 3x20 calf extension
- Lunges w/ barbell
- Not 100% sure on reps - I walk the length of the gym and back per set.
- Superset leg extension with leg curl
- 4x8 drop sets (start as heavy as I can, drop when needed to meet the 8 repetitions).
- When I move onto shoulders, I'll do 10-20 body weight squats between most sets.
Like I said, the barbell squat is what has really made me start seeing the strength and size gains. It is responsible for the development of a lot of muscles in my legs that I don't even think existed beforehand.
Side note: Bringing my deadlift to a more serious level seems to have helped a lot as well.
Slide 10, a no to smith squats!
– Prakash Wadhwani Mar 14 '14 at 14:50