This is not trivial, have you thought about chain reorgs?
This is a client side phenomenon, depending on who you're connected to in the P2P network. If there's a race between two miners - they publish two blocks, A and B, at the same time, some people will believe one or the other. Eventually one of the chains will be elongated making it the longest and valid chain. We'll say that B was added to. Any tx's in A would no longer be valid, so your app needs to deal with this, and not show duplicate transactions.
I'm not entirely sure on the behavior from the client either - if a reorg is 3 blocks deep, does it issue callbacks (blocknotify) on all 3 new blocks? Or just the highest, best block? Maybe you need to recurse backwards from the latest block every time until you find a parent block you've already encountered?
Scaling is an issue here - I've started down this path before and you need to try reduce the number of queries you make over the RPC. Try decoding transactions using https://github.com/Bit-Wasp/bitcoin-lib-php instead of using the RPC, maybe that'll help.