I am currently training a pattern classification network, and seem to get very inconsistent result.
The network seem to overfit, but the validation accuracy is very inconsistent.
I am currently trying to do pattern recognition on audio spectrograms. The audio consist of utterances of yes/no, and acoustic model (gmm-hmm) has been trained to detect these. Given i already have a gmm-hmm, I can from the alignment decode which state obervation has the highest probability, and use that as output target for my neural network.
I am currently using different network structure such as Densenet, Resnet.
I once had a run using densenet which gave me a validation accuracy of 0.8, but this seem to only have happened once. When i tries to train the network now i get validation scores around 0.3, which annoys me...
In general does it seem like the the validation score is very inconsistent, it loop around 0.2 - 0.4, which while the system keep overfitting.
Is training really supposed to be so inconsistent?..