4

In Keras, I am training a model as in below

model.fit_generator(data_generator(),
                samples_per_epoch = count,
                validation_data=(x_val, y_val),
                nb_epoch=50,
                callbacks=getCallBacks(),
                verbose=1)

In data_generator function, I am printing few debugging statements.

On running above fit_generator function, it takes really long time to print what I had put in the data_generator().

Is there any series of steps which keras takes before picking out the data batch for the training which makes the process so slow or there is some other caveat?

Divyanshu Shekhar
  • 569
  • 1
  • 5
  • 15

1 Answers1

1

I have noticed this when training on one or more GPUs. I think it is due to Tensorflow having to acquire the resources (in the background it blocks the entire GPU memory). Perhaps some amount of data e.g. the first batch is copied over to the GPU.

I didn't find a way to reduce the waiting time dramatically, but you could try the tensorflow options to allocate less memory and allow memory growth as required. Check out the official docs and this issue for the Keras version.

Here is the latest recommendation form that issue: setting Tensorflow options befor eimporting Keras.

-------------------------- set gpu using tf ---------------------------
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)
-------------------  start importing keras module ---------------------
import keras.backend.tensorflow_backend as K
import keras......
n1k31t4
  • 14,858
  • 2
  • 30
  • 49