10

I am currently storing my training data into HDF5 files and I want my team and I to switch for a database for two main reasons:

  • the data is not used only by me and the different datasets are stored in different folders at different paths etc so I want to create a single database containing all the data
  • we need to store a lot of metadata like hashes, insertion dates, ground truth (that may differ depending on the application) etc.

I don't know which database to use. My first thought would be something like mongodb which is not relational, but I wonder if there are databases specialized in deep learning training out there. I heard of SciDB but I am not sure what to think about it, and I searched for "databases for machine learning" in google scholar without success. That is why I am asking the question here.

Thank you in advance for your help.

Ethan
  • 1,633
  • 9
  • 24
  • 39
ava_punksmash
  • 241
  • 1
  • 5
  • 1
    there is no machine learning db per se, instead various stirage models are used as throughout other areas of computer science. So a key-value store can be more than enough and really fast. Else if relational queries are needed, then a relational db – Nikos M. Feb 02 '21 at 19:09

2 Answers2

4

I share here the results of my quick research:

I think I will create a mongodb database to store everything, because it is easy to use and accross the NoSQL databases it is the one my company knows best.

If you want to use a database, you may also consider things like parallel access, distributiveness of the database, resilience, etc.

I think more advanced solutions like shared filesystems, spark, hadoop, sciDB (in particular, sciDB seems specialized for analytics on arrays only) etc are good for data lakes and when we want to process ALL the data or a big part of it at the same time, per batch or through a pipeline for example. Therefore, I don't need it for now, and the simpler the best.

ava_punksmash
  • 241
  • 1
  • 5
3

From what you shared I would definitely recommend a NoSQL database to make it easily scalable and enable flexible data structures. Here are some options that came to my mind reading your Big Data use case:

  • Mongo DB: All data that can be transformed to JSON format, can be stored and it is actually very storage efficient due to BSON.
  • Hadoop: Probably your way to go since it offers many additional applications that are optimized for deep learning and ETL. HDFS would be a high-available alternative for storing your data and is suitable with all other tools we know from Hadoop.
technik
  • 361
  • 1
  • 4