2

Im starting to learn about ml and most of examples show very simple examples on how to train your ml modell. For me this looks more like statistic calculations than actual ml. These kind of examples could very easily been written in sql server etc.

What Im looking for is the ml to find pattern within data that i dont know about. For example supose i have a lot of devices which is logging temperature, humidity, hours used and much more. At some point these devices fails for some unknown reason and Will be marked as "failed"

So what om looking for is if there is a way to use ml to analyze all the log files and find common patterns to why these devices failed, without me have to "tell it what to look for first" in this way I Will be able to predict feature failures based on log files.

Would this be something I can use ml for and would Azure be a good starting point for this?

Mr Zach
  • 121
  • 3

2 Answers2

2

Hi and welcome to the community!

Sure you can use ML. I would start with brainstorming to formulate the problem. So one idea could be Sequence Learning:

  1. Injecting a unique and recognizable number instead of F and training a NN-based model (RNNs are used in this case e.g. ESNs or LSTMs) will learn the dynamic of the data themselves as they model memory. But for training such models you need lots of data actually (to be more precise, you need long Time-Series). In this solution, the task is to see that unique and recognizable number in the prediction and output is a failure. See this for more info.
  2. Hidden Markov Models are also designed for such tasks.

Another idea could be:

  • Segment each time series into intervals ending in a failure. For example:

    ts1 = [1,2,1,2,F,1,2,3,4,F1,2,F]

Becomes

> ts1_1 = [1,2,1,2]
> ts1_2 = [1,2,3,4]
> ts1_3 = [2]

Now you have a set of training samples (positive samples only!) for Time-Series 1 (e.g. Temperature). Now produce some negative samples from those sequence segments which do not result in "F" and train a model for e.g. classifying. Then on real-time prediction, you need to make a prediction after receiving every new point (temp for instance) and see if it is going to result in a failure.

DISCLAIMER: This way is not what you really do in practice but a nice starting idea for a newbie learner in the field!

I recommend having a look at this answer, this tutorial and this video as well.

Hope it helps!

Kasra Manshaei
  • 6,570
  • 1
  • 21
  • 45
1

Anytime that you are looking at logs, basically you want to do anamoly detection. There are various methods I have tried and want to use few more:

1) Topic Modelling 2) Supervised Learning and can scale using H2O.ai 3) XGBOOST If you are using Elastic Search, you can use NBoost. âš¡NBoost is a scalable, search-engine-boosting platform for developing and deploying state-of-the-art models to improve the relevance of search results.

Use Azure, if you want to scale at speed with all other advantages Azure provides. You can also work without Azure. Firstly clean your data and see if datetime are significant to identify patterns. For Ex: the CPU temperature might have fluctuated for 2-3 days before the device failed or the Fan had failed before the device failed. CApture the patterns and draw probabilities around these pattern as a feature.

Most people would recommend BERT but BERT would not perform accurately unless you create your OWN BERT for the task.

I hope these points are helpful.

Syenix
  • 359
  • 1
  • 6