3

I have no background AT ALL in data science/stats/mathematics. However, I've always been interested in what data shows.

I have a huge dataset right now - daily attendance figures for a factory of ~300 for the past 10 years. I'm interested in finding out answers to questions like is there a pattern of leaves correlated with public holidays? For example, around which holiday (+/- 2 days) are workers most likely to take a leave? This is an expected pattern. Or, was there a significant increase (+10%) in on-time reporting after bonuses were issued. Maybe there are hidden patterns which an algorithm can find.

Is there a tool I can plug this data in which can help me find these patterns? Google tells me there's a tool http://www.i-programmer.info/news/84-database/3501-mine-finding-patterns-in-big-data.html but I'm not sure if this is the right direction for me.

I'd appreciate any advice!

hasan
  • 139
  • 2
  • 1
    I'm voting to close this question as off-topic because it belongs on Software Recommendations SE – Dawny33 Jan 03 '16 at 13:16
  • @Dawny33, I did search through that SE but did not find any tool. Since this seems like a specialized requirement, I was hoping I could get suggestions in this SE. – hasan Jan 03 '16 at 13:19
  • Pattern Recognition is an entire domain in itself. In fact, one of the primary goals of data science and statistical modelling domain is pattern recognition. So, qns about algorithms and techniques would be more suited here, rather than software suggestions! :) – Dawny33 Jan 03 '16 at 13:22
  • @Dawny33, thanks for that. As a complete beginner, this is good for me to know. Given that this is the kind of information I'm looking for, what resources would you suggest I start with? – hasan Jan 03 '16 at 13:26
  • You can learn about the software used for pattern recognition here and these posts would help you get a nice headstart! – Dawny33 Jan 03 '16 at 13:31

1 Answers1

1

You are unlikely to find a tool that will truly do this for you. Various bits of software may produce some nice graphs. However we're not at a stage where tools can generate interesting questions to ask, ask them, and then explain the results to you.

If this is commercial, your best bet is to hire someone with an appropriate background and set them some questions. Tell them what your biggest goals and costs are, explain the data to them, and see what happens. So the answer is a contractor.

conjectures
  • 321
  • 1
  • 5
  • It's not at all commercial. I'm just trying to learn something I'm interested in. Did you see the link in the question? Seems to suggest it can find "hidden" patterns. – hasan Jan 03 '16 at 21:28
  • You won't find a tool that can understand the business context of a factory, formulate an interesting question based on it, relate the question to the data, answer it and then explain the answer to you. If you find one, you've stumbled on an almost complete Artificial Intelligence. I'd suggest you look up an Operational Research syllabus and get reading :) – conjectures Jan 03 '16 at 22:44
  • Of course. I'm not looking for something that can think of and solve business questions - that would be revolutionary! I was hoping for a tool that could look purely from a mathematical/statistical point of view. That tries to identify common patterns. Maybe we're not there yet. Thanks for the suggestion, will look at it. – hasan Jan 04 '16 at 07:10