1

Obviously most employers, when hiring a data scientist, would prefer experience with big data and/or data science. But what can one safely assume they will acknowledge as experience?

Let's say someone often launches software on a computing cluster that typically generates an amount of data. I'm not sure what the best measure of this data is for data science. I'll call it one or two thousand rows, 200k or 300k points per row...certainly under 500k. Then for each point, let's call it 25 or 30 values. This amounts to 30 or 40 gig of data. 300 or 400 times of this and you can call it a study - maybe one or two studies per year. I'm under the impression that this is much smaller than a data scientist at Google or Facebook would be used to, but it's certainly too big for my home computing systems.

If someone's been working with this for years (some people at this company have been doing it since before data science was coined/before social media existed), is it fair for them to claim big data experience? According to this answer, it's not the amount of data but what needs to be done with the data that matters - is that a universally accepted opinion?

For what it's worth, working with this data entails manipulation/cleaning of the data with some proprietary languages, shell scripts, and a lot of Python. A little bit of R but that's a more recent thing. It involves tons of data visualization, drawing conclusions, and presenting to management/convincing decision makers. Some of it involves trend determination, extrapolation, and comparisons made between data sets that aren't related in a straightforward way, so it sounds data science-ish to me. But I will be the first to admit that I have a limited understanding of what data science currently is.

...bonus points if you can tell me whether or not this is an Easter egg or is the actual answer total for this site at the moment: 1337

Edit:

I'll try to clarify. What do data science employers acknowledge as experience with big data/data science? Does the size of the data above qualify experience with it? Or is it universally accepted among those in the field that it's not at all the size of the data, but what you need to do with the data?

Community
  • 1
aeroNotAuto
  • 322
  • 2
  • 7
  • Wait, what exactly are you asking? This sounds more like a blog post than stackexchange question tbh. – LauriK Apr 07 '15 at 09:51
  • 1
    @LauriK well 'what constitutes data science/big data' has been asked here before and there doesn't seem to be a black/white answer. so i'm giving a specific example and asking where it falls in the gray. – aeroNotAuto Apr 07 '15 at 10:49

1 Answers1

3

Data can be big in various ways. It can be large N (observations) and small P (variables per observation), like access logs for example. It can be large P and small N, like in biostatistics where you have tens of thousands of gene expressions from tens or hundreds of people. Or it can be both N and P, like in Facebook data. Data can also be made big, by normalizing every single detail of it, meaning that you'll be able to make a massive relational database schema that actually represents simple data.

All kinds of big data needs different approaches and companies mostly care about their own version of big data. That's why the answer isn't black and white.

To answer the question you posed in title then I believe it's ethical to claim experience with big data as long as you can put it into context of the business or domain where you encountered it and can describe some situations where a different set of tools or skills was needed than is used for smaller toy examples.

LauriK
  • 591
  • 2
  • 6