R and Python, memory differences yielding performance differences

Question

I think there are numerous posts regarding which one to use: R or Python. However, I'm curious about how their architecture differences yield differences in speed performance, not which one to use.

This blog post performs a small test between R and python to show that the (optimized) python code was 2x faster than R code.* And I've read in this post that R tends to put everything in memory, which is why computations on large datasets is generally slow.

But what makes python's low level memory management so much different than R, which helps it yield these benchmarks?

*Though python was 2x faster in this test than R, I'm not saying that python is generally 2x faster than R.

Answer 1

It depends on the usage of various packages (Numpy/Scipy etc), which are written in C and be incredibly fast also Python can be complied using JIT. Here is an excellent comparison between R and Python : https://learnanalyticshere.wordpress.com/2015/05/14/clash-of-the-titans-r-vs-python/

Answer 2

Looking the responses of knn and Sreejithc321, i only can remark some ideas still valid 8 years latter:

1. Python 3.12 is incredibly fast against R; as Sreejithc321 mentioned is written in c and since 3.11 is improving their speed by leaps and bounds.

   • R apparently performs better than raw python managing large datasets, but python as general language have a lot of specific libraries like: numba jit, Intel® oneAPI Math Kernel Library, Intel® Modin, and so on.

2. Vectorization is the king in every language, but not only Vectorization also recursion and other Computer science toolkit.

   • We can see a lot of useful explanations of this in python like:

     1. Vectorise all the things! How basic linear algebra can speed up your data science code (PyCon UK)
     2. Vectorize using linear algebra and NumPy to make your Python code fast (PyCon US)

   • As you can see, " you mostly should never use for loops in..." no language.

3. R is optimized for statistical analysis, then

   • "novices can be running data analysis tasks within minutes"
   • statisticians tend to use R

The two languages binding nice together, then if you need:

1. Develop a data product like software as service, i think use:
   • python as prototype
   • R for avant garde statistical technique
   • Make a c++ or java binding for production, or python can work well depends the situation.
2. To learn i think both, python and R. A little bit of two, but python is wildly used as foundation in CS backgrounds and R at statistical or research backgrounds. Then, newer the math problem, more specific the language or tool (see process mining as example). A mature model is easily to see in pure python.

Finally the tool depends on the problem rather than speed and speed depends in the specific data, model, programming style and paradigm (functional, declarative, objects, and so on), and of course the hardware or network latency.

Reductio ad absurdum: We can load better a one terabit list with a GPU in GWBASIC rather than a vectorized approach in a 486 with 256 KB ram and a complied language.

Answer 3

R's performance depends incredibly on how you write it. For example, you mostly should never use for loops in R - they're horribly slow because they execute a function call with every iteration. (One should vectorize and use the apply family of functions instead. Weird, I know..) Vectorization is king in R if you want fast code. Assuming you vectorize both your R and Python code (and other factors), you should probably get the same order of magnitude in speed. For data larger than memory (you can specify the limit), R starts to become a bad choice. I don't know much about python's internals, so I can't speak on that.