Why is Emacs code so unorganized?

Question

I've been looking at Mickey Petersen's exegesis on the Emacs 25.1 NEWS file, and realized that built-in Emacs Lisp files are a mess.

The worst offender is without a doubt subr.el, whose description simply says

Basic lisp subroutines for Emacs

From what I can see, it is a motley collection of functions that has accumulated over decades, with no rhyme or reason as to why anything is or is not in there.

There's also subr-x.el, etc. which add to the confusion.

Is there something about subr.el (and Emacs Lisp files in general) that makes refactoring difficult? Would things break if, say, string- functions were moved into string.el, keymap functions were moved into keymap.el, etc.?

Answer 1

subr-x exists for a purely technical reason that goes down to a peculiarity of Emacs' build process: The Emacs' binary contains the loaded byte code of some selected core libraries for efficiency. subr is among these, and thus every running instance of Emacs as subr loaded—which in turn means that every single definition in subr is loaded in every running instance of Emacs, whether you use it or not. Hence it's crucial that subr only contains definitions that are very likely to be used by Emacs.

subr-x serves as a staging ground for new subr functions: It exists to make these functions available without dumping them into all Emacs' binaries. If there's sufficient interest in these functions they'll be moved to subr and included in the dumped Emacs. Otherwise they'll be removed at some point, into a different library or entirely from Emacs.

---

As for the general sentiment I disagree. I'd say that it's a strong move to call something "messy" without trying to understand the reasons and history first.

subr is probably not the best part of Emacs, but it's also one of the oldest, and one touched by many different people. I'd say that subr is to Emacs what the dreaded util module is to other software: None likes it, but you'll also have a hard time to find any software that doesn't have this module in one or another manifestation.

Every system beyond a certain size feels the need for shared code that seemingly fits nowhere else, and it takes a very skilled and disciplined development team with strong respect for policies to abstract shared code in a way that avoids the util trap.

In a project with as many contributors as Emacs that's almost impossible, or at least very expensive in terms of time spent in code reviews and refactoring, time that's—given Emacs' limited resources—better spent elsewhere. We may not like util modules but let's be honest they're not doing any harm either. A project shouldn't strive for perfect quality, it should strive for the quality that is sufficient to do the job. You shouldn't spent 80% of your resources for a 20% increase in code quality, the less, if your resources are as limited as Emacs' in relation to the project's size.

Emacs code has its dark corners, but it's not any more messy or unstructured than other large systems I've seen. On the contrary I find it very remarkable that Emacs maintains discipline and structure despite it's decades old history and huge fluctuation in contributors.

Answer 2

Several reasons. First, emacs has grown over time. Thousands of people have contributed, and not all of the code is of identical quality. Like all things which grow over time, the final shape was not planned out in detail from the beginning.

Second, emacs is more than a little bit influenced by image-based systems such as the Lisp Machine, Smalltalk, etc. In these systems, there is no separation between typing out a function into a source file and defining that function for immediate use. In these systems you might save a set of functions to a file in order to send them to someone else, but that file isn't needed for the functions you've defined to exist, even through a shutdown and reboot of the machine. Emacs does however live on systems that don't work that way, so it does have canonical source files. Once you've loaded a source file, however, you can call the functions they contain without reference to where they came from. To a first approximation you could rename all of the files to randomized names and emacs wouldn't care.

On the other hand, not all files are actually loaded all of the time. If you don't use feature X, then the source files for feature X don't get loaded. If you did mangle all of the file names, then loading optional features would often fail unless you knew the new filenames.

The result is that once it's running emacs doesn't care much about how functions are organized into files, but it is frequently necessary to know that filename in order to load the stuff you need. One makes reorganizing things easier, the other makes it harder.

On the other hand, it's not quite so dire as that. These days we mostly require packages, rather than load specific files. That makes refactoring them easier again! Maybe we just need people to dive in an propose patches.

To address your specific questions about subr.el, I don't think that it's part of any package. It literally is just a collection of functions that always need to be available. Moving them around shouldn't be hard at all. There are probably some details of the early-stage loading process that need to be kept in mind (gotta define things before you use them, for instance), but otherwise it should be pretty straight-forward.