How to come up with ideas that will actually be worthy of a paper?

Question

I'm a researcher (completed PhD) in CS. I have a very clear pattern: I can certainly come up with "punchy" sounding ideas, but they are suitable to write up a small workshop paper about but will never make it to a full conference paper. They're simply too impractical, they won't have real impact; they're not deeply theoretical contributions, but I also will not be able to implement my ideas enough to show (even for a paper figure) that they're potentially useful. It's more like a clever idea that someone will think "that's fun!" but no one would seriously encourage me to write a full paper on it. My research record is absolutely littered with such ideas that either get written up as a short abstract or (more often) get abandoned. A couple ideas have seemed more promising and those turned out to have been done already. If I pitch these ideas to potential collaborators/mentors I pretty regularly get shot down/discouraged from continuing to work on it.

I don't seem to be short on creativity or motivation, but I struggle very seriously to come up with any ideas that are both interesting and also suitable for a full conference paper.

I'm losing faith in my ability to be an independent researcher rather than just being known as someone who is good at executing other people's ideas. But I'm currently trying to address my issue by doing just this - intentionally focusing only on helping other people's projects until I figure out how to direct my own ideas in a better way.

Can anyone here provide advice on how to resolve this particular failure mode I have?

Answer 1

If I were to speculate (without further details it's difficult to diagnose where the problem might be), it sounds to me like your pattern is: focusing on the details and not on the big picture. A key skill to learn as one matures as a researcher is how to place one's work into a bigger narrative.

It sounds like you are starting with a very particular idea, but then struggle to fit it into a motivating narrative that will actually make it worth people's time to read your paper. This is a reasonable way to proceed if your particular idea is very clever and seriously non-trivial and therefore pursuing it produces real progress on its own, even in the absence of a compelling over-arching narrative. However, in other cases, you end up with neat tricks disconnected from issues of real interest.

Instead, I suggest that you learn how to proceed in the opposite direction: start by identifying places where the narrative of your field can be expanded on, and then hone in on the details. That is, start with the big picture, and then slowly zoom in, instead of starting with a particular place where a trick can be applied and then struggling to motivate your work.

In other words, learn how to do "narrative thinking" instead of (in addition to) "detail-oriented thinking". As for how to do that, I don't have much useful advice other than simply getting into the habit of zooming out and locating what you are doing inside a bigger picture. Reading papers by mature researchers where this kind of narrative thinking is exhibited helps. You can also ask your collaborators or mentors why they are doing what they are doing, i.e. what is the higher-level goal they are trying to achieve by writing a particular paper. In science, it's never a bad idea to stop and ask (yourself or other people): wait a second, what is actually the point of doing X?

Overall, my experience has been that a productive way to write papers is: "here is a vague idea based on some analogies and speculation, if I succeed in gradually fleshing out the details, then this will make for a good paper". In contrast, "here is a neat trick, if I succeed in finding a way to motivate it, then it will make for a good paper" has been less useful for me. But, note that everyone's style of doing research is different. I figured out my strengths lie more in the former area than in the latter, but yours might lie somewhere else.

A related but distinct factor could be focusing on projects that are "too small". Here I don't have much advice other than: don't be afraid to be more ambitious and jump into deeper water, i.e. to tackle problems where you don't immediately see a way out. This involves developing a greater tolerance for situations where you don't know what to do. "Knowing what to do even when you don't know what to do" is another key research skill to learn (easier said than done).

Answer 2

Being an "independent researcher" doesn't necessarily mean working alone. The best advice I could give would be to increase the collaborative efforts you seem to have already started. But don't just work on other people's ideas. Find a way to share ideas and to evaluate them in a small group. University working groups are like that, as are coffee rooms, in fact.

In CS, conferences are excellent for that sort of thing. Think in terms of joint papers. And, the more practice you get, the more likely it is that you bring forth your own ideas.

One idea can lead to another. A simple idea can lead to a deeper one. Trivial stuff can be easily cast aside.

It can also be that you are devaluing your own work, thinking that things are not of general interest. You could try to submit a few papers based on those ideas and see how it goes. And, some "simple" ideas are of interest to the teaching community, who have conferences of their own.

Answer 3

I need to qualify that I'm still only a research trainee, and that I'm in chemistry, but I think my experience may be applicable here.

I relate to your desire to create something "good" or publishable. However, that standard is a moving goalpost (see this thread for some encouragement). The most important thing I'm learning as I do my own research is to treat the work as a journey, not as a goal. I would encourage you to spend a small amount of time working towards some of these "impractical" ideas - they may be easier than you first considered, or the process of considering them deeply may spark your thinking in a new way! You also mention that many of your ideas have already been implemented. This does not mean that you are incapable of making "good" ideas; instead, it's an indication that you are thinking about publishable ideas! Be persistent, and as you gain a deeper understanding of the work that exists in your field, you will gain a sense of the gaps of knowledge that exist.

Know that you're not alone in that struggle!