10

I usually analyze data of experiments and although I have a general schema of steps I need to do, I might need to tweak it to the specifics of the experiments or the questions behind. I am usually the only one coding.

I looked at wikipedia but I am unsure which methodology I can use, partially because I have never followed any, and partially because I sometimes just explore the data, to see what it looks like, and other times I just want an answer. (And because I am not much expected to test or have a certain quality on my code)

I was prompted to ask this question after an hour or two discovering that r function table relies on the order of the vectors and not on the name of the elements to compare them to. Then I though I should have tested the behavior and functions where I used with some mock data. But I used table after other analysis resulted in lack of information, thus I couldn't have followed the test-driven development methodology (if I understood it right). However I feel that with some improvement in the way I face the project, I could be more efficient, aside from detecting errors sooner, but also how and what to look for in case I doubt a result, so please don't focus in just this example mistake.

Which software methodology fits best in research?

I am basically asking how to to ensure quality and timed progress as well as keeping the specificity of research.

Example of how I work:

A biologist has in mind a question and knows that doing a experiment will lead to have data of interest (ie, gene expressions levels in two conditions), then she/he set the experiment and recollect samples from 10 people/mice/rats... Now I must analyse that data for those 10 samples using existing libraries and tests (or implementing new tests) but taking into account the question the biologist had in mind (ie which genes are more expressed in one condition than in other). The structure is the same as in previous experiments (which involved 6 conditions and another animal) but the statistical test, normalizations, data structure may change. So I usually copy a previous version and adapt it to the current needs.

gnat
  • 21,213
  • 29
  • 113
  • 291
llrs
  • 171
  • 7
    what you're dong now is fine. No methodology will stop you making mistakes! just make sure you're using a version control system and keep your codebases well organised. – gbjbaanb Jun 08 '16 at 09:38
  • No methodology will stop mistakes. But some will catch the mistakes sooner! Design by contract, or contract based design. – Frank Hileman Jun 09 '16 at 16:29
  • Could you please elaborate your last sentence? I didn't get it at all. – llrs Jun 09 '16 at 16:31
  • perhaps https://en.wikipedia.org/wiki/Test-driven_development with some kind of automated test framework - small tests are useful for catching bugs and larger tests may map (roughly) onto your hypotheses – lofidevops Jul 19 '16 at 13:16
  • you may also be interested in https://wiki.debian.org/ReproducibleBuilds but it is probably overkill (even computer science doesn't demand it) – lofidevops Jul 19 '16 at 13:17
  • @d3vid Thanks for the link to the debian ReproducibleBuild, I didn't know about it. About the TDD I am unsure how to implement and how to fit it with my day to day work – llrs Jul 19 '16 at 14:19
  • 1
    @Llopis ideally you write a test first, it fails, then write the code, the test passes, then you commit your code - if you discover a bug later down the line, you write the test that would have caught the bug, it fails, fix the code, the test passes, then you commit your code - you can't forsee everything, but you can make sure the same thing doesn't happen again – lofidevops Jul 19 '16 at 15:40
  • @Liopis Are you working alone or with others? –  Jul 26 '16 at 22:53
  • I am the only programmer on the grup but there is another who is usually consulted and worked with them until I arrived, on my research group there are biologists and doctors. – llrs Jul 27 '16 at 06:36
  • @LLopis, If I understood correctly, you develop programs to analyze/interpret data? For example, after collecting data from you research, you want to explore what is contained there. To do that, you develop in some statistical language to extract relevant information. Is that it?

    Is this program used again in the future or only for the purpose of the research?

    Could you provide us an example?

     – Eduardo Copat Jul 27 '16 at 10:09
  • @EduardoCopat Yes, I use mainly R to extract relevant information from this experiment. The script/program I write usually is not used again as it is only for research, but I use a similar script for similar experiments. I updated the question with an example that happened yesterday – llrs Jul 27 '16 at 12:08
  • @LLopis Does the CRISP-DM methdology fits in your scenario? https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining – Eduardo Copat Jul 27 '16 at 12:20
  • Maybe it could, but how would this methodology improve my work? – llrs Jul 27 '16 at 12:34
  • how can I get 100 reputation points from OP, if he/she has only 31? – steffen Jul 28 '16 at 08:17
  • @steffen I believe it is possibly a misfeature of Stack Exchange and worth discussing on meta. I'd thought about doing so but is a bit hesitant. When the bounty expires and the new site user loses the reputation points, it may lock the user out of certain site privileges, which may deprive the user of the ability to curate one's own content (e.g. the question and answer). – rwong Jul 28 '16 at 12:28
  • @rwong: anyhow... looks like a bug to me ;) http://meta.stackexchange.com/questions/281754/it-is-possible-to-put-a-bounty-on-a-question-which-is-higher-than-the-own-reputation-count – steffen Jul 28 '16 at 12:34

3 Answers3

6

What is needed is perhaps not a software methodology, but a political change in academia that fixes the issue of lack of recognition of the role played by software development in science.

The Software Sustainability Institute (UK) is the organization closest to what you are looking for: how to argue for more conscientious use of computer programming in scientific research.

It also provides information pointers for those interested in software development methodologies.

However, I have to point out that methodologies typically govern teams of software programmers, with iterations and gradual refinement of project goals, and works with stable code bases that last for a long time. They are for projects that are orders of magnitude more complex than what you're doing.

As to why this very obviously correct thing (more conscientious use of computer programming in scientific research) hasn't been accomplished and always upheld, here is the inconvenient truth: in academic administrative environments, scientists can be seen to degrade the importance played by computer programming. Sometimes they can be seen to band together to deny the recognition of contributions from people involved in the software, even if the nature of that contribution fits within the scientific discipline.

At your workplace, there are things that were missing, and things that you can do.

Things that were missing:

  • Lack of guidelines
  • Lack of supervision or person to ask questions
  • Lack of mentors or computer programmers who are knowledgeable with the tools you use (e.g. R)
  • Lack of software reuse, archival, version control, or documentation of previously developed software, for repeatability and learning purposes

In short, the overall culture is that the people involved aren't really interested into ... you guessed it ... more conscientious use of computer programming in scientific research.

Things you can do:

  • Devote more time to learning your tools.
    • Spend more time reading the documentation and code samples for your programming languages
    • You will have to learn to love the tools you use.
  • Try to write down something, for the benefit of the next computer programmer who will be enslaved to the same group of people for the next couple of years
    • A wiki will be excellent.
  • Try to set up source version control
    • Be able to retrieve commonly reused code snippets
    • Be able to save a snapshot of the code used in a particular experiment

For career software developers, guidelines of this nature can be found in:

These are considered the basic requirements for running a software development business. However, when you are fighting a war of apathy, alone, you need to prioritize. Getting better with the tools, writing down and maintaining information, keeping versions of source code are the bare minimum for a sole environment.

rwong
  • 16,878
  • Interesting resoure on Software Sustainability Institute thanks! I will write my own guidelines of the code and data management, I do have a supervisor but he don't seem to have "knowledgeable with the tools", I use git, but I'll try to follow your advice on documentation – llrs Jul 27 '16 at 13:44
  • ha yes, a wiki... for having tried a few I would reccomend https://www.dokuwiki.org/dokuwiki# here. Simple to setup and keeps the documents as text files rather than in a database. I found it was the best balance between ease of setup, ease of use and sustainlability of the data. – Newtopian Jul 27 '16 at 13:51
  • The problems in computer-aided science that @rwong describes are present in most institutes I have worked (physics and astronomy) – steffen Jul 28 '16 at 08:20
2

Don’t bother so much on the methodology but try and focus more on what you need to keep track of, your requirements, for the software development itself.

Having done a short stay in a position relatively similar to yours here is what I can extract from my personal experience.

Algorithmic exactness

Probably the most important aspect, you should be able to prove that your software does what it was designed to do. Here automated testing is your best ally. I realize it can be hard to do without a proper data set but actually you should make a habit of creating your own data sets. Their purpose is however somewhat different, you are not trying to extract trend from the data but ensure the software produces predictable and correct results from a known data set. So for pattern recognition for example you don’t need a multi gig genetic makeup, just a few lines of text could be enough to ensure the algorithm detects the pattern.

I used to craft my data to represent corner cases, impossible cases. I tended to focus more on the extremes than on the expected norm. Many times I can remember testing for something impossible only to see this situation arise in the actual data set. Had I not tested for it I would not have put in place the error detections and logging necessary to identify potential corruptions or errors in the data set. TDD is a good fit for this part though creating a good test set is I think more important to regardless if you do it before or after the actual code.

Versioning

Too often this part is left out. A good versioning schema for your code and produced packages/executable will help immensely to keep your progress in order. To be able to recover exactly the code that was used to create results previously obtained can help when tracking down bugs or discrepancies. Branching can help too when experimenting with different approaches or algorithms.

Make sure you tag the code used in the actual calculations, Check out semantic versioning if you are need help on naming the versions.

Automated build

A corollary to the point above. Make sure you automate as much as possible the process of building and packaging your software. You don’t need to go full monty, just enough to make it trivial to create the final system from source and dependencies. The goal here is to save you time but also to have a reproducible mean to re-create the software from source, including dependencies and other externals. Groovy, Maven, ant, Scons, cmake, are but a small sample of build automation tools and scripting systems that can help.

If you want to go the extra mile, install Jenkins or teamcity or any other continuous integration system. Added bonus if you have to maintain multiple servers or workers for distributed computing. Most of these system will have means to help in the maintenance. Plus you will be able to fully automate the test runs so you don’t need to wait for the results before you continue, just commit and receive a mail later. I've had a system that took hours to get through the test sets. Putting up this automation was the best investment of my time. Especially so if you already have the scripts in place to build everything.

Environment Isolation

Researcher spend inordinate amount of time isolating a single or small set of variables of interest from complex systems through their protocols. This should also be extended to your software development practices. You can also check for containerization with Docker or Vagrant. It will give you better control over the environment under which the software is run.

You don’t need to have a big team before this pays off, I was alone most of the time yet benefited greatly putting these in place. The peace of mind and time saved it provided far outweighed the overhead it costed me doing this.

Newtopian
  • 7,201
  • I usually left the code as is when I am done with it, so the latest version is the one used for the calculations, so I might need to improve that. Also about the algorithmic exactness, shouldn't I assume that the libraries I use work properly? – llrs Jul 27 '16 at 15:31
  • you can, though I've done tests on external dependencies before but that was rare... it's your own code you should be testing, did you use the libraries correctly ? Does it fail redictably (your code) ? etc. – Newtopian Jul 27 '16 at 19:23
0

  1. Can you use R? That's what it's for.

  2. Keep your code simple. Go for readability, and don't worry about performance unless it's a problem. Methodologies exist to try to keep teams of programmers from putting bugs in each other's code, even if the team is one person.

  3. That said, coding discipline is super important. I've seen code from highly-trained advanced scientists and mathematicians, and it's awful. They totally ignore formatting. They squish the code together like it's vacuum-packed. Their variable names are totally mystifying. They don't write comments, or they write inscrutable comments, or the comments say one thing while the code says another. Don't do those things. Always think ahead to the future changes you or others might have to make.

Mike Dunlavey
  • 12,835
  • 1
    I am using R, hope my code is simple enough to spot bugs I could have written and any mistake I could have done. I follow Google R code formatting style, and I would like to think that comments are useful to explain why I take such decisions in the code. – llrs Jul 27 '16 at 13:24
  • @Llopis: Then I'd say you're on the right track. – Mike Dunlavey Jul 27 '16 at 13:25
  • @Llopis in team-based software development, it is routine for team members to ask another to review the code, based on the assumption that more eyes can catch more mistakes. Unfortunately, in your situation there is nobody to review yours, and the secrecy culture in research will have prevented you from letting others (outside your workplace permissions) review your code. – rwong Jul 27 '16 at 13:29
  • 1
    @rwong in fact I am allowed now to share my research code, so anyone coulde review it on github – llrs Jul 27 '16 at 13:41
  • @Llopis: All the more reason to make it readable. One thing I try to do is give a very small tutorial (in comments) on the subject matter, because chances are the reader's expertise will differ from mine. – Mike Dunlavey Jul 27 '16 at 13:48