23

For more than 6 weeks now, I have been attempting to contact a post-doc or their (former?) PI to request access to either the source code or software of a tool that was published in BMC Bioinformatics. The authors did not provide the source as a supplemental file, but assure the reader in the article that it will be made available upon request. This is the first time requesting source code of a research group, but I have yet to receive any kind of reply to my polite requests.

I am unsure how to proceed. Pursuant to the guidelines for publication in the journal, "[i]f published, software applications/tools must be freely available to any researcher wishing to use them for non-commercial purposes, without restrictions such as the need for a material transfer agreement." This group is located in Europe, and it is highly likely that they should have a working knowledge of English, after all, the publication and their websites are in English, so I don't think there a language barrier exists.

Possible actions that I am considering:

  1. Contacting the managing editor for the publication to explain the situation, and see if his/her email attracts more of a response.

  2. Contacting any granting agencies who have provided supporting funding for the project to determine whether they have stipulations about providing source code.

  3. A phone call to the corresponding author.

  • 13
    I think both of the proposed actions can perceived as (extremely) hostile by the other group, so I would not do that. If you are familiar with how academic labs work: that code maybe is on a hard disk or a long-gone postdoc in an undocumented form or documented only in Russian/German/French, whatever is their mother tongue. It may be code written in LISP or Pascal, whatever that favorite of that guys was. Best case scenario is that someone trying to fix this for you, worst case scenario they just ignored and forgot your message. Your best chance is to have a normal relationship with them. – Greg Nov 05 '14 at 18:46
  • 1
    FYI: Reference on the time it takes for a source code used for a research article to be released; Reference on availability of source code used in computer science research articles. – Franck Dernoncourt Nov 05 '14 at 21:34
  • 14
    I highly disagree with the majority of answers. Most likely the research (and software) was funded with taxpayer dollars. The OP should not be a pushover. He should demand they fulfill their promises to taxpayers. There is no need to be polite. OP has already waited too long. – emory Nov 06 '14 at 01:03
  • 2
    To go along with @emory's answer, most funding (at least in the US) also has data retention policy requirements attached. I would think the same is true in Europe. They'd obligated to keep the source code around for several years after the funding ended. If they lost the source code then they'd probably in violation of the terms of their funding. – DaoWen Nov 06 '14 at 03:38
  • Can you provide more info on what steps you have taken to contact the authors? For instance, if there were mutliple authors, have you tried contacting all of them? Have you asked any of your professors whether they know any of those researchers, and perhaps have an alternate email for them? I think contacting the journal editor is a reasonable move when the time comes, but you should really exhaust all less extreme measures first. – BrenBarn Nov 06 '14 at 03:49
  • 3
    Have you considered simply giving them a phone call? –  Nov 06 '14 at 12:12
  • 1
    @DaoWen: I have never heard of such a policy. From what I have seen here in Europe, at least in some fields/with some funding agencies, only the written reports about projects matter. These reports have to be handed in to certain libraries where they will be stored indefinitely. Whatever was prototypically developed in the course of writing those reports is considered irrelevant by the funding agencies and is likely to disappear when the respective employees quit, unless someone makes specific provisions to keep that code. – O. R. Mapper Nov 06 '14 at 13:36
  • 1
    @O.R.Mapper http://osp.fad.harvard.edu/content/retention-of-research-data-and-materials "Harvard and its researchers have legal, institutional and ethical obligations to manage and retain records of research conducted at Harvard. These obligations are not new and are not unique to Harvard: they arise from express provisions in agreements with federal and other research sponsors, overarching regulatory requirements relating to funded research, and fundamental precepts of research integrity." Even if 100% of a research team got hit by a bus and died, Harvard would remain obliged. – emory Nov 06 '14 at 16:00
  • 1
    @emory: Sure, I believe you. Kind of the "opposite case" exists as well, though; I am more familiar with funding grants that explicitly guarantee that the involved parties have to reveal whatever code was created while conducting the project only during the project duration and only to the other project partners. Any kind of funding for maintaining the data/tools created in the project after the project has ended is virtually impossible to obtain. That said, that usually refers to projects that involve both universities and industry partners, and the latter insist on such clear rules. – O. R. Mapper Nov 06 '14 at 16:07
  • 1
    @Roland - I had not considered that option actually, so I have added it to my question as well. –  Nov 06 '14 at 17:25

5 Answers5

24

I think contacting the editor of the journal is your best bet. Contacting grant agencies will most likely not warrant a reply, and I don't imagine many of them have stipulations for sharing code (yet).

That said, I have been in a similar position numerous times, and I have had very little luck every obtaining the code. The editor will most likely not be willing to retract a paper because the author's won't share, and they have little incentive to do so, since it will at most garner a single citation, but could lead to more problems down the road (e.g., the code is buggy and you can't reproduce their results, etc.)

Another tip would be that senior people (i.e., PIs) usually have more luck at this kind of thing because they are harder to ignore and/or have contacts, but it can be harder to get the to actually do it, because it can become political.

Danny W.
  • 3,249
  • 19
  • 22
  • I agree that the editor will likely not retract the paper, but I am pretty sure that it will make him more careful handling the next paper by these authors. If the editor takes up on this, it should be a strong incentive for the authors to actually share the code. – silvado Nov 06 '14 at 13:26
  • 2
    It would be interesting to know how this went. – Captain Emacs Feb 15 '16 at 20:02
9

Stop. Do not do any of the things you are thinking of doing.

The authors did not provide the source as a supplemental file, but assure the reader in the article that it will be made available upon request.

Have you thought that they need to polish the code before releasing it? The fact that I am planning to release my source code does not mean that I have to do it now or whenever suits you.

Pursuant to the guidelines for publication in the journal, "[i]f published, software applications/tools must be freely available to any researcher wishing to use them for non-commercial purposes, without restrictions such as the need for a material transfer agreement."

I have seen journals like that in my area (CS). Still, this initial rule proposed when those journals came out, might atone through the years. Since in certain areas, conferences are the main publishing venue, journals sometimes "relax" their original rules to get enough submissions to get them going. So, I would not count towards this rule to pressure the journal or the authors of this work. Check out some other works on this journal. Do they actually released the code? If not, then releasing the code is the exception and not the rule.

Also, "software applications/tools must be freely available to any researcher wishing to use them for non-commercial purposes, without restrictions such as the need for a material transfer agreement." does not necessarily mean releasing the source code but just the binary or a web-application created from the code. Where did you make the assumption that they should give their source code to you? The word "tool" refers to full apps and not original uncompiled source code.

Contacting the managing editor for the publication to explain the situation....

And what do you think the editor would do? Punish the authors because a random stranger on the internet tells him something bad about them? You can rest assure, this action will have little effect on the authors and only reflect bad on you.

Contacting any granting agencies who have provided...

Why do you assume that under the rules of their funding agency they should release everything as open source? I have worked in many research projects in Europe and I have never heard of such a strict rule. Perhaps there are some projects or agencies demanding that but I do not think it is the norm as you suggest it is. In many projects, participants are commercial companies and they are usually not interested in sharing their work with anyone else (except the project partners and only during the project's duration). Enforcing such a rule would make all commercial companies to not want to participate and that is against the policy of funding agencies.

Have you ever stopped to consider that the PI perhaps relocated and did not get those emails? Are you 100% sure that the authors did not reply your emails on purpose? And even if they do, are you sure that they broke some rule as you assume they did? In your shoes, I would not be too sure. And starting a full-scale war, will do more harm to you than them.

Also if you want something, be nice. Sometimes it does not work. OK. Bullying people into doing what you want is not an efficient long-term policy.

Cape Code
  • 27,102
  • 8
  • 98
  • 151
Alexandros
  • 18,921
  • 3
  • 49
  • 84
  • 15
    Very detailed reply. Let me rebut:
    1. The article is >18 months old now. How much "polishing" can the code still require? This claim makes sense in a recent publication, but not one this old.
    2. The software in this case is a python script, so releasing the tool is the source code. Not only is that the journal policy, which they should be adherent to, it is quite common in bioinformatics to release the software (not necessarily source code, but a binary/web interface at a minimum). Further, their abstract and paper state the software is available upon request, thus, my request.
     –  Nov 05 '14 at 18:43
  • 1
    Yes, but your main assumption about those owing to give the src code is wrong, as the journal policy says nothing about source code but "applications/tools" – Alexandros Nov 05 '14 at 18:45
  • That's fair, and I don't necessarily care about the code, but I do want access to the software. Neither of which are "public". –  Nov 05 '14 at 18:46
  • I propose to contact the editor not to impose a punishment, but it may simply be that my email as a graduate student is simply ignored or trapped in spam, whereas their email request should not be. Internet searches strongly suggest they are still at their home institution, and are still covered by the same (multi-year) grant. –  Nov 05 '14 at 18:47
  • 1
    I think you are mis-reading an aggressive tone in my request. I have been very polite in my requests, and continue to be, but I also do not know if I am being deliberately ignored (for whatever reason) or if they did not receive my emails. However, they have made certain explicit promises in their publication, and I am trying to get access to a research tool. I am not interested in reproducing or verifying their own work, but to simply apply it to my own. –  Nov 05 '14 at 18:50
  • 11
    While this advice is generally sound, for this particular journal, availability of software described in a software article is a hard requirement. – ff524 Nov 05 '14 at 18:50
  • 1
    As for the funding agency policies, it can be stipulated that software created under funding from a grant needs to be made available (these exist in Canada), but it may also be that in the proposal they have made such claims. I do not think this would be a fruitful avenue to contact the granting agency, but I was just brainstorming my options. –  Nov 05 '14 at 18:51
  • 5
    @ff524 If the journal would take this requirement seriously, they would have asked the code submitted in a repository just like crystal structure files required e.g. for Chemistry papers. Making that optional is a sure way the many authors will not have a reliable solution for code sharing. It is poor practice. Academics are very bad at development / management of code. – Greg Nov 05 '14 at 19:05
  • 6
    So you would advise being a pushover and not caring about scientific accountability. A scientific article must give the reader the knowledge required to retrace the steps leading to the result. That is why the journal requires that the software is available, without it, one of the steps is missing, and the result cannot be verified. And the code most certainly should not be changed before release, the version that the article is built upon is the version that will most accurately retrace the steps. If the author has a newer better version, he can release both. – aaaaaaaaaaaa Nov 05 '14 at 23:32
  • WRT funding agency requirements, I don't know anything about European funding agencies, but if this was US funded research and I was not getting what I wanted through normal channels, I would just FOIA them. – emory Nov 06 '14 at 00:59
  • 14
    The idea that they need to polish the source code before releasing it is not compatible with a commitment (according to the journal guidelines) to provide the software "upon request". If it is requested before it is polished, it needs to be released in its unpolished state. – BrenBarn Nov 06 '14 at 03:47
  • 4
    The idea that the status quo is more important than transparency and correctly done science is extremely annoying and does nothing but slow down progress. So what if most other papers don't bundle their code? Then they are bad papers which cannot be reproduced. It is not in any way an excuse to avoid providing the tools with which results are obtained. – Svalorzen Nov 06 '14 at 15:18
  • @Svalorzen. My answer was probably written way after, you abandoning your PHD. I am sure you had your reasons, but my answer was not one of them. – Alexandros Nov 06 '14 at 15:22
  • 2
    @Alexandros I've deleted that part because it was not relevant, but what I meant was about the general attitude. I've seen horrible things written in papers simply because authors could reasonably expect to never be called out for code. Encouraging behaviour that allows this to happen is not a thing I like. – Svalorzen Nov 06 '14 at 15:27
  • 2
    "Have you thought that they need to polish the code before releasing it?" Possibly. But if that was the reason, wouldn't they have replied to the request saying something like, "Actually, the code's in a bit of a mess but we're working on it!" – David Richerby Nov 06 '14 at 16:22
  • People in bioinformatics take the requirement for software availability particularly seriously. My for my last publication I was required to dump all code in github and zenodo as well as provide it as a supplement to the paper. If the authors need to polish the code they should do so before publishing. If they are commercial and don't want to freely release, then they don't also get a paper. – Ian Sudbery Nov 15 '17 at 15:20
5

It's perfectly possible that your polite request is sat in a queue of jobs behind several others. If you mailed them six weeks ago, then that would just about coincide with the start of the teaching term at many institutions (such as my own). That time of year is quite crazy, so it might just be that they haven't got around to it yet.

Failing that, do you know of anyone else who might need to use the software, and might also send a request? If people feel that their "product" is actually in demand, that might act as a spur to further action...

And, to reiterate the points already made, do not contact the journal - at this stage - and absolutely do not contact the funders until all other possibilities have been tried (and, even then, think very carefully about taking this action).

GeneMachine
  • 966
  • 6
  • 11
  • 2
    What I don't understand is why this should be a logistical problem for the original researcher - what is there to "get around to"? This is exactly why the web was developed in the first place: to facilitate the sharing of academic materials of this sort. Drop a tarball on a server and it's done and dusted. This doesn't take more than two minutes, and should have been done when the paper was published. – Jon Kiparsky Nov 06 '14 at 16:46
-1

Do you absolutely need the source code or the tool they created to reproduce their work? If so, then I think them coughing up the code would be paramount for you and any other group wanting to pursue or validate their work. If not.. then try coding your own solution.

I know, I know, your logic may be that it would be easier to start with their source code and build from there. IE: see how they did it. But, what programmers since the dawn of time have learned is it's easier to start and build your own code then it is to take on and learn someone elses. That's why programmers love to go into new things saying "we need to start from scratch". It can be a royal hairball trying to untangle someone elses' code.

If their research provides a basic process that their code simply helps expedite, then try coding your own solution and see if it also works. If they said they would provide code-upon-request, contact them again, but remain friendly. This isn't something to burn a bridge over. Research teams can sometimes get pulled different directions (esp. depending on who they work for), and a past project may get filed away and all the resources they used for it (hard drives, email addy's, etc) may get mothballed. So, asking for the code may require someone spending time digging through archives and such. (In retrospect, if they say the code is available they should have tossed it on a publicaly available repo, so folks like you could grab-n-go without having to pester them). Research goals also change over time. The person writing up the thing you read may have though the code would be available upon request.. but, the folks that funded the research may have changed their minds ... perhaps they want to patent the code as a tool to sell later?

I think you should contact them again, and simply ask that you requested the code, haven't heard from them, could someone please let you know if the code availability has changed, and if so, why? Be polite about it. At this point you just want a response saying whether you might get the code or not.

blahblahblacksheep
  • 1
-2

I think you need to forget about how you do things on the internet and remember how you do them in the real world. The fact that you do not think they have treated you with the courtesy and respect you believe you deserve does not justify you treating them any worse then you would hope to be treated.

You have asked for something, you didn't get it. Move on.

Paul Smith
  • 217
  • 2
  • 6
  • 6
    Or, he has asked for something, but hasn't gotten it yet. It isn't like an email requesting that I dig up something and send it off is the highest thing on my priority list... – Jon Custer Nov 05 '14 at 22:01
  • @JonCuster: well, that's a matter of how you relatively prioritise different promises you've made to different people. Yeah, the promises you made to your employer (and students) out-rank the promises you made to the scientific community at large, but only because eating (and students) are more important than your scientific contribution ;-) – Steve Jessop Nov 06 '14 at 12:17
  • 1
    Well, that plus how many random emails one gets asking for stuff to be done! Right now I have waaaaaay too many emails in my in box, most of which are high priority to the sender... A few weeks or months to dig up some old code and package it neatly to ship off to someone I don't know at all? That would be a pretty fast response, actually. – Jon Custer Nov 06 '14 at 14:01
  • 1
    @JonCuster Do you expect your papers to be read only by people you personally know? Preparing the software together with the paper before actually publishing and then keeping it accessible online seems a more sensible solution that also allows others to verify your work. Otherwise it's asking people to take your results by your word. – Svalorzen Nov 06 '14 at 15:23
  • I don't appreciate your tone at all. I am not angry or upset, and I have no ego in this request. It is simply asking for a tool, from one peer to another, for a simple tool. My correspondence has been patient and courteous, and I am not trying to bully them. I am trying to keep them honest and to uphold their promise to provide either source code or a tool (again, this is a python script, so it is equivalent to the source code). –  Nov 06 '14 at 17:23
  • 1
    My tone is exasperation with the attitude of entitlement that pervades both your original post and your subsequent responses. Who gave you the right or the authority to keep them honest? Have you entered into some sort of personal or financial contract with them? Having access to their material will benefit you, but you are doing nothing to benifit them in return, instead, you are proposing barely disgused threats to get what you want. – Paul Smith Nov 11 '14 at 15:24