40

We have undertaken a small statistical study of M.S. students in our department, including their application information and their eventual performance in our program. The goal is to develop criteria for making admissions decisions for new applicants based on their likelihood of success in our program (as predicted by the performance of recent, similar students). I won't get into the details of the methodology.

One outcome of this study was that different attributes predict success for students with undergrad degrees from different countries. For example, for applicants to the program from schools in country X, but not schools in country Y, undergrad GPA is correlated with the students' GPA in our program; for schools in country Y, but not X, GRE scores are correlated with the student's GPA in our program. (I'm simplifying a lot here.) A "toy" example I just completely made up to illustrate is shown below:

enter image description here

(of course, for real applicants the criteria and the relationships are more complicated)

There are many possible reasons for this: for example, we could think that the grading system is more consistent in X so undergrad GPA is a better predictor there, and in X students study "for the exam" so the GRE becomes virtually meaningless as a measure of knowledge. I could speculate, but I don't think it would be helpful. The bottom line is, we find that different factors predict student success among different populations.

Therefore, if we wanted to admit students based on their likelihood of achieving a certain GPA in our program, we would apply an undergrad GPA cutoff for students from X, and a GRE threshold for students from Y. (Again, this is vastly simplified from the criteria our study actually suggested.)

Is it fair to apply different criteria to students with undergrad degrees from different countries in admissions decisions?

Does the answer change if this would significantly skew the admissions decisions in favor of a particular country of undergrad study (because statistically, applicants to our program whose undergrad degrees are from X have done much better than those with undergrad degrees from Y).

My concern is that we're effectively saying, "Students with undergrad degrees from Y with a GRE score < T will be rejected, but students with undergrad degrees from X with GRE < T may still be considered for admission (pending other criteria)."

On the other hand, if we ignore these statistics and reject students with undergrad degrees from X with low GRE scores, we are rejecting applicants even though we have no valid reason to believe that they won't do well in our program.

To those who doubt the results of the study:

  • When I say the study is "small" I don't mean it isn't statistically significant - just that it was not designed to be generalizable beyond our applicant pool. (This is the same reason why I won't give too many details about the study - I don't want anybody to read it and try to generalize from our results.)
  • As we know, the sample size is not the only factor that determines whether a given effect is significant. We found that the results are significant, given the sample size.
  • The results also seem "sane" (which of course is subjective). It's not unexpected that undergraduate grading standards (and the standard-ness of grading standards) differ by country; or that different educational systems and cultures prepare students differently for standardized exams like the GRE and TOEFL; etc. The specifics of the results (i.e., which criteria are good predictors for which undergrad country) are consistent with what students who studied in those countries have told us about grading standards, student culture, and exam prep. So, we really have no reason to doubt them.
ff524
  • 108,934
  • 49
  • 421
  • 474
  • You are assuming that different schools in the same country have consistent(or same) grading standards. – Nobody Mar 03 '14 at 04:52
  • 2
    @scaaahu We are not - the school is also considered in the analysis, but turns out to be less (statistically) important for our particular applicant pool. – ff524 Mar 03 '14 at 04:53
  • 8
    Of course it is. Moreover, it's unfair to apply the same criteria to all students from the aame country, instead of varying the criteria by the undergrad institution. Alternatively: You aren't using different criteria; the criterion for all applicants is their predicted GPA. – JeffE Mar 03 '14 at 04:55
  • For our applicant pool, the undergrad institution is statistically significant only for a small minority of applicants; for most applicants, the criteria they would be evaluated by depend mostly on the country of undergraduate study. – ff524 Mar 03 '14 at 05:23
  • 1
    @JeffE every answer so far disagrees with your comment - care to elaborate in an answer? – ff524 Mar 03 '14 at 10:19
  • Not an education professional here, but I'd like to give my 0.02... If they're already taking a standardised test such as the GRE, then why would you further rank them by country of undergraduate degree? Also aren't graduates interviewed by their respective professors/admissions officers? What you want from them isn't just test scores, but to see their enthusiasm to learn, contribute to the academic world. – Populus Mar 03 '14 at 13:41
  • @Populus we don't interview M.S. students, we judge them entirely on their applications. Thus, the reason for the study: to find out how to better judge their ability to succeed based on the information in their applications. – ff524 Mar 03 '14 at 14:40
  • Then I'm not sure why you're even teaching them, if you just want to churn out masters degree holders that you already know will succeed. Education institutes should be looking at teaching people how to succeed, people don't just inherently become good at something (except for a handful of exceptional ones). If you think the GRE results can be rigged, that's another matter and you should really be looking at how to change that instead of just generically labeling groups of people due to their geographical location (a.k.a. stereotyping) – Populus Mar 03 '14 at 15:08
  • 8
    @Populus: It's well-known that GRE and other "standardized" tests are not administered in a standardized way worldwide. And why would it be up to a single department recruiting students to try to fix the irregularities in the GRE? – Ben Voigt Mar 03 '14 at 15:16
  • Not asking 1 department or 1 person to fix anything themselves, but it would help to get the ball rolling instead of just saying it's broken and then step over it. If you think GRE is not good enough, then you can either put some actual work into it: interview your students, or create another test after the GRE to further screen the ones you don't want. Screening people based on skill and knowledge is fine, screening geographically is unfair and plain laziness. – Populus Mar 03 '14 at 15:30
  • 5
    @Populus We're not proposing to use geography as a criteria for admission; we're suggesting that different criteria measure skill and knowledge in different parts of the world, and wondering how we can fairly account for that. – ff524 Mar 03 '14 at 15:33
  • @ff524 I wouldn't use geographical location as a variable for anything relating to admissions. I would either test all candidates individually, or look at their past achievements and qualifications, you can say look at someone's A-Levels or GCSEs or SATs...etc., and rank them according to their performance in those standardised tests, since it's a clearly different standard than the GRE you can then rate your candidates differently according to the different tests that they have done. This doesn't eliminate some edge cases, but nothing in this world is risk-free. – Populus Mar 03 '14 at 15:40
  • 3
    @Populus Not every country has standardized tests, and the tests are different from one country to the next! You're suggesting that we use different exams to evaluate students in different countries? – ff524 Mar 03 '14 at 15:42
  • If you don't have enough information on the applicant, give them another test. And no, not a test for each country, it will be a test that your institute has to maintain for itself, and I hope you're going to confident with your own tests... – Populus Mar 03 '14 at 15:45
  • I can see why this would raise a lot of red flags. Still, I commend the effort, in that it's trying to accomplish the main goal: to admit the students who have the best chance of succeeding. I don't sniff any malicious intent, but it's still a legal minefield. – J.R. Mar 03 '14 at 23:20
  • 1
    I wonder why you would use GPA in the graduate program as a measure of success. Surely completing a decent thesis is the only really measure of success? Or going on to a successful career afterwards? I have not really cared at all about my graduate school GPA, and if a class seems irrelevant, or boring, half way through I just stop going. – Steven Gubkin Mar 04 '14 at 00:34
  • 2
    @StevenGubkin In my dept, a thesis is not required for M.S. students. Also, student career outcomes info is not available to me (I believe those outcomes surveys are anonymous, so we can't map them to admissions data.) In any event, the criteria of success is as follows: Successful = GPA >= 3.0, Unsuccessful = "On academic probation" = GPA < 3.0. I agree that finer GPA distinctions would not be particularly useful. – ff524 Mar 04 '14 at 00:48
  • I see. That does make sense. – Steven Gubkin Mar 04 '14 at 01:33
  • 3
    You (the OP) are getting very misleading criticisms of a defensible procedure, based on misunderstanding of how the statistics works. There are many (huge) errors here, such as assuming that people in the "negatively discriminated" country will have lower, or uniformly lower, chances under the nationality-aware statistical model. The main problem is assuming discrimination if the result of a fictitious procedure "all else equal, change the nation of origin" alters the chances. That is an incorrect way to interpret the output of the model when variables correlate. I can add an answer later. – zyx Mar 06 '14 at 18:34
  • @zyx still planning to add an answer? – ff524 Mar 14 '14 at 01:33
  • I am starting with comments, answer is in progress but will take longer. – zyx Mar 17 '14 at 07:40
  • For your information, http://academia.stackexchange.com/q/11984/546 Please pay attention to the last paragraph of that question. – Nobody Mar 31 '14 at 09:16
  • @scaaahu what's the relevance? I don't understand. – ff524 Mar 31 '14 at 14:13
  • The OP of the other question claims that people from unknown colleges with higher GPAs getting in to places like Stanford and he cannot get to top 20 school because of low GPA. Again, this is his claim. We have no idea if he is good or not. My point is that this is an example a student believes he could not get to a school he wants only because of low GPA. – Nobody Apr 01 '14 at 02:39

8 Answers8

15

I'm going to advise against.

  • You say your sample size is small. And honestly, unless you hired a professional statistician, I have doubt the analysis was carried out properly or shows exactly what you think it shows. Good statistics are hard to do.
  • Assuming my google results are truthful, National Origin is a Protected Class in the US. You can't discriminate against people because of it. I assume this is an american institution as you talk about GRE, but similar applies in other places. EDIT: OP has since clarified a misunderstanding of mine - the judgement is based not on the student's country of origin, but the country of the school they attended, which makes me less certain this point still applies.
  • You're department is actually, seriously proposing to say to people "Sorry, while you have the same official qualifications as another candidate we accepted, we're rejecting you because of your country of origin"? I mean, really? Whoa. Just stop and play back how this kind of justification would sound in almost any other circumstance. Think about how the press will make it sound when they (inevitably) get wind of it.

SUBSEQUENT EDIT

So, this question interested me, and I've bounced it off a couple of other people in my lab. Their main concern seemed to be that grouping by country was just to coarse a measurement, as every country has its own mix of good and bad schools. The basic idea, though, didn't seem to elicit the same gut reaction as it did from me.

Moreover, I've googled and found at least two examples of courses that vary their requirements based on country of origin, so there is precedent:

https://www.auckland.ac.nz/en/for/international-students/is-entry-requirements/is-minimum-overseas-entry-requirements.html

http://sydney.edu.au/business/futurestudents/postgraduate_study/pg_coursework_studies#app_req

So maybe I've got this wrong, and should just be ignored. At any rate I'm no longer sure it's quite so clear cut.

Pat
  • 1,063
  • 6
  • 6
  • "Students who earned their undergrad degree from a university in X" Okay, that's a better criteria than what I thought you were doing at first (The line "Is it fair to apply different criteria to students from different countries in admissions decisions?" really confused me on the front, as there's no mention of institution there - only the student). However, having a blanket rule on universities based on country they're in still seems to have massive potential to blow up in your face later to me, legally or otherwise. – Pat Mar 03 '14 at 09:33
  • 2
    I agree that it seems wrong to have a double standard - but it also seems wrong to me to reject a student based on a GRE score, when I don't have any reason to think it will predict his performance. I can't seem to find an acceptable approach here. – ff524 Mar 03 '14 at 09:40
  • 3
    Yeah, it's not I problem I envy you for having to deal with. What kind of numbers of students are we talking about processing here? Tens? Hundreds? Thousands? Would it be possible to set a departmental test they all have to carry out and judge them on that? Or run interviews? – Pat Mar 03 '14 at 09:46
  • I'd rather not be too specific. Enough to have statistically significant results by country, not enough to be significant by individual institution (which would certainly make things easier). But interviews are not practical at the M.S. level for us. – ff524 Mar 03 '14 at 09:52
  • At this point then, I'm probably all out of suggestions, few as they have been :/. I think the reaction you've received from myself and the other two people who have answered so far is pretty representative of that you'll get from the public in general. This is potentially a huge ethical/legal/PR minefield. If you can, I'd say it may be worth organising a meeting with other concerned parties (admissions, department heads, legal, student representative, whatever is appropriate), putting your findings to them complete with all the details we can't see, and getting input on where you should next. – Pat Mar 03 '14 at 10:13
  • We use proxies to judge future performance. What you've found is that your proxies aren't that good. I think what your dept should do is try to create a better proxy. Create a tough programming problem and ask your M.Sci students to solve it as part of their app. Think outside the box a little and see if you can't create something that is a more reliable indicator than GPA/GRE. Your new thing doesn't have to be the exclusion of considering GPA/GRE scores, of course. But it might be an additional data point that improves the overall accuracy of your admissions process. –  Mar 03 '14 at 13:47
  • 2
    @shane it's really not that simple. For example, how would we verify that students submit their own work? Who says ability to solve one particular programming problem predicts success in our M.S. program? (I don't think it would). – ff524 Mar 03 '14 at 14:45
  • 2
    I think the last point would be more like: "So, you got the same score as this other candidate, but as we think you have had to overcome greater obstacles, and your university has prepared you better for this job than the exam, we are taking you in instead". – Davidmh Mar 03 '14 at 19:16
  • 1
    Of the three arguments, (1) is irrelevant to the question, which was about a presumed-correct analysis; (2) assumes that foreign nationals on foreign soil are subject to all the protections of US law (e.g., that a university cannot apply quotas or diversity considerations to foreign applicants); and (3) is not how the statistics works. You can't assert discrimination based on one of the variables (such as nationality) by "controlling for all the other characteristics" when the other characteristics, and the outcomes the study is examining, strongly correlate with that variable. – zyx Mar 17 '14 at 07:26
  • 3
    That is, it is invalid (statistically) to argue about discriminatory impact by pretending to hold everything but the national origin constant, changing the national origin, and observing that this would change the applicant rating computed using the model. If the variables include GRE Verbal or TOEFL, it is essentially impossible to change an applicant's nation from China to Canada without altering those scores. The whole point of the statistical model is to account for the difference in the interpretation of such scores as detected by performance measures that are nationality-blind. – zyx Mar 17 '14 at 07:31
  • (That's in reference to the 3 arguments at the top of the posting, before the edits.) – zyx Mar 17 '14 at 07:52
12

You asked an ethical question and got a lot of scientific, legal, and political answers.

Ethically, you should:

  1. Do your best to admit people based on merit.

  2. Be open about your policies.

  3. Make a serious and competent effort to ensure that your statistical methods are valid logically and empirically, and that they can withstand scrutiny from people who are experts in the field.

  4. Carefully consider the historical legacy of racism, colonialism, and nationalism, and work hard to make sure that you aren't inadvertently reinforcing this legacy.

Your behavior so far is probably far more ethical than that of most people in your position, since most such people are probably secretive about their practices (you publicly asked for advice) and probably apply various heuristics without carefully considering whether those heuristics could withstand professional scrutiny.

You might want to expand this small, informal, unpublished, nonprofessional study into something more serious and systematic, done by people who have expertise in psychometrics and the (very difficult) methodological issues of the social sciences.

You are unfortunately operating within real-world constraints imposed by (1) the existence of countries where GRE scores are fraudulent, and (2) the existence of countries with such poor undergraduate education that undergraduate GPAs don't mean much.

  • I publicly asked for advice - under an anonymous username. Not sure how much that's worth :) – ff524 Mar 31 '14 at 03:28
  • 5
    @ff524 I don't know what others think. It's worth a lot to me regardless of the anonymous username. At least I know what some other opinions are. – Nobody Mar 31 '14 at 03:37
11

I was waiting and hoping that @JeffE would expand his comment, since I share the opinion he expressed there, and moreover I think he is far more qualified to give advice on the subject. (Quite possibly, if he decides to expand his comment to an answer that has better facts and arguments than what I am about to offer, I will delete my answer).

DISCLAIMER: I do not know either where you are, what the laws are like there, or if your final decision can have any legal repercussions. I advise seeking legal counsel for any questions of that sort.

I think it if fine to have different criteria for students that obtained their undergrads in different countries (and/or in different institutions), as long as you can ensure the criteria are fair.

Now, how to approach ensuring that the criteria are fair, I have no idea, but I definitely know that if in some countries the GPA is indicative of a person who has potential as a researcher, in some countries it is simply not (I for example did my Masters in Croatia, and I know both people with around 3/5 GPA that made excellent PhD candidates and people with a perfect 5/5 GPA that I do not think would be capable of much independent thinking that were happy to take jobs where they have strictly defined output they have to produce and no research to speak of).

I have two different examples of using "different" criteria in the admission process, but unfortunately both of them are for the wrong level of study:

  • First one is about undergrad admissions to my former Computer Science Faculty* in Croatia. While the big majority of the candidates take a standardized test and are admitted based on that, a small amount of students (maybe 1%, maybe even much less) are invited.

    These invitations work as such: the Faculty regularly does a continuous performance study on the previous invitees according to their High schools, as well as (I assume), overall performance of students from different High schools. Each year, a number of invites is extended to High schools, who can then "award" their students with those invites.

    Based on the performance study and possibly High school size, some schools get a larger number of invites (up to 5, I think), while some get only 1 invite or even none.

    And, actually, I think it's fair. E.g. the strong mathematical High schools will get the most invites (those are the schools "prepping" their students for technical studies after all), strong general schools would get some, and weak schools would get none. And still everything would be re-evaluated year after year. Also, additionally, nobody still loses their fair chance to enroll: this "invited" students make less than 1% of the enrolled students, while everybody can still take the standardized test.

  • The second example is about the interview for a PhD position / pre-PhD internship.

    Recently, a permanent professor from my lab started looking for a person to hire for an internship with a strong chance of offering them a PhD upon the completion of the internship. I mentioned that my ex supervisor used to supervise people with a similar profile to what was required, and that I could ask him weather he has somebody interested (and good enough) to apply.

    When he received the preliminary application documentation, he asked me to comment on the profile because he was not familiar with the Croatian University system at all. I said that good grades were usually indication of a good student, but bad grades do not have to be an indication of a bad student.

    There was also some other points in the CV adding value to his application, not directly obvious to somebody non-Croatian. After talking to an ex post-doc of his (who worked a bit with the applicant), the professor decided to interview with the student.

    A few days later, he told me that based on the interview, he offered the applicant a position. He also told me that he would most probably not interview a person with that profile, if not for what me and his Croatian ex-post doc said about the "interpretation" of the profile.

I agree with both these cases. The ("absolute") criteria is indeed not the same for everybody. But still, in both these cases, the goal was to be fair, to base the decision on the applicants abilities, and finally to judge the applicants abilities and potential "on the same scale" for all applicants, just based on different information that was available, finally causing different "absolute" criteria.

*In Croatia, the Universities are not wholesome entities, and all the administrative decisions are made on the level of the Faculty. There is no identification of students with the Uni in Croatia; if you ask a Croatian studying "at home" where he is studying, he is going to provide the information about his Faculty.

ADDITION. I wanted to add that, despite what any statistical research might show, if students from different countries have the same, internationally standardized tests, then I think making a different criterion would be wrong.

Maybe, for the country X, those tests are not good indicator because students get prepped exactly for those types of exams, but there just might be a few students who learned the material in the "proper way" (as do the majority of students from the country Y), who might get rejected for no real reason.

So, bottom line, I think

  • studying, and then declaring your own criteria to interpret different (non-standardized) national grading schemes is fine (even if they use the same scale "on the outside" -- e.g. two countries which both have a national standard grading schemes on a scale 1 to 5)

  • but, if a criterion is based on an internationally standardized exam, that same criterion should be used with everybody where applicable (e.g. everybody who took that standardized exam).

    As a compromise-suggestion, I think it would be fine to say that you accept all students from everywhere if their score is extremely high (e.g. >95%), and all the students with medium-to-high scores (e.g. between 75% and 95%) will be considered based on additional criteria (where you can introduce personal interviews, standardized test from your institution, personal research statement + recommendation letters)...

Edit2: This opinion was supported when I talked to a person with a background in law (not in academia thou).:
The final verdict was: interpreting criteria that are initially different (e.g. GPA) differently is OK, but if something is standardized on an international level, it would only be OK to either take it in to account equally for everybody, or not at all. (purposefully using OK instead of "allowed" or "lawful" since I don't know the laws at your place). With a note that, unfortunately, What is lawful and what is just and fair is sadly not always the same.

Community
  • 1
penelope
  • 11,580
  • 1
  • 41
  • 82
  • The specific example you mention, of asking a trusted individual from country X for advice in evaluating students from X, is I believe very common. I think what we're trying to do is essentially the same thing, though less subjective. – ff524 Mar 03 '14 at 18:45
  • 2
    Exactly. The point was finding the "context" in which to evaluate the student. The same way the first (undergrad) study is trying to put the schools in context (i.e. you might get an invited admission if you're an excellent student from an excellent school, but not from an average school), the second one is trying to put the transcript in context (i.e. does the mediocre transcript by absolute value really mean a mediocre student or not). And of course, in your case of a large student pool, such a context has to be based on a large pool of "opinions" - evidence, and not on one individual. – penelope Mar 03 '14 at 18:58
  • Neither of the examples actually answer the question on whether or not you can officially demand applicants to meet different criteria based on where they come from. More specifically, #1 would be considered unacceptable and immoral in Scandinavia, it would be some serious sh*tstorm if something like that were to come up on the news. #2 happens on a regular basis pretty much lab I know of. It does not mean that the professor in question can go out and write it out in plain text that if the applicant is, from China/India/Croatia/Madagascar/... they don't need to meet the same criteria. – posdef Mar 03 '14 at 22:52
  • Consider asking the professor in question whether or not he would or could write the job advert in the following way: "The applicant should have a GPA of 3.5/4 (~87.5%), unless they are from Croatia, in which case 3/5 (60%) is good enough." – posdef Mar 03 '14 at 22:54
  • Well, actually, the 3/5 GPA in Croatia is more like 75% since the grade 1 is a failing grade... But again, not really, since all the passing grades are distributed as in lowest 15% of the students gets a 2, next 35% gets a 3, next 35% gets a 4, and the best 15% a 5. So, "yearly score" of 3/5 actually means that the student is "somewhere between the the best 75% and 50% of best students who got all passing grades"...or something similar like that. I'm just trying to demonstrate that to recalculate the GPA of ~87.5% to Croatian system is really not so trivial and obvious. – penelope Mar 04 '14 at 12:59
  • @posdef Ups, sorry, meant to @-reply with my last comment. But, also, I wanted to add what I edited in the question: I think interpreting the results from an internationally standard measure different across countries is not okay. But if a country has it's own country-specific grading system, studying and making criteria based on their grading scheme is better than just "wigging it" and hoping you interpreted it correctly. Also, I don't think the "job add" sounds very unfair -- I actually kept my GPA unnecessarily high during Uni, because I didn't think non-Croatians could interpret it well. – penelope Mar 04 '14 at 13:17
  • @penelope I can sympathise with your example, especially after having seen the german and danish (by far the least intuitive I have seen so far) grading systems. I definitely agree that any local grading system should be rated individually instead of a direct conversion. I just didn't interpret the OP that way; but rather as feasibility of having different GRE score thresholds for different countries (GPA is a crap method of evaluating the student anyways, especially when multiple different grading schemes are involved). – posdef Mar 04 '14 at 13:26
  • @ff524 I am quite familiar with that question :) Anyways, I hope I managed to provide a valid, albeit negative, opinion to your question. I intend to retire from the discussion otherwise :) – posdef Mar 04 '14 at 13:27
  • let us continue this discussion in chat – ff524 Mar 04 '14 at 16:13
10

I don't think it has to be bad. GRE is a measurement of how good you are at doing exams, and not necessarily the most relevant kind to your graduate work. Some universities and education systems are very good at making people excel at set fixed format exams; but they don't teach them how to think by themselves and be creative.

In my opinion, GRE is a very biased measurement in itself, and it is not very well designed (just look at the ridiculous maths part). Also, it is based on a lot of fairly simple exercises, whereas in some universities, we are used to completely different kind of exam: a few long and complicated exercises, and perhaps don't have the ability to work fast enough in simple and repetitive tasks.

Davidmh
  • 21,215
  • 5
  • 60
  • 105
  • 2
    I hope by "Ridiculous maths" you meant "ridiculously easy, to the point where it fails to differentiate great applicants to technical programs from mediocre ones." Non-technical programs oughtn't to be using the math score for much of anything. – Ben Voigt Mar 03 '14 at 15:22
  • Yes, that is what I meant. I haven't look much at GREs, but judging from the maths one, I fear many parts will likely be useless measurements: or too easy, or too specific for a certain background. – Davidmh Mar 03 '14 at 18:36
  • If the GRE is too easy you need to find another test. This is not an argument for biasing it against people from certain countries – Philip Gibbs Mar 03 '14 at 18:52
  • 4
    It is not only how difficult it is, it is also what it measures. Changing GRE for something more sensible would be the ideal option, but it is way outside the scope of a department. The question here is if they could, with what is available, make a better choice. – Davidmh Mar 03 '14 at 18:56
  • 2
    @PhilipGibbs, the purpose and quite possibly the result of the statistical analysis is to un-bias the GRE (which favors or disfavors various nationalities, languages, types of education system, etc) by validating it against real outcomes that are defined uniformly with respect to nationality, such as performance in the program. It is absurd to imagine that national origin, either per se or using undergraduate institution as (in effect) a proxy, does not play a huge role in the way that, among many other things, GRE verbal scores should be weighed. – zyx Mar 17 '14 at 07:37
7

I think the underlying questions are (ought to be) first, whether it is fair to reject a student because you cannot accurately measure their potential to do well in the program; and second, if you can perform equally accurate measurements on two groups as long as you use different criteria, is it okay to do so?

Answering the second question first: I think the answer is "yes". The reason is made apparent by advocating the alternate view: should you take students who are likely to do poorly simply because you fail to apply a more sophisticated measurement? That seems pretty boneheaded, and not very fair to the best students. You should leave the analysis up to computers, as they are good at this sort of thing and impartial, but it's a good idea if you can implement it. Don't forget, though, that data on whether a score distinguishes students within your program does not tell you about whether it helped you reject students that didn't make it in! So you should be skeptical but open to the idea if that's where the data leads.

The first question could be rephrased as: "if I know I'm not getting a good sense of some students, can I just ignore them all?". And that guides my advice there: no, that is not fair. Find a way to do your job better--to get more information so that you can get a good sense of these students, or just accept that you would rather make mistakes in acceptance than find a way to do better on your predictions.

Putting these two together: if you have equal statistical power across groups when subdividing one population, then great! Use the information. But if you end up with one group better-measured than the other, you should only apply different criteria if you can show that students who are great and do all the right things in their less-measured context still have a shot at being admitted.

(Also, simple thresholds for individual scores, e.g. "We only take people with GREs above X" are rarely a smart way to run admissions unless those thresholds are set amazingly low. You should be thresholding an overall score to get the top N students, so you'd be applying different weights for GRE in one context vs. another.)

Rex Kerr
  • 3,834
  • 16
  • 18
  • simple thresholds for individual scores - yes, I simplified for purposes of the question - the classifier is more sophisticated than that. – ff524 Mar 03 '14 at 16:08
3

I would argue that you are interpreting your data incorrectly and using the results in an unethical and discriminatory way. Your model is not identifying groups of individuals who should not be accepted, but rather groups of individuals that require additional support so that they can reach the full potential predicted by their past performance.

Consider the following example: For a variety of reasons that have nothing to do with ability or commitment, women have historically been less likely to become full professors in STEM fields than men. To reject female applicants base on a model that captures this historical effect is completely unethical. What the model would show is that female applicants need additional support to maximize their potential (e.g., flexible working hours and mentoring).

StrongBad
  • 104,237
  • 30
  • 262
  • 484
  • 2
    I don't think it shows that at all - it's more like saying, "GPA doesn't historically capture the ability of female applicants to succeed, so we should consider another criteria, like work experience, in evaluating them." – ff524 Mar 03 '14 at 16:41
  • 3
    I would argue that you do not know which data the OP has, how he has done the analysis and on what exactly is his interpretation based. Try to, just for a second, imagine that his interpretation is correct. If you apply your reasoning to e.g. sports, would you rather compose an Olympic team from 5 best sportsman in the country that accidentally all practice in the capital, or would you say that this results are skewed and entice favoritism towards capital-dwellers, and instead rather train 5 sportsman who were only good in their local city/county, in order to "reach their full potential"? – penelope Mar 03 '14 at 16:47
  • @penelope I would want to train the 5 best sportsman independently from where they were from. I would not care that historically the only people who win Olympic medals are capital dwellers. If the 5 fastest and strongest (or highest GPA and GRE) happen to live in the capital so be it. – StrongBad Mar 03 '14 at 16:54
  • @ff524 the point is that the best predictor (e.g., GPA) may be the same for both groups (e.g., male and female), but that predictions of future success are going to be skewed by historical discrimination. – StrongBad Mar 03 '14 at 16:59
  • 3
    That's not what the data shows - the data shows that the value of every predictor we have available varies (statistically significantly) according to the country a student previously studied in (at least, for our applicant pool). Answers that assume this isn't possible aren't really helpful to me, they don't address the question. – ff524 Mar 03 '14 at 17:02
  • 2
    @StrongBad As far as I can see, the OP made a study of predictors. And he is saying that the best predictor is not GPA, but something far more complex, and the country of undergrad study plays a significant part in that predictor. As far as I understand, we are not here to try and tell him that he did a wrong study (because some of us might not like the results), but rather to tell him what to do with the study results assuming they are correct. If he wanted to asses the validity of his study, I'm hoping he would have gone to a different SE site. – penelope Mar 03 '14 at 17:14
  • @penelope the OP asked if it is ethical to base admissions decisions on the results. My answer is that it is unethical since the design of the study would potentially reinforce existing discriminatory practices. – StrongBad Mar 03 '14 at 17:51
  • 1
    Given that you don't know the details of the study, I'm not sure why you assume that it reinforces existing discriminatory practices. We haven't found that, e.g., because we have historically had fewer successful students from Y we should reject all students from Y. We've found instead that the criteria that distinguish between a successful and unsuccessful student from Y are different from the criteria that distinguish between a successful and unsuccessful student from X. – ff524 Mar 03 '14 at 18:05
  • Your answer, as it reads to me, is based on the assumption that the OP is interpreting his data incorrectly. Thus, I do not think it is actually the answer to the question. But, it is your answer and I have no right to try and change your opinion expressed in this answer. – penelope Mar 03 '14 at 18:05
  • @ff524 can we move this to chat? http://chat.stackexchange.com/transcript/message/14083802#14083802 – StrongBad Mar 03 '14 at 18:44
  • The example with women is fallacious. The point is to get the best results in a, I assume, enviroment that provides the same opportunities for all. Academia in the mid XX century was not offering equal oportunities for women. – Davidmh Mar 03 '14 at 19:12
  • I completely agree with this answer. The comparison with gender bias is perfectly correct. – Philip Gibbs Mar 03 '14 at 19:15
  • What if it went the other way around? This answer seems to assume that the fluid rules would always result in barring people who would be otherwise qualified. But what if an admissions office used this data to relax admissions criteria for some, rather than make them more stringent? – J.R. Mar 03 '14 at 23:16
  • "Model is identifying ... groups of individuals that require additional support." -- This has nothing to do with the OP's question. In situations where some group tends to do less well, most or all of that tends to be predicted correctly by the distributions of credentials in different groups. If group X has a worse distribution of credentials than other groups, that would lead to the prediction and observation of lower overall performance for group X. The OP's question is about additional (statistical) effects of group beyond what credentials alone predict. – zyx Mar 19 '14 at 18:16
  • For example, you used women as a comparison, and women do have a lower distribution of things like GRE math scores. But studies that use the OP's type of analysis quite often find women performing better than credentials alone predict. This is the opposite of the historical pattern or relative group standings assumed in the answer. And the admissions argument that it would make is, as a matter of "merit" or performance prediction, to admit more women than credentials alone would suggest. – zyx Mar 19 '14 at 18:25
1

Does the answer change if this would significantly skew the admissions decisions in favor of a particular country of undergrad study (because statistically, applicants to our program whose undergrad degrees are from X have done much better than those with undergrad degrees from Y).

My concern is that we're effectively saying, "Students with undergrad degrees from Y with a GRE score < T will be rejected, but students with undergrad degrees from X with GRE < T may still be considered for admission (pending other criteria)."

To my ears this sounds analogous (with certain exaggeration) to: "we only want rich, white men from successful parents that could afford to send them to good schools through out their childhoods."

I realise that this is not what you intend but you got to realise that having different criteria based on the country of undergrad studies is unethical, biased and I can't see how you could implement something like that without having major headaches. You can, by all means, have different criteria for an individual (for instance by having an interview) but if you clump up an entire population based on some statistics, which you can't elaborate on for some reason, I call that discrimination.

Bottom line is, you cannot evaluate the chances of an individual being successful or not, without giving that person a fair chance. If you deem a particular GPA/GRE score to be "good enough" for your graduate program, all applicants that satisfy that criteria should be considered good enough, regardless where they come from. Any additional selection criteria could be justified only if it warrants additional information, such as TOEFL score for those who don't have English as a mother tongue.

One other option would be to calculate "success rate coefficients", something like a multiplicative factor for the GPA/GRE score for applicants from different undergrad institutions, which could in theory be a fair assessment, but practically unfeasible considering the number of possible institutions involved.

Another alternative would be to devise a test for your institution, that you consider to be more fair than using GPA or GRE score. But even that, judging by your comments, is not an acceptable solution. Honestly it sounds a bit like you just want confirmation of some sort.

Community
  • 1
posdef
  • 17,711
  • 5
  • 75
  • 125
  • 7
    It's "Country of undergraduate study," not the applicant's nationality. How do you propose we adjust undergrad GPA for different grading standards between universities if we apply a single numerical criteria to all? – ff524 Mar 03 '14 at 09:21
  • Dont use GPA as a proxy for student's potential for success, if you cannot rely on it. – posdef Mar 03 '14 at 09:23
  • 1
    The point is we don't have any single factor we can rely on for everyone - we've found that the indicators of student success are different for different populations. Hence, the question. – ff524 Mar 03 '14 at 09:26
  • 1
    I understand the essence of the question, and the need to ask it, I do... and my answer is that if you are operating within the boundaries of fair treatment and equal chances (which you should) then it's not as simple to change the thresholds and change the criteria depending on the candidate, because it is not fair to the person in question. If it is about evaluation of significance of the GPAs, you can do an assessment of several hundred universities from which most of your applicants come from. But it is a daunting task – posdef Mar 03 '14 at 09:46
  • @Downvoter: care to explain? – posdef Mar 03 '14 at 14:45
  • @penelope I did in fact edit the question to clarify that point – ff524 Mar 03 '14 at 16:45
  • I dont think this answer needs any edits in light of the changes to the question. It is a good answer +1 – Philip Gibbs Mar 03 '14 at 19:20
  • 5
    Huh? You think the meaningless numbers on the GRE are absolute indicators of research potential? I've never heard of a good physics program in the US even looking at GRE's from, say, China, because experience has taught that every student there memorizes the answers, whether the student is capable of doing research or not. So as long as people game the system along national lines, it's discriminatory to control for it statistically? –  Mar 03 '14 at 20:45
  • @ChrisWhite you may weigh in different components to varying degrees in evaluation but if you formulate your criteria like: "GRE score > X" for some applicants and "GRE score > X + 100" for some others (I have no idea about the actual scoring system for GRE), based on country the applicant has graduated from, that's called discrimination. – posdef Mar 03 '14 at 22:42
  • @penelope no rage, just frustration. the question was edited as the OP was kind enough to point out, so my first response was associated with what I read at first. I don't have any expectations regarding your vote, but hopefully this version of the answer depicts my ideological standpoint better – posdef Mar 03 '14 at 23:13
  • @posdef I added an image to the question, which I think illustrates that it is not a case of "GRE score > X" for some applicants and "GRE score > X + 100" for others. I'm not looking for confirmation of anything, just genuinely asking what the best way to evaluate students is given different predictors of success in different populations. – ff524 Mar 04 '14 at 02:48
  • @posdef I appreciate the "negative" answers too, but they don't seem to acknowledge that if we follow the advice and apply the same rules for everyone, we reject many candidates who we really think would do well (e.g. low GRE candidates from X, low GPA candidates from Y in my toy example). I wish some answers would address that issue. – ff524 Mar 04 '14 at 14:06
  • 3
    we only want rich, white men Please don't make inaccurate, offensive characterizations of what the OP is trying to do. Please don't impute racism to the OP when race was never mentioned. Please don't impute classism to the OP when class was never mentioned. Please don't impute sexism to the OP when sex was never mentioned -- and BTW sex is not strongly correlated with the country in which one does one's undergraduate studies. –  Apr 01 '14 at 05:39
  • @BenCrowell you should read the whole remark and properly so; I said "to my ears", "analogous to" and "... with certain exaggeration". I am not imputing anything, and I did not accuse the OP with being a racist, or sexist or anything really. All I am saying is that the original formulation of the OP could be seen discriminative, and the global stereotype of positive discrimination is "rich, white men". If you dispute that, I have no further remarks for you, but don't reframe my words. – posdef Apr 01 '14 at 06:49
0

I would "massage" this problem by including a dummy statistical variable for the country. E.g. 0 for the country where grades are more important, and 1 for the country where GREs are more important. You could almost separate the applications into two piles this way, and tackle one pile at a time.

If asked about it, I would answer that neither grades nor GREs alone are dispositive, and they both have have greater explanatory power when you introduce the third (country) variable into the equation.

As far as I'm concerned, it is ethical to use any combination of statistical variables that satisfactorily explain performance, and to strive for "best fit" (statistically).

Tom Au
  • 5,366
  • 1
  • 17
  • 28