Outlook ranking debate Magazine responds with its methodology Are the magazine rankings transparent

NUJS and its faculty led by Prof. Shamnad Basheer are not happy with Outlook’s response to the 2010 Rankings debate. NUJS on January 14, 2011 has replied to the responses sent by Outlook and MRDA and has requested the magazine that “Anyone undertaking this exercise must be sensitive to the impact it has on students, who may end up making a wrong choice, if the ranking methodology is shoddy and yields obviously false results”.NUJS and its faculty led by Prof. Shamnad Basheer are not happy with Outlook’s response to the 2010 Rankings debate. Outlook had ranked several law schools in its June 2010 issue and NUJS was overall ranked fifth. NUJS had raised eight issues in its previous letter dated August 18, 2010 to the Outlook Magazine where it questioned the magazine’s criterion on ranking law schools and had asked it to reveal its parameters and findings..Outlook had hired Marketing and Development Research Associates (MDRA) to conduct the 2010 rankings survey..Outlook and MDRA in its response revealed some data and sought privacy protection to not reveal the entire findings. Outlook and MDRA said, ““We respect the privacy of data provided by the colleges and this has been clearly stated on the objective questionnaire provided to the colleges. No data has been shared or would be shared as it is completely private in nature”.NUJS on January 14, 2011 has replied to the responses sent by Outlook and MRDA and has requested the magazine that “Anyone undertaking this exercise must be sensitive to the impact it has on students, who may end up making a wrong choice, if the ranking methodology is shoddy and yields obviously false results. And the nature of this moral responsibility only increases when the rankings are done by a magazine such as yours that stands for openness and the truth. As we have demonstrated, your ranking this year suffers from serious lapses and methodological flaws. This has no doubt impacted the future of many who may have relied on your rankings. We therefore call upon you to immediately take steps to nuance your methodology and make a better effort next year. Our interest is that you become more evolved and better in your approach, as we think that although rankings have inherent flaws, they still serve a useful purpose on the balance. We hope that you are able to incorporate some of these suggestions as you work towards a more robust methodology in future”..A copy of NUJS letter dated January 14, 2011 is available here..What is interesting are the issues raised by NUJS and the response provided by MDRA. Here are the summary of the issues raised by NUJS in its August 18th letter and verbatim detailed responses by MDRA sent to NUJS on September 1, 2010..1. Points/Weightage for “Selection Process”.NUJS: NUJS questioned several criteria to which Outlook assigned different points/weightage. One such criterion was the “selection process” that law schools typically deploy for picking students. Although four of the top 10 law schools (NLSIU, NALSAR, NUJS and NLIU) admitted students on the basis of a centralised exam called CLAT (common law entrance test), Outlook had assigned differential points for each of these top four schools. NUJS in its letter had stated “It therefore astonishes us that these colleges should be awarded different points under the “selection process” head!”. NUJS also questioned higher marks being given to Faculty of Law of Banaras Hindu University (BHU) has been given the highest marks under this category..MDRA Response: MDRA and Outlook have been transparent in their study of colleges and have also provided in-depth information on all parameters used for the published rankings. As mentioned in page no. 42 of the published issue of Outlook (28 June 2010), the parameter “Selection Process” had a number of sub-parameters apart from the “Type of entrance examination” mentioned here. These were –number of applications to selected ratio, fee structure, age/establishment of the institute, etc. Therefore, these sub-parameters together determine the overall marks/ score under this head – not only the type of entrance examination..For sake of convenience, the same is displayed below:.2. “Academic Excellence” given relatively lower marks/weightage .NUJS: NUJS had questioned to allocation of differential weightage points. Colleges were marked on “288 marks” under the category “Selection Process”, whereas the total marks/weightage given to “Academic Excellence” was only 200. In its reply to Outlook, NUJS had said “It astonishes us that “Academic Excellence” has been given lower marks/weightage than the “Selection Process”. This appears to be a severe methodological flaw and we would appreciate your clarification in this regard.”.MDRA Response: As already mentioned in the “Methodology” section of the published magazine, “a panel of 120 experts from the corporate sector and academia were carefully selected for consultation on identifying the parameters and sub-parameters for ranking and their importance (weightages).” We respect the privacy of data provided by the colleges and this has been clearly stated on the objective questionnaire provided to the colleges. No data has been shared or would be shared as it is completely private in nature, however, certain information on some parameters has been provided in response to points below in view of the concerns raised by your esteemed institution..3. Discrepancies in marks allotted for placements.NUJS: It is rather strange that NLSIU which has had the best placements overall has been given lower marks (139.3) than NALSAR (148.5). NUJS submitted Bar & Bench conducted detailed recruitment report as one of the evidences of placement record.MDRA Response: The key parameter “Placements” is a function of many sub-parameters namely – percentage placed, number of recruiters, salary offered in campus placement (average, highest), regularity/ repetition of companies visiting the campus, illustrious alumni passed out during last ten years from the institute and return on investment. Each parameter is either a combination of questions or a key question itself is a sub-parameter..Further, the overall ranking is based on combination of objective data and perceptual survey data..4. Faculty Performance Not Taken Into Account.NUJS: NUJS questioned the missing link of faculty performance. Although MDRA in their questionnaire had asked various data about faculty performance, faculty publications (both international and national), NUJS questioned why was this not taken into account?.MDRA Response: There is no doubt that faculty performance is critical to the reputation an educational establishment and also during the process of evaluating an educational institution. In fact, researchers at MDRA have gone a step ahead and included faculty performance as a principal component of the key parameter “Academic Excellence”. Academic Excellence in itself is an umbrella parameter under which besides Quality of faculty and Faculty participation in national and international seminars many other sub parameters are examined. It also incorporates important factors such as research papers, books, consultancy opportunity for the faculty. The complete list of sub-parameters is given below:.Student-faculty ratio in the instituteQuality of permanent faculty (qualification, knowledge, research experience, classroom interaction)Quality of visiting faculty (qualification, knowledge, industry & research experience, classroom interaction etc)Low Attrition rate of FacultyRatio between practical and theoryStudent evaluation criteria being used in the course durationNumber of moot courts arranged in a yearFaculty participation in seminars (national & international)Research and consulting opportunities for facultyPublication of research papers and books by facultyPost Graduation (LL.M) and Ph.D. available in the campusRemuneration of faculty members.5. Influence of Advertisement.NUJS: Outlook had solicited for advertisements, a request which was turned down by NUJS which did not participate to advertise in the Outlook magazine which had various ad proposals starting from Rs. 1,00,000 and going up to Rs. 5,00,000. NUJS stated, “Sponsorships and paid advertisements in the very same issue in which you rank colleges taint the objectivity of ranking and create an impression of bias”..MDRA Response:.Re: your observation about the “influence” of advertisements, rest assured that has NO influence on the rankings or our stories..The parameter-wise score is given below for the two institutes- the first table shows what was published in one-decimal digit..Published Scores: Figures rounded off to one decimal place. The rounding off was done for the five parameters only. Then, the overall score was obtained from the sum of the parameter-wise scores so that a layman reader does not get confused..The Decimal Difference.As we can see that the sub-parameters scores when rounded off to one decimal place, yield NLIU’s total of 779.4 as well as 779.4 for NUJS, however, the actual total when sub-parameters scores are rounded off to three decimal places give a sum of 779.514 for NLIU and 779.374 for NUJS..6. Explaining the sudden drop in rankings.NUJS: NUJS was also concerned about the sudden drop in its rankings and parameters that may have influenced this sudden fall. NUJS was ranked second in 2009 and ranked fifth in 2010. It had questioned the sudden drop and the reasons for its fall from second to fifth position..MDRA Response: It is well understood that there has been a drop in ranking of NUJS over the past year, however, the quantum of change is completely independent of any past rankings and past record. The rankings are conducted based on – i) the data made available by various colleges subject to scrutiny and governed by the research methodology and ii) Perceptual survey among recruiters, practising professionals, final year students and experienced faculty members. As per the privacy condition mentioned in the objective data questionnaire, MDRA would be unable to publicise data of any college..7. Risk of Inflated Data.NUJS: NUJS had questioned whether the data sent by the law schools was independently verified by the MDRA. NUJS believed that some law schools may have inflated their achievements.MDRA Response: As per research methodology, MDRA has taken adequate steps to counter such errors/flaws/inaccuracies. Appropriate evidence for verification of claims were necessary for inclusion in data such as offer letters, proof of establishment, etc. The data was carefully examined and colleges were contacted for verification/ audit of suspicious data..8. Overall methodological flaws and lack of transparency in the survey:.NUJS: Outlook has used three different agencies to rank different colleges over the past few years (SYNOVATE in 2008, GfK-MODE in 2009 and MDRA in 2010). The criterion adopted in the different years appears to be different, without any accompanying explanations for the changed methodology or why a particular methodology/heads were preferred over others in the first place..MDRA Response: MDRA cannot comment on the studies undertaken in 2008 and 2009 by other agencies..In the Perceptual Study, students, faculty and recruiters / practising professionals were considered. Based on MDRA’s experience, the responses of aspirants was not included as their opinion is again based on just perception based on various rankings and discussions with friends/ others. Instead, an opinion based on pure experience would be better for perceptual survey. Therefore the key categories – recruiters, experienced faculty and final year students – were taken for perceptual survey..The final score is a culmination of perceptual as well as objective score..There are two key reasons for not publishing the list of experts:.Experts were asked to give the weightages and complete privacy of their confidentiality needs to be respected by MDRA.Given the available publication space and the number of streams that were ranked, it would not be possible to publish the details of the experts in the magazine..Are these responses good enough? Do students really rely on these rankings before deciding on joining a law school? What weightage would you give for Outlook / India today rankings before you decide on joining a law school?