Program Impact

iMentor Program Evaluation

iMentor uses mentoring to bridge the divide between New York City’s diverse communities so that mentees are better equipped with the skills and resources critical for (1) personal, (2) academic, and (3) career success. Through participating in iMentor, mentees become more hopeful about the possibilities available to them in the future and are more aware of the powers of social change.
iMentor expects: - Each mentor-mentee pair to develop a strong relationship, upholding benchmarks for program participation which include an 85% weekly email frequency and at least 1 in-person meeting every month between October and June of each program year. - These relationships to produce significant and measurable improvements in mentees' personal, academic, and career growth.

Quick Facts

92% of mentees graduate from High School

82% of mentees enroll in College

93% of mentors would recommend volunteering with iMentor to a friend or co-worker

87% of mentees think that getting to know each other through iMentor has been a positive experience

See iMentor's 2008-2009 Evaluation Highlights

iMentor's Independent Evaluation

The iMentor program staff have been conducting internal and external evaluations since their inception and have developed a number of strong evaluation tools. In an effort to update and strengthen these instruments, iMentor contracted with Kim Sabo Consulting (KSC) who proposed a highly participatory process that engaged iMentor staff members in clarifying and sharpening key outcomes, indicators, targets, and data collection strategies.

The iMentor evaluation took place in 2008, with four specific goals in mind:

- Articulating an iMentor Logic Model

- Strengthening iMentor evaluation methods and strategies

- Analyzing all pre/post 2007-2008 iMentor surveys

- Analyzing 2007-2008 Department of Education data

(Download evaluation results as a PDF)

The Methodology   |   Quality of iMentor Program   |   Quality of Mentor-Mentee Pairs   |   Mentee Outcomes   |   Mentee School Grades, Test Scores and Attendance   |   Mentor Outcomes   |   Conclusion and Next Steps

The Methodology

KSC analyzed all iMentor pre/post outcome surveys from 2007-2008. In total 425 survey respondents from 14 iMentor sites in 2007-2008 completed the pre- and post-test surveys. Pre/post data were analyzed using t-tests to measure the statistical difference of change over time. Post-only data were analyzed by calculating basic frequency of response per question.

In order to explore the impacts of iMentor on academic achievement, 2007-2008 Department of Education (DOE) data were also analyzed using parametric tests such as ANOVA and t-tests. All comparison groups were formed through block randomization to match iMentor participants' gender and grade ratios. iMentor participants' were compared to those of similar students in terms of age, gender, ethnicity, and those who attended the same or a similar school (i.e. geographic isolation, school size, school performance, percent of students who receive free lunch, percent of students who are in minority groups and percent of students who are recent immigrants).

Academic data from the DOE was available for the 11 participant sites that are New York City public schools and included a total of 568 students who had participated in iMentor for at least one full year.

Quality of iMentor Program

The demographic data shows that iMentor is serving its target populations. Most of the mentees are African American and Hispanic, and almost all are eligible for free lunch. Most live in Brooklyn and the Bronx and are enrolled in schools that are geographically isolated. In addition, because of the process that iMentor uses to recruit partner sites, inputs such as school geography and capacity of partner sites are ensured.

Almost all mentees (87%) and mentors (97%) would recommend iMentor to a friend.

Almost one-half of mentors (48%) felt that they learned more about their mentee through email discussion and just over one-half (52%) said that they learned more about them from in-person meetings. More than one-half of the mentees (57%) said they learned more about their mentors through email discussion, and less than one-half (43%) said they learned more about their mentors during in-person meetings. The data illustrates that to ensure all mentor-mentee relationships fully develop, iMentor’s program model combining both email and in-person meetings is essential.

About one-third of all mentees reported that their mentor was most helpful in preparing them for career success, one-third stated that they were most helpful in building personal relationships, and one-third said that they were most helpful in working toward academic success. It is interesting that these responses are evenly distributed, with no single core impact area dominating the relationship, indicating that iMentor’s programming provides broad support for students’ diverse developmental needs in these three distinct but interrelated areas of personal, academic, and career success. The combination allows for multifaceted mentee growth that increases mentors’ impact.

Many mentors also rated the iMentor staff above average or excellent in terms of: providing them with strategies to improve their relationship with their mentee (69%), helping them with technology issues (64%), and giving them additional information about their mentee’s home/family life and school/academic situation.

Quality of Mentor-Mentee Pairs

Almost all mentors (95%) and mentees (95%) agreed that getting to know each other was a positive experience. Additionally, almost all mentors and mentees agreed that mentees were made to feel important by mentors and that mentors helped to challenge mentees to succeed. Mentors and mentees also felt safe working together and felt their match with one another was a good fit.

Almost all mentees agreed that their relationship with their mentors had a positive impact on their lives. Most stated that their mentors gave them advice and answered their questions (95%), and that their mentor was someone who could be trusted and depended on (90%). Most also said that their mentors helped them to spend more time and put more effort into their learning (85%), made them more comfortable interacting with adults (83%), and that they would like to continue their relationship with their mentor after the end of the program (80%).

Mentee Outcomes

iMentor has several short and intermediate mentee outcomes. The 2007-08 survey data illustrate how iMentor facilitates personal, academic and career pathways for mentees across sites. The findings indicate approximately three-quarters of all mentors and mentees agreed that iMentor had a positive impact on mentees’ attitudes toward school. More than half also agreed that mentees now have better grades and test scores, come to school better prepared, and have better classroom behavior.

Several questions on the post-program survey address concrete ways that mentors and mentees work together to identify goals and take steps towards career success. Prior to iMentor, fewer than half (40%) of the mentees had written professional resumes. By the end of the year in iMentor More than half (59%) had written professional resumes, and this difference was statistically significant at p<.001.

Almost all mentees (85%) agreed that their participation in iMentor had made them more enthusiastic about their future after high school. Almost all mentors and mentees (90%) agreed that iMentor had a positive impact on mentees’ views of his/her future, thoughts about the importance of helping others, expectations about his/her future, comfort interacting with adults, and thinking that he/she is a better person.

Mentee School Grades, Test Scores and Attendance

iMentor participants who attended the program for at least one year, scored higher on English Regents Exams, attended school more regularly than their peers in the same or similar schools and significantly increased English and Math grades. Below are the key findings from analysis of the DOE data.

English Regents Exam

iMentor participants’ scored higher than the comparison group students on the English Regents Exam; iMentor students had an average score of 70.47% compared to 63.98% for the comparison group. It is important to note, a passing score on this exam is 65, illustrating that the average iMentor student passed his or her English Regents exam, while their peers in the comparison group did not.

Attendance

The data illustrate that both male and female iMentor students are attending school more regularly than comparison group males and females. While there was a decline in attendance for both iMentor participants (93.63% to 92.6%) and comparison students (88.46% to 83.85%), there was significantly less of a decline for iMentor participants (t=9.94, p<.001). Additionally, iMentor participants who had low attendance in the 2007 school year (75% or less before enrolling in iMentor) increased their mean attendance by 7.95%, which is statistically significant. This is the only group of students (iMentor or cohort) who had attendance increases from 2007 to 2008.

Math and English Grades

Grades are calculated differently in each New York City public school, making comparisons across schools less reliable than the standardized Regents Exam scores. However, the comparison cohort was thoughtfully constructed with fifty percent of the students drawn from iMentor schools and fifty percent drawn from a very similar school. All comparisons were formed through block randomization to match iMentor participants’ gender and grade ratios. Paired sample t-tests were used to determine the significance of changes in English and Math grades after one year of iMentor participation. ANOVAs and independent sample t-tests were used to determine the significance of the changes between groups. Thus, the following data provide strong evidence that there are key differences between iMentor participants and their peers.

iMentor students’ English grades increased significantly (t=4.05, p<.001) from the beginning of the academic year to the end, from 72.04% to 77.39%. The p value illustrates that we can be 99.9% confident that this increase did not happen by chance. The comparison group’s English grades also increased slightly from the beginning of the academic year to the end; however, this change was not statistically significant. A comparison of the iMentor students’ increase with the comparison group’s increase showed no statistical difference.

iMentor students’ math grades significantly increased (t=3.90, p<.001) from the beginning of the academic year to the end of the academic year from 73.69% to 81.96%. The comparison group math grades also increased slightly from the beginning of the academic year to the end; however, this change was not statistically significant. A comparison of the iMentor student increase with the comparison group increase showed statistical difference, with a (t=1.97, p=.05). It is typical for students to show greater gains in math grades compared to English grades. While each year students will start with a new math course (geometry, calculus), English skills are cumulative over several years. Therefore, math grades may be a better indicator of academic engagement.

Mentor Outcomes

Volunteers in iMentor’s program--100% of whom have graduated from college--are largely first-time mentors: 70% have never mentored before and 52% do not participate in any other volunteer opportunity. Additionally, the flexibility and structure in iMentor’s model allows it to provide mentors to the young people living in geographically isolated and socio-economically underserved communities--the same areas to which mentors are traditionally least able or willing to travel.

Besides affecting their mentees’ lives in a positive direction, mentors report that their own lives are altered by participation in the program. Survey data also provides evidence that the iMentor program is having an impact on mentors. More than half of all mentors agree that mentoring through iMentor has increased their leadership skills (65%), that they are now more likely to advocate for social change (51%), and that events held in their mentees communities’ made them feel more connected to the larger NYC community (61%). Almost half (46%) stated that iMentor increased their job satisfaction . More than one-third stated that because of mentoring through iMentor they are now more likely to make financial contributions to organizations working for social change.

Conclusion and Next Steps

The demographic data, along with iMentor’s strategy for school selection, illustrate that iMentor is serving its target population and providing a high quality program with which both mentors and mentees are very satisfied. In addition, this evaluation provides evidence that there is progress toward all iMentor outcome areas identified within the logic model. Of particular note, iMentor participants scored significantly higher than the comparison group students on the English Regents exams, significantly increased both their English and Math grades, and attended school more regularly than comparison group males and females. In addition, the iMentor program had a significant impact on those students who had low attendance at the beginning of the year.

As part of this evaluation, iMentor staff defined outcomes for mentees, mentors, and pairs. Once these outcomes were defined, it was evident that many of the outcomes for mentees could be measured more directly and robustly, which guided the revision of surveys for 2009-2010. Revised surveys now include items that should ensure construct validity and standardization across pre-test and post-test for mentee outcomes. In addition, items have been added in order to measure outcomes that were not measured or were not sufficiently measured, such as mentor experiences of the program and pairs seeing through barriers of race and class.