GSoC Avoid Disappeared Students

From Google Summer of Code Mentor Wiki

Use this page as a whiteboard to get some discussion of how to avoid disappeared students. Add comments here or on the talk page.

Contents 1 Overview 2 Selection process 3 Controlling 4 Objectives 5 Success control 6 Reasons for Students Disappearing 7 Predictors of success 8 Notes from session

[edit] Overview

One of the big problems of GSoC right now is having disappeared students, because

• they waste a slot for another student that could have done the work
• there reasons for disappearing are not clear
• some of them disappear after midterm having done just half of the work
• they leave you with a bad feeling

DavidHorat: Here is a list of proposed improvements to standard selection process and controlling (although some organizations may already implement this). It will be good to share and analyze how we picked the students that after GSoC keep on with the organizations and the ones that just disappear.

JamesCrook: Two issues here then? Students who disappear during GSoC and then as a secondary topic keeping students active after GSoC is over...

DavidHorat: Yes, the point is, how to pick the right ones :)

[edit] Selection process

• Create an internal questionnaire per project for students
  • Remark how important is to have time to develop a full time job as the SoC
  • Remark how important is to have willpower to develop a work from home or try to search other places to work
  • Remark how important is to do the job after accepting it and not wasting a slot that another student could have used
  • Remark how good is for the student being successful in the GSoC (visibility, good for CV, recommendation letters, money, etc.)
  • Remark how important is to collaborate and interact in the forums with the community even before choosing a project
• Test the student to see if he has got the necessary skills
  • Communication: Ability to communicate clearly and effectively with the concrete community methods
  • Bug tracker: Ability to create a tracker issues and effectively use the tracker as a means of communication
  • Coding: Ability to create and use diff files demonstrated by creating a patch for an existing issue
    • Can we work out what 'best practice' is for selection by comparing the selection strategies of very successful organisations and ones which had higher drop out rates?
    • Do 'scamming' students tend to target first time organisations or particular kinds of ideas, such as code refactoring, or is there no particular pattern?

[edit] Controlling

• Updates:
  • Use Wikis as the main projects documentation, which must be up-to-date everyday
  • Force the student to blog publicly each day/2 days about his progress and in what he has spent his time working
  • Use a Jabber chat / IRC channel to communicate all, both mentors and students from a concrete organization
  • Emails should be used as the last option if all the aboves don´t work
• Self-controlling: create a weekly questionnaire that the student should complete for himself to see if he is doing good. It should contain things like
  • I am on track with the project´s planning?
  • Have I talked with my mentor about the status of the project this week?
  • Have I posted in the forums this week?
  • Have I uploaded all the code to the CVS/SVN/Tracker each day?
    • Should we prohibit students from working in a private repository (e.g. git) with periodic less frequent (e.g. weekly) check ins?
    • Is 'presence (lurking) in IRC' any good as a measure of student activity?
• Using the tracker to complete subtasks could provide a more objective analysis of project completion
  • How do we measure the mentor input, e.g. how do we measure the time and attention paid to code reviews as opposed to the mentor who only tries running the code and only comments on that to the student?

[edit] Objectives

• Requirements must be closed by the mentors before the start of the coding phase
  • At least midterm objectives evaluation process should be very very easy to do
  • Force to have a midterm public release
  • Final objectives could be a bit more subjective when it involves graphic design, etc.
  • How do we cope with 'ambitious projects' that are likely to only really come together right near the end? Should such projects automatically be rejected at selection phase no matter how stellar the student appears?

[edit] Success control

• Should we create a list with all students that have succeeded in their GSoC?

[edit] Reasons for Students Disappearing

• The ones who disappear, how do we determine the main reason? Is the general experience that students who disappear tend not to fill in feedback form? Could we change that so that we hear more from those who leave about why? Do we have to guess what the real reason was?
• Can we identify standard disappearing act 'profiles' such as: "able student who didn't intend to complete if job-offer came through", "weak student who bit off more than they could chew", "student who just didn't have the language or communication skills"?

[edit] Predictors of success

• Is prior involvement in Open Source a predictor for 'good behaviour'? Is prior successful involvement in GSoC?
• How common is it for students to leave without any warning signs? Can we be more sensitive to the warning signs? If we can, what can we do about it that might get a better outcome?

[edit] Notes from session

Notes takes during the session (Oct 25, 15:00-16:30)

• "Disappeared" means "disappears permanently after being selected to participate in GSoC".
• Why do students disappear?
  • Student had a great proposal, but then finds a better paying job during the summer.
  • Sometimes students just disappear with no explanation whatsoever.
  • Sometimes mentors are partially to blame. Mentors have to realize that they're working with students (not minions) and that they'll have to help students (and, if they don't, the student will be more likely to fail).
• Only trust students you know in person?
  • Not necessarily. Some mentors knew students in person, thought they would do well, and then failed out of the program
• What control measures can we have in place to reduce the number of students who disappear?
  • Negative consequences if you disappear for a week or two?
    • Not necessarily: some mentors report that some students disappeared for a couple weeks, but were still able to complete their project successfully
  • Delay midterm payment? (i.e. "you get the midterm payment if you actually pass at the end")
    • Some mentors disagree. It's too much administrative overhead, and it shouldn't be about the money.
• Suggestions
  • VideoLAN: Harder selection process with a qualification task
  • VideoLAN: Require a weekly status update from student to mentor + admin (these status updates are published on a wiki).
  • VideoLAN: Monitor commits. If there is no commits in more than one week, ask for an explanation.
  • (VideoLAN had dropouts last year; they used the above measures this year and they were very positive)
  • Have a secondary contact (parent, friend, etc.) to rule out personal circumstances that prevent the student from checking in with the mentor.
  • Communicate expectations and have hard deadlines: make it very clear that failing the midterm or the final is a very real possibility if they don't meet expectations.
  • Be flexible: missed deadlines are occasionally a part of software development (if we miss a deadline, we don't just cancel an entire software project). However, do expect communication from your students (if they get stuck, they should take the initiative to contact the mentor; overcommunication is better than undercommunication; etc.)
  • KDE: Their meesage to students: "Communication is not an option. You have to communicate."
  • Mixxx: They pointed their students to this document: http://mixxx.org/wiki/doku.php/soc_student_info
  • Informal communication with a student helps (so it doesn't feel like the mentor just shows up periodically and says "Report!")
  • If students want to be considered for GSoC they have to submit a patch or pass a test.
    • Disagreement: Some projects get very few applications, and having this barrier would make that number even smaller. It's a good idea, but it's not for everyone.
    • Disagreement: Some projects require special hardware to do their project, and it won't be provided unless you're accepted to GSoC.
    • K3b: asked for a pretty substantial working code for the application, and it worked out pretty well for them
  • Students should be aware of what Google Summer of Code is: it's not just a job that only lasts during the summer, we also want to bring people into the community beyond the summer. Find a way to measure the student's willingness to stay around after SoC.
  • Mentors should be realistic about how much hours students will be able to work per week.
• What can we do as a whole or ask Google to do to minimize disappearing students?
  • Google could provide guidelines about managing expectations (like those in http://mixxx.org/wiki/doku.php/soc_student_info) to mentors and students
  • Google could help "train the mentors". One possibility is to provide a checklist or list of guidelines that mentors / mentoring organization should follow.
  • Google could provide a way to fail students at the end of the community bonding period if they haven't used the time to familiarize themselves with the their org's technology (some orgs have very complex technology that requires being familiar with it when GSoC starts)
    • Apparently this was possible this year, but required contacting LH directly. It would be nice if this was better integrated with the GSoC timeline.
    • If a student is failed at the end of the community bonding period, have the ability to bring in a waitlisted student.
    • Objection: some students can't use the community bonding period because they're still in school.
    • Counter-objection: We're not asking them to work full-time during the community bonding period, just to familiarize themselves with tech (which just requires a few hours a week; most of us do this even with full-time jobs)
  • More flexibility in failing students and bringing in "backup students"
  • Google could provide a way to put a student "on notice" (officially). This means that they would get a notification directly from Google about their status (which would carry more weight than just being nagged by their mentor).
  • Make sure these suggestions make it into the official GSoC documents!