Course Recommendation Engine
A machine-learning system that recommends online courses from what learners like you went on to take, served through a Flask API.
- Platform
- Research project
- When
- 2023
- Role
- Developer
- Status
- Research
The problem
Online learning platforms recommend what’s trending, or what they want to sell, not what’s relevant to you. The question was whether collaborative filtering on real interaction data could do better.
What I built
The model finds learners whose history looks like yours, then recommends courses they completed that you haven’t seen, with Scikit-Learn working out the similarity. A Flask REST API takes a user and returns a ranked list of courses with confidence scores, and PostgreSQL holds the users, the courses and every interaction.
Pushing to main ran the tests, and when they passed, deployed it to Heroku.
Hard partThe cold start: a new learner has no history for collaborative filtering to work with.
Decisions
- 01
Collaborative filtering, not matching course descriptions
It finds recommendations from what similar learners did, which a description can’t tell you.
Trade-offA new learner has no history, so anyone with fewer than five completed courses gets suggestions from course content instead.
- 02
Similarity worked out ahead of time
Comparing every learner with every other on each request grows with the square of the users. A nightly batch does it once.
Trade-offRecommendations can be up to a day old, which is fine for courses.
A bug I remember
Recommendations for someone else
- What it did
- Now and then, the precomputed similarities returned recommendations meant for a different learner.
- Why
- The matrix was indexed by DataFrame row position, and a sort further up reordered the rows, so user 42 was no longer row 42.
- The fix
- An explicit map from user id to row, built before the sort. Row order is an implementation detail, and code across functions mustn’t depend on it.
What I tried first
Neural collaborative filtering
The dataset was too small for it to beat classical matrix factorization. The simpler model did better at this scale.
What I learned
- There’s no free answer to the cold start: ask new learners questions up front, or accept weaker early recommendations.
- When a model’s output drives an interface, test the API’s contract. A change in the response’s shape broke the front end silently, until end-to-end tests caught it.
- Free Heroku dynos sleep after 30 minutes idle, and jobs scheduled while they sleep fail silently.
Built with
PythonScikit-LearnFlaskPostgreSQLRESTHeroku