When you useToList you transform IQueryable in IEnumerable and so you wrongly select the user's todos in memory.
public ActionResult Index()
{
var currentUser = manager.FindById(User.Identity.GetUserId());
return View(db.ToDoes.ToList().Where(todo => todo.User.Id == currentUser.Id));
}
You should delete the above said ToList in order to optimize the Where condition into the linq/EF autogenerated SQL