When is the most polite time to refuse someone?

Every day after work, I drive home and ask my colleague if he wants me to give him a ride first,
but he always says no. I secretly feel pleased inside—it's good he doesn't want a ride, or else I’d have to take a detour and deal with traffic jams.
A few minutes later, he takes a taxi home himself. I was puzzled because he's usually a very approachable person, but why does he always refuse me?
I asked him, and he said something I will never forget in my life.
He said: Anyone who asks me if I want something, I always say no, because someone who truly wants to give you something wouldn’t ask if you want it.
When he asks if you want something, it’s best to refuse.