Interpretable machine learning has been a popular topic of study in the era of machine learning. But are we making progress? Are we heading in the right direction? In this talk, I start with a skeptically-minded journey of this field on our past-selves, before moving on to recent developments of more user-focused methods. The talk will finish with where we are heading, and a number of open questions that we should think about.