This confused me too. It is tricky because sometimes it's more performant to copy the data rather than use a pointer, and there's not a clear boundary as to when that is the case. The advice I was given was "profile your code and make your decision data-driven". That didn't make me happy.
Now I always use pointers consistently for the readability.