Why? Admittedly I do not know what kind of challenge-answer is involved but I woukd imagine that the whole point is identifying you.

Now, there are (currently) three ways of identification, based on something that you...

• ...are. This is the strongest method and can be based on various biometric parameters (iris scan, voice signature, fingerprint...). Difficult to do with gizmos such as the one mentioned by the OP.
• ...have. The centuries old lock-and-key belongs here. As do various gizmos provided by the banks. The gizmo may well be just to identify that you have the right card, in which case the card is the key, not the gizmo. I do not see replacing the gizmo with a phone any less secure than replacing the gizmo with another gizmo from the same bank.
• ...know. The most common and also weakest method involving various kinds of password. Both my banks use this method, trying to reinforce it by having three different passwords and using three different input methods (to fool key loggers etc).