In order to not worry about ssh-agent integration, it will take care and ask for passphrases on first zsh start.
Daniel Siepmann