Should I have stopped after this incident?

No need what for? No harm no foul as often said. As you said no actual harm came to you or to that guy. So why stopped on the same given circumstances. I just hope you drive carefully after this and brought you a lesson from here on.