I would rinse, or shake or somehow clean up the brush in between plants. That would cause cross pollination.
Just FYI, for most purposes, hand pollination of tomatoes is rarely needed even in the most sterile of situations (greenhouse applications, etc). Tomatoes not setting fruit is much more...