I have mostly the same problem, but only with one person, and it’s exacerbated by my switching back to the iPhone on weekends. Even with iMessage off for a couple of months, I still get prompted to switch it back on, which is ridiculous — it should be possible to, say, have iMessage on for an e-mail address and off for a phone number in a consistent fashion.