Sensitive subject with boyfriend, please read?

My boyfriend used to be a whore and do all sorts of drugs before we got together, to put it lightly. We've been together for two years in about two weeks. Two years ago to this day, we met up at the fireworks that they have every year. We were "dating" at the time but not officially boyfriend and girlfriend. He held my hand and was kissing me. Afterwards, he met up with a girl that's liked him since grade 5 and they did stuff on many occasions. I didn't know that happened on the same night till after we started dating, and I felt like sh*t and used. I asked him to the fireworks tonight and he didn't wanna go. I told him he went two years and no one made him. He obviously doesn't remember because he was wasted, and laughed at me. What do you think of this? I know it was two years ago and just thinking of this stuff this time of year makes me sad. I love him so much now, but I still know he's not "the one". Thoughts?