Worried, no. Concerned, yes. I’d schedule an appointment with my doctor. Of course if it concerns you that much, next time it happens go to the hospital.
It could be serious, then again it could just be stress. It also depends on the pain in the chest. If the pain is a “lightning pain” that “streaks” across the chest, it’s most likely stress. However if it radiates down your left arm or up the left side of your neck or jaw, see a doctor pronto.