Un-opened Rose of Sharon pods

I live in Toronto. My rose of sharon get 3-4 hours of afternoon sun June, July, August. Many pods do not open. 1. Do I remove them now (late November) 2.Remove them next spring? 3. Just leave them?…