Can driving with no muffler ruin my car?

my muffler was rusted trough and falling off so i removed it ! just wondering what will happen if i dont replace it soon or if it even matters if i drive without one!!