The impact is determined by the energy of the projectile, not the mass of the shell. Think about the outer space trashes orbiting the earth that can cause disaster to the satellites, anything lager than just a couple centimeters diameter is considered as great threat, simply because of the high velocity, thus the high kinetic energy they may have.
Also think about the bullet of a handgun, which also relies purely on kinetic energy to hurt or kill target. If your logic stands, then the gunner should suffer the same impact as the victim, yet we don't see the gunner will be hurt as hard as the target person. There is something more than just the equivalent of impact in such a situation.