We should be worried.
Have you never heard of the hydrogen bomb? The USSR had a tested H-bomb more than a year before we did, thanks to a certain genius named Sakarov. They also had rockets capable of carrying them to the US - one of which launched Yuri Gagarin into space.