A 3.8-magnitude earthquake centered near the Maine coast rattled houses in northern New England on Monday and was felt by surprised residents of states hundreds of miles away.