I've seen the House of the Seven Gables -- it's across the street and down a bit from the wonderful Customs House on the Salem harbour, which I did tour. I should have gone into the gabled house too, as well as Hawthorne's childhood home, but I ran out of time.
I think that stone and brick are more inland building materials than they are on the east coast. From Massachusetts to Nova Scotia, the old harbour houses I've seen are usually frame buildings (often painted pretty pastels), and they mostly resemble one another. They're usually more vertical than the Cape Cod, which is wider and has more of a cottage-y effect, I guess, although it looks very different made of stone, as in Lower Canada.