SHP2 is a tyrosine phosphatase that plays a regulatory role in multiple intracellular signaling cascades and is known to be oncogenic in certain contexts. In the absence of effectors, SHP2 adopts an autoinhibited conformation with its N-SH2 domain blocking the active site. Given the key role of N-SH2 in regulating SHP2, this domain has been extensively studied, often by X-ray crystallography. Using a combination of structural analyses and molecular dynamics (MD) simulations we show that the crystallographic environment can significantly influence the structure of the isolated N-SH2 domain, resulting in misleading interpretations. As an orthogonal method to X-ray crystallography, we use a combination of NMR spectroscopy and MD simulations to accurately determine the conformation of apo N-SH2 in solution. In contrast to earlier reports based on crystallographic data, our results indicate that apo N-SH2 in solution primarily adopts a conformation with a fully zipped central β-sheet, and that partial unzipping of this β-sheet is promoted by binding of either phosphopeptides or even phosphate/sulfate ions.