SHP2 is a ubiquitous protein tyrosine phosphatase, whose activity is regulated by phosphotyrosine (pY)-containing peptides generated in response to extracellular stimuli. Its crystal structure reveals a closed, auto-inhibited conformation in which the N-terminal Src homology 2 (N-SH2) domain occludes the catalytic site of the phosphatase (PTP) domain. High-affinity mono-phosphorylated peptides promote catalytic activity by binding to N-SH2 and disrupting the interaction with the PTP. The mechanism behind this process is not entirely clear, especially because N-SH2 is incapable of accommodating complete peptide binding when SHP2 is in the auto-inhibited state. Here, we show that pY performs an essential role in this process; in addition to its contribution to overall peptide-binding energy, pY-recognition leads to enhanced dynamics of the N-SH2 EF and BG loops via an allosteric communication network, which destabilizes the N-SH2–PTP interaction surface and simultaneously generates a fully accessible binding pocket for the C-terminal half of the phosphopeptide. Subsequently, full binding of the phosphopeptide is associated with the stabilization of activated SHP2. We demonstrate that this allosteric network exists only in N-SH2, which is directly involved in the regulation of SHP2 activity, while the C-terminal SH2 domain (C-SH2) functions primarily to recruit high-affinity bidentate phosphopeptides.