Nitric oxide (NO) contributes to carbohydrate metabolism and decreased NO bioavailability is involved in the development of type 2 diabetes mellitus (T2DM). NO donors may improve insulin signaling and glucose homeostasis in T2DM and insulin resistance (IR), suggesting the potential clinical importance of NO-based interventions. In this review, site-specific roles of the NO synthase (NOS)-NO pathway in carbohydrate metabolism are discussed. In addition, the metabolic effects of physiological low levels of NO produced by constitutive NOS (cNOS) versus pathological high levels of NO produced by inducible NOS (iNOS) in pancreatic β-cells, adipocytes, hepatocytes, and skeletal muscle cells are summarized. A better understanding of the NOS-NO system in the regulation of glucose homeostasis can hopefully facilitate the development of new treatments for T2DM.