The dominant model of number processing suggests the existence of a Number Form Area (NFA) in the inferior temporal gyrus (ITG) that supports the processing of Arabic digits as visual symbols of number. However, studies have produced inconsistent evidence for the presence and laterality of digit-specific ITG activity. Furthermore, whether any such activity relates to mathematical competence is unknown. This study investigated these two issues using functional magnetic resonance imaging. Thirty-two adults performed digit and letter detection tasks and reading and mathematics tests. During digit detection, participants determined whether digits were present in a string of letters (e.g., AH3NR versus AHTNR). During letter detection, participants determined whether letters were present in a string of digits (e.g., 93R78 versus 93478). Results showed four clusters in frontal, occipital, and temporal regions for digit detection, including a left ITG cluster. Five clusters in frontal, parietal, occipital, and temporal regions were associated with letter detection, including a left ITG cluster. Digit and letter-related ITG clusters were spatially distinct; however, a direct contrast of digit and letter processing did not reveal greater activity in the left ITG for digit detection. Whole brain correlations showed greater digit-related activity in the right ITG for participants with higher calculation skills, but there was no correlation between letter activity and calculation skills. Together, our results suggest functional localization, but not specialization, for digits in the left ITG and provide the first evidence of a relationship between calculation skills and digit processing in the right ITG.