This paper investigated the rheological properties of asphalt binder, asphalt mastic and asphalt mortar and the interaction between asphalt binder, mineral filler and fine aggregates. Asphalt binder, mastic and mortar can be regarded as the binding phase at different scales in asphalt concrete. Asphalt mastic is a blend of asphalt binder and mineral filler smaller than 0.075 mm while asphalt mortar consists of asphalt binder, mineral filler and fine aggregate smaller than 2.36 mm. The material compositions of mastic and mortar were determined from the commonly used asphalt mixtures. Dynamic shear rheometer was used to conduct rheological analysis on asphalt binder, mastic and mortar. The obtained test data on complex modulus and phase angle were used for the construction of rheological master curves and the investigation of asphalt-filler/aggregate interaction. Test results indicated a modulus increase of three- to five-fold with the addition of filler and a further increase of one to two orders of magnitude with cumulative addition of fine aggregates into asphalt binder. Fine aggregates resulted in a phase change for mortar at high temperatures and low frequencies. The filler had stronger physical interaction than fine aggregate with an interaction parameter of 1.8⁻2.8 and 1.15⁻1.35 respectively. Specific area could enhance asphalt-filler interaction. The mastic and mortar modulus can be well predicted based on asphalt binder modulus by using particle filling effect. Asphalt mortar had a significant modulus reinforcement and phase change and thus could be the closest subscale in terms of performance to that of asphalt mixtures. It could be a vital scale that bridges the gap between asphalt binder and asphalt mixtures in multiscale performance analysis.