The safety and efficacy of drug substances or products do not solely depend on its active(s). The quantity of impurities present in the product has a significant role in its safety profile. Pharmaceutical impurities are one of the primary reasons for the withdrawal of many approved products from the market. Therefore, the level of impurities in the pharmaceuticals needs to be controlled within a specified safe limit. Nowadays, setting impurity level specification remains a great challenge for pharmaceutical manufacturers. Regulatory guidelines recommend to control the impurity based on the concentration level criteria and provides limits of allowable impurities in pharmaceuticals. However, a single set of impurity limits cannot work for all the drug substances. There are numerous reasons which demand to set the impurity level specification based on safety dominated critical quality attribute principle. In this review, we have discussed the need for the consideration of both concentration based and patient safety-related approaches for setting the impurity level specification. To achieve this goal, it is required to identify the safety limits of the impurities during clinical development and provide a specification for the finished pharmaceutical products before entering the market. However, tremendous challenges faced by pharmaceutical companies to have an appropriate balance amongst the critical factors like safety, efficacy, analytical variability, process knowledge and regulatory requirement. Finally, the specification for API and finished drug product should be established considering both quality and patient safety. Considering all such factors, we have included a systematic and scientific approach that can guide to establish the safe and flexible impurity limit specification for pharmaceuticals.