If the noise goes away with a small quick hit of WD-40 or similar when idling, then yeah it's a belt issue or an alignment issue with the belt. If it doesn't change, then it's something else like a tensioner/idler pulley/water pump bearing.