The Age of Humanity lasted thousands of years. To the short lived races it lasted many ages, but to the long-lived races it was but one.
Near the beginning of the Age of Humanity, Garyx once held the portfolio of Fire. The primitive humans worshiped the elements and raw power of nature with much more fervor than later in history. In those early days Garyx was powerful and had grand temples.
As humanity relied less on the raw cleansing power of fire, their worship of him waned. His time in Pandemonium (timeline???) also saw his decline.
Eventually Sindri Battleforge stole Garyx's flame, leaving him with control only over red dragons but their fire, no more.