"Whoa! You're learning super fast. Keep in mind, if you switch skills you'll lose your super speed."
This is the message I see on the Skills page after spending favor at a shrine to learn a skill faster. However, since the time saved appears to be applied instantaneously, I'm not sure if this message is true.
Switching between learning skills appeared to get around the "You're already learning as fast as you can!" limitation with no penalties.
* Start learning Skill #1
* Spend Favor at Shrine of Primary Giant
* Pause learning Skill #1
* Start learning Skill #2
* Pause learning Skill #2
* REPEAT