Herbs take a long time to grow, a lot of room in your pouch, and they're pretty useless. The only one I use regularly is Hairball Flower, and even that only adds a minuscule amount to your speed for a relatively short time.
The simplest fix to these would be allowing the buffs to stack - so instead of one Yellow Crumb flower increasing your learning speed for a couple of minutes, you could eat a whole bunch and have the effect last for a few hours. Alternatively or in addition, the time each herb lasts should be drastically increased. Also, the stack size you can carry needs to be increased - it should be 80, just like other plants.