I think the animation for hoeing makes it seem like it doesn't make a difference.
I have only used it for herb plots, though the card said it works for all gardening plots.
The sound on crop gardens seems to be shortened as described -- but the animation (and delay) seems unchanged. So, no actual speed-up yet there. But nice on herb gardens!
Hmm, I suspect you're right guts. It's possibly tied to the croppery and/or herbalism badges. Perhaps the 151 crop and the equivalent herb badge is required?
My guess is that it's tied to the number of herb plots tended. (I have learned all skills and tended many times many crops, but the cards haven't shown up for me.)