Perhaps at one of the higher-level croppery or herbalism skill levels, it could be possible to add a verb "dig up" to allow for those times when you planted something by accident and don't feel like spending the guano to speed up the growth process. Such an action would almost surely destroy the thing you're digging up, and it could be restricted to garden plots in your own home or one for which you have a key, or in a "locked plot" which is held for you in public gardens (per today's change).