Residents in a Derbyshire village are up in arms over the planned conversion of an agricultural building into a farm shop, bistro and cafe.