The Material Girl hitmaker purchased the eight-bedroom mansion and its 24 acres of surrounding land in Bridgehampton for $2.2 million (£1.4 million) on the promise the fields could not be built on and would be used for growing agricultural products.
Pictures obtained by the New York Post show Madonna has now begun creating a tree nursery on the property after planning permission was approved last year.
However, the move has angered locals who are convinced she is using the trees to protect her privacy and to receive tax breaks entitled to farmers in the area.
East Hampton farmer John White tells the publication, 'That's not a nursery... She just doesn't want people to see her riding (a horse) - and falling.'
Madonna cracked three ribs and broke her collarbone after falling from a horse at her country estate in England in 2005.