Construction dust all over my garden!

The builder next door filled in a large trench adjacent to the property line by blowing in aggregate. Now there is a grey-ish dust all over my garden! I am worried that if it rains, it will turn into concrete.…