Some Georgia clay got into my new fish pond during construction. Do I need to empty it and refill?

There is about a 1/4 inch of clay in the bottom and on the marginal shelves of my 300 gallon pond. I have not added fish or plants yet. The water is already murky but has only been in the pond for 3 days. Should I empty and refill before proceeding with living things or just let it sit for a few more days first?