The Potomac shore below Horsehead Cliffs, where the cliffs shed Miocene sea floor onto the sand. Virginia State Parks permits fossil collecting along the day-use beach and at Fossil Beach, a trail walk from the visitor centre, and shark teeth are what most people carry home; the visitor centre lends sieves. A fifteen-million-year-old whale skull came out of these cliffs. Keep to the beach: the ground below the cliffs is closed, the cliffs drop without warning, and digging into them needs a park permit.
Shark teeth, whale bone · Miocene, ~15 mya
Open to the public for collecting, and what you find is yours to keep. Check the posted rules before you start: they can change with the season and the landowner.
Other places on the map within a couple of hours.