Hey Matt. This topic does get airplay from time to time. I found a string on this forum from 2013 talking about the same. Suffice it to say that one cannot enter the Potomac River to swim or wade in the parks of the National Capital Area. See the link below.
So of our members have been fined, and/or warned, by law enforcement for entering the waters at Gravelly Point. I do not think that you have the same issue (kayak, canoe) if in a water craft with a flotation device fishing from the same point.
Gregg