NEVER build on clay if you don't absolutely have to. It has high erosion potential, is very susceptible to freeze/thaw, and has a suuuper low porosity, hence why it's used to cap things or line things. (ponds, sanitary facilities, etc...)
Silt is a very light and small in size, so it is also not an ideal base for building. Usually, silt is the culprit of sedimentation in streams and what not, hence the word siltation. It has better drainage than clay, but not as much as sand (soil triangle). It also holds more nutrients than sand, but less than clay due to the particle size.
This is a super simple explanation, but PLEASE don't quote me on it. I'd definitely cross reference your materials Cheers!