Redis uses the strict definition of "memory", meaning RAM. Many people refer to a computer's disk storage as "memory" which is very inaccurate. Disk is disk. Memory is RAM.
RDB and AOF are files on disk. Since they are not stored in RAM, they do not count against Redis's maxmemory limit.
(you could set up a ramdisk and configure Redis to save its RDB/AOF files there, and therefore be consuming RAM/memory with them, but that defeats the point of having RDB/AOF files, and Redis would still not count them against maxmemory)
The things that Redis counts against its maxmemory limit are only the keys/values it holds in its data store in RAM. Setting maxmemory can be a reasonable way to avoid your Redis RAM consumption from growing too large for the server, but it's important to read the whole section of comments in the source code's example redis.conf file to understand the other configuration parameters that give Redis the power to reduce its RAM consumption when the maxmemory limit is reached.