ListViews are virtualized, they dispose of and rebuild their unviewable items to recycle memory. Consider caching the video playback position in a Map<string, int> to look up the last playback position of the video when the widget rebuilds. You can pass in the position from the builder callback. Keeping the videos around would be like having a memory leak as the user scrolls past 10s or 100s of videos. Your videos are likely to have UUIDs assigned at upload time, use that for the string of the hash map. The int is obviously going to be the time in seconds.
You can get the position from the VideoPlayerController.
https://pub.dev/documentation/video_player/latest/video_player/VideoPlayerController/position.html