What exactly fails? Is there any difference between pressing play and waiting till the end, versus trying to randomly seek to different parts of the video?
And it's be useful to consider if the OS, Browser, video plugin (if any) which consumes the stream has any influence on the outcome. Does the behaviour change if the browser is not on the same machine as the server? With a different player other than VideoJS? Can you run curl or wget on the video URI without any problems?
If you invoke your browser's debug mode can you observe the network behaviour when things go wrong?
Also, can you find through experiment the exact length (in seconds, and/or in bytes) where it goes from ok to not ok? Which leads to a final variable, what is/are the bit-rates of the video files? Do different bit rate files change the outcomes?
I'm not saying with these answers that I (or anyone) will be able to provide a solution, but just performing the experiments yourself might lead to some eureka moment, or, pinpoint much more precisely a genuine bug somewhere.