Most of the repositories that comprise the Open edX project are licensed under the
AGPL license, while a few are licensed under the
Apache 2.0 license. I am not a lawyer, so I am not qualified to give legal advice. With that said, the short answer to your question is, you may use non-open software in conjunction with the Open edX software, as long as you meet the following requirements:
* The non-open software only interacts with the Open edX software through a clean API, such that it's possible to swap out Open edX with some other piece of software that provides the same API. This strict separation proves that your software is not part of Open edX, and therefore is not required to use the AGPL license.
* You continue to abide by the terms of the AGPL as it applies to the Open edX software. For example, you will need to run the Open edX codebase in order to use the APIs that it exposes from your non-open software. If you make any changes to your local copy of the Open edX software, you still must release those changes under the AGPL as well, and make them freely available to anyone who uses your software.
I should point out that this only applies to the repositories that are AGPL-licensed. The Apache 2.0 license is a more permissive license, which does not require you to make your modifications open source. For example, the
XBlock repository is licensed under the Apache 2.0 license, so if you want to fork that repository, make changes, and keep your changes closed-source, you are free to do so. It also means you are allowed to make custom XBlocks that are closed-source.
Does that answer your question? If you need further clarification, ask away! I believe that there are also several companies that are already doing things similar to what you want to do -- perhaps someone from one of those companies can reply, as well.
David Baumgold
Developer Advocate, edX