Well I just figured it out. Steps:
1- Created a new lib folder inside my administrator/components/com_xxxxxx folder
2- In command line cd to that new folder then ran composer require xxxxx/xxxxxxxxx
3- Required the new dependency in file administrator/components/com_xxxxxx/services/provider.php like this:
* @license GNU General Public License version 3, or later
*/
defined('_JEXEC') or die;
require_once JPATH_ADMINISTRATOR . '/components/com_xxxxxx/lib/vendor/autoload.php';
use Joomla\CMS\Component\Router\RouterFactoryInterface;
use Joomla\CMS\Dispatcher\ComponentDispatcherFactoryInterface;
use Joomla\CMS\Extension\ComponentInterface;
...
That's it, I can now use any package installed in administrator/components/com_xxxxxx/lib