Yes!
You can use directives as if you where using any other way of loading your scripts. Though make sure you have loaded them all before you bootstrap the application.
I don't fully understand your design. But the basics are that you don't need the router if you are not doing a one-page-app. But basically you could have many pages with there own mini router to only manage a small set of views.
If i where you i would ask myself:
* How are my app going to be used.
* What type of server are im going to write with express (great choice btw).
* What things can i do without.
It seams to me that you have started with way to many unfamiliar technologies at one same time.
It might help you to configure express to only serve static content and build a prototype with only static pages and the angular router. It will be enough to get a feel for angular and what you want to build. Then you can pick a path from there. To load new pages or json form express.
When having such a amazing back-end it sounds to me a little loony to not send data as json and serve static html (angular templates). The only strong argument against it that I see is that its harder to make a page like that index correctly for a search engine crawler.
And if you can remove anything from your stack in a first attempt, do so!