If the paths to jQuery and facebox.js are OK then it may be how you're
calling it.
<script type="text/javascript">
jQuery(document).ready(function($) {
$('a[rel*=facebox]').facebox()
})
</script>
in the head should work.
If any other javascripts are included in the main page or login.php
try disabling them to test for conflicts.
If the page is online, provide an URL and I'll take a look.