Terri,
Don't be discouraged about the passport of your ancestor. The date that is online, 1875, is not a limit for your search in finding your ancestor passport.
You can find record books of those requesting a passport and, maybe passports, back to 1840 in Azores. I researched at BPARAH, in Terceira, and had on my hands books with records going back to the 1840's decade. The oldest passport that I have dates from 1861 and I got it there and it is from one of my Azores ancestors, my great great grandmother that left Terceira island to go to Rio de Janeiro, Brazil. After having it on my hands and analyzing all the information (they have several pages), I was able to do my research in Brazil and learn more about her. Usually the passport contains the name of the city that they are going to, and sometimes the ticket they bought with the ship name.
At that time, several important ports were used in Brazil - Recife, Salvador, Rio de Janeiro and Santos - where hundred thousands of Portuguese people arrived. It will be much easier if you try to contact the Archives/Library in São Miguel and request a search on his passport so you will know which one was the port of arrival, in Brazil, that your ancestors went.
Isabella Baltar