issues with accessing a resource when in reduce phase

[Andy Xue]

Hi -- I am seeing errors when jobs go into the reduce phase and it can't find a resource -- stack trace looks like this:

Caused by: java\.lang\.IllegalArgumentException: URI is not hierarchical
	at java\.io\.File\.<init>(File\.java:363)
	at sun\.reflect\.NativeConstructorAccessorImpl\.newInstance0(Native Method)
	at sun\.reflect\.NativeConstructorAccessorImpl\.newInstance(NativeConstructorAccessorImpl\.java:39)
	at sun\.reflect\.DelegatingConstructorAccessorImpl\.newInstance(DelegatingConstructorAccessorImpl\.java:27)
	at java\.lang\.reflect\.Constructor\.newInstance(Constructor\.java:513)
	at clojure\.lang\.Reflector\.invokeConstructor(Reflector\.java:183)
	at schema_reader$read_json\.invoke(schema_reader\.clj:12)
	at schema_reader$load_schema\.invoke(schema_reader\.clj:15)
	at schema_reader$get_fields\.invoke(schema_reader\.clj:20)
	at sql_dump_tap$get_sql_dump_tap\.invoke(sql_dump_tap\.clj:35)
	at sql_dump_tap$get_taps$fn__1968\.invoke(sql_dump_tap\.clj:52)
	at clojure\.core$map$fn__3811\.invoke(core\.clj:2430)
	at clojure\.lang\.LazySeq\.sval(LazySeq\.java:42)
	at clojure\.lang\.LazySeq\.seq(LazySeq\.java:60)
	at clojure\.lang\.RT\.seq(RT\.java:466)
	at clojure\.core$seq\.invoke(core\.clj:133)
	at clojure\.core$into_array\.invoke(core\.clj:3040)
	at sql_dump_tap$multitap\.invoke(sql_dump_tap\.clj:60)
	at sql_dump_tap$get_multi_tap\.invoke(sql_dump_tap\.clj:62)
	at sql_dump_tap$get_query_distinct\.invoke(sql_dump_tap\.clj:70)
	at views\.silverpop\.silverpopeventfacts$mk_user_id_lookup_query\.invoke(silverpopeventfacts\.clj:44)
	at views\.silverpop\.silverpopeventfacts$mk_silverpop_event_facts_q\.invoke(silverpopeventfacts\.clj:50)
	at views\.silverpop\.silverpopeventfacts__init\.load(Unknown Source)
	at views\.silverpop\.silverpopeventfacts__init\.<clinit>(Unknown Source)
	\.\.\. 60 more

so the issue is that when the reduce task starts, it has to init my class (in this case silverpopeventfacts). it fails when trying to load schema.json which is a resource i have put in my /resources folder.

it seems odd that this is failing only on the reduce tasks -- the maps tasks have no issue loading this resource on init. 

part of this is sloppiness on my part -- i bind a (<- ) to a def so it forces a query to be built (and it tries to init a tap using schema.json). i simply removed the bindings and the job works -- however i am curious why only fails on reduce tasks. any ideas? thanks!

[Andy Xue]

more details:

i call (clojure.java.io/resource "schema.json") to get my resource

again, this functionality works in all other cases except on the class init during the reduce task