so i have a lookup function that basically does a mapreduce job to
read small dimension data from S3 and then puts it into a hashmap. i
memoized the function so that the map is stored in memory. code looks
like this

(defn- get-referral-dimension-map* [referral-dimension-path]
  (let [rd-src (sdt/get-query referral-dimension-path ["!referral_key"
"!ref_name"])  <- this makes a query and selects fields !referral_key
and !ref_name
        tuples (??- rd-src)]
    (into {} (first tuples))))

(def get-referral-dimension-map (memoize get-referral-dimension-map*))

(defn get-referral-name [referral-key referral-dimension-path]
  (let [m (get-referral-dimension-map referral-dimension-path)] (m
referral-key)))

this gets called in a "main'"query, something like

(<- [?referral_name]
    (src ?referral_key)
    (get-referral-name ?referral_key :> ?referral_name))

oddly, the behavior I am observing is that a mapreduce job is launched
for the referral-dimension data for every map task in the "main" query
-- just seems like once one map task has called the get-referral-name
function, it be memoized and all subsequent map tasks on a node that
call that function should not need to re-do the mapreduce job.

(let [rd-map (get-referral-dimension-map* referral-dimension-path)]
(<- [?referral_name]
    (src ?referral_key)
    (rd-map ?referral_key :> ?referral_name)))

the odd thing is that in this case there is only one mapreduce job for
the referral dimension data -- but this also seems odd; my intuition
feels like there should be at as map mapreduce jobs as there are nodes
(ie, copies of the app jar)? what is actually going on under the hood?

On Apr 28, 6:12 am, Andrew Xue <and...@lumoslabs.com> wrote: