Shell Script Fails when I try to populate dynamic oozie parameters

[Łukasz Włodarczyk]

Hi,

I have strange problem. I defined Oozie Parameters for workflow where one of the action is shell script:

Oozie Parameters look like:

Name: year
Value: `date -d '1 hours ago' +%Y`
Name: month
Value: `date -d '1 hours ago' +%m`
Name: day
Value: `date -d '1 hours ago' +%d`
Name: hour
Value: `date -d '1 hours ago' +%H`

My Shell action file looks like:

hdfs_distribution_hive_and_impala.sh

#!/bin/bash

YEAR=$1
MONTH=$2
DAY=$3
HOUR=$4

SUCCESS__YEAR=`hadoop fs -stat /data/logfiles.db/install_callbacks_csv/year=${YEAR}`
SUCCESS__MONTH=`hadoop fs -stat /data/logfiles.db/install_callbacks_csv/year=${YEAR}/month=${MONTH}`
SUCCESS__DAY=`hadoop fs -stat /data/logfiles.db/install_callbacks_csv/year=${YEAR}/month=${MONTH}/day=${DAY}`
SUCCESS__HOUR=`hadoop fs -stat /data/logfiles.db/install_callbacks_csv/year=${YEAR}/month=${MONTH}/day=${DAY}/hour=${HOUR}`

I pass year, day month, and hour values via arguments. I am attaching screen with settings (hdfs.jpg)

However when I run job I am getting following error. It looks like passed arguments aren't executable and they are passed to shell scrip as String values:

Error Log:

 Log Type: stderr
  
  Log Length: 2307
  stat: `/data/logfiles.db/install_callbacks_csv/year=`date': No such file or directory
  stat: `-d': No such file or directory
  stat: `'1': No such file or directory
  stat: `ago'': No such file or directory
  stat: `+%Y`': No such file or directory
  stat: `/data/logfiles.db/install_callbacks_csv/year=`date': No such file or directory
  stat: `-d': No such file or directory
  stat: `'1': No such file or directory
  stat: `ago'': No such file or directory
  stat: `+%Y`/month=`date': No such file or directory
  stat: `-d': No such file or directory
  stat: `'1': No such file or directory
  stat: `ago'': No such file or directory
  stat: `+%m`': No such file or directory
  stat: `/data/logfiles.db/install_callbacks_csv/year=`date': No such file or directory
  stat: `-d': No such file or directory
  stat: `'1': No such file or directory
  stat: `ago'': No such file or directory
  stat: `+%Y`/month=`date': No such file or directory
  stat: `-d': No such file or directory
  stat: `'1': No such file or directory
  stat: `ago'': No such file or directory
  stat: `+%m`/day=`date': No such file or directory
  stat: `-d': No such file or directory
  stat: `'1': No such file or directory
  stat: `ago'': No such file or directory
  stat: `+%d`': No such file or directory
  stat: `/data/logfiles.db/install_callbacks_csv/year=`date': No such file or directory
  stat: `-d': No such file or directory

[Romain Rigaux]

AFAIK Oozie does not support `date -d '1 hours ago' +%Y` for value of parameters?

Maybe there is an EL function to compute dates:

https://oozie.apache.org/docs/4.1.0/WorkflowFunctionalSpec.html#a4.2_Expression_Language_Functions

If not, the usual way is to create a coordinator that will fill up these date parameters:
https://groups.google.com/a/cloudera.org/forum/#!topic/hue-user/SF4GXpMKpp8
http://gethue.com/hadoop-tutorials-ii-3-schedule-hive-queries-with/

And BTW, Hue has a new Oozie editor coming out in the next release, it is much simpler.

Romain

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

[Łukasz Włodarczyk | Fyber]

Hi Romain,

nice to meet you!

I am using year=`date -d '1 hours ago' +%Y` as a Oozie parameters in workflow. The thing is that it doesn't work for shell script actions but it works perfectly for pig actions when I pass those parameters to pig script action as an arguments. Why it doesn't work for shell script? I think this is a bit misleading.

Thanks,

Lukasz

--

Łukasz Włodarczyk

Business Intelligence Analyst

Skype: l.wlodarczyk-fyber
URL: www.fyber.com

[Romain Rigaux]

When the shell command was started by Oozie, space was used as delimiter for the parameters so it cuts '`date'.

Did you look at the generated command by Oozie in the Oozie logs?

Maybe try to escape the space of the full command?

Romain

On Tue, Feb 24, 2015 at 5:14 AM, Łukasz Włodarczyk <lukasz.w...@fyber.com> wrote:

[Łukasz Włodarczyk | Fyber]

Hi Romain,

yes, coordinator EL functions are usefull:

year => ${coord:formatTime(coord:actualTime(), 'yyyy')}

month => ${coord:formatTime(coord:actualTime(), 'MM')}

day => ${coord:formatTime(coord:actualTime(), 'dd')}

hour => ${coord:formatTime(coord:actualTime(), 'HH')}

I defined them in Advanced Settings (Step 5) as a Workflow properties in Coordinator.

Thanks!

Lukasz