trouble using local variable in shell scripts

[Andy Davidson]

The following will run a shell script and store the results in a juypter variable ‘dataSet’

dataSets = !aws s3 ls s3://com.pws.twitter/json/StreamingKafkaGnipCollector/gnip/ |grep PRE | awk -F'PRE' '{print $2}'

If I try and define a variable that represents the URL the command will fail. 

baseURL="s3://com.pws.twitter/json/StreamingKafkaGnipCollector/gnip/"

!echo $baseURL

dataSets = !aws s3 ls $baseURL | grep PRE | awk -F'PRE' '{print $2}’ 

The problem is with my awk command. I think maybe the note book is trying to expand $2. 

Any idea how I can fix this?

Thanks

Andy

[Fernando Perez]

Mmh, what version are you using?  We don't expand the $2 inside quotes, the following similar code works fine for me in the notebook (not trying yours as I don't have the aws cli installed):

user = 'root'

!ps aux | grep $user | awk '{print $2}'

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/D3A97FFB.40D33%25Andy%40SantaCruzIntegration.com.
For more options, visit https://groups.google.com/d/optout.

--

Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

[Andy Davidson]

Hi Fernando

$ pip-3.4 list |grep jup

jupyter (1.0.0)

jupyter-client (4.2.2)

jupyter-console (4.1.1)

jupyter-core (4.1.0)

Thanks

Andy

From: <jup...@googlegroups.com> on behalf of Fernando Perez <fpere...@gmail.com>
Reply-To: <jup...@googlegroups.com>
Date: Monday, July 11, 2016 at 6:28 PM
To: Project Jupyter <jup...@googlegroups.com>
Subject: Re: [jupyter] trouble using local variable in shell scripts

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAHAreOoTgjR2X%2BZdJnOOsB9DsBR8ndbhvhCFNRRJxVcxSV-y5Q%40mail.gmail.com.

[Fernando Perez]

On Wed, Jul 13, 2016 at 11:41 AM, Andy Davidson <An...@santacruzintegration.com> wrote:

Hi Fernando

$ pip-3.4 list |grep jup

jupyter (1.0.0)

jupyter-client (4.2.2)

jupyter-console (4.1.1)

jupyter-core (4.1.0)

Sorry, I should have been more specific: what version of *IPython*? That logic is in the IPython kernel itself, so the Jupyter tools have no impact.

BTW, does this snippet work for you as expected? It does for me (note that my original version did have a problem, I should have typed {user} instead of $user:

user = 'root'

!ps aux | grep {user} | awk '{print $2}'

I get this even at the terminal:

In [4]: user = 'root'

   ...: !ps aux | grep {user} | awk '{print $2}'

70594

70592

Cheers

f

[Andy Davidson]

Hi Fernando

Strange I could not cut-n-past from chrome so I include screen shot images

[ec2-user@ip-172-31-22-140 emojiSampleData]$ pip-3.4 list |grep ipython

You are using pip version 6.1.1, however version 8.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

ipython (4.2.0)

ipython-genutils (0.1.0)

# your example with output

# variable substitution works as long as I am not using AWS $2

# variable substitution AWK $2 does not work

# works with if there are no variable substitution. 

From: <jup...@googlegroups.com> on behalf of Fernando Perez <fpere...@gmail.com>
Reply-To: <jup...@googlegroups.com>
Date: Wednesday, July 13, 2016 at 11:49 AM
To: Project Jupyter <jup...@googlegroups.com>
Subject: Re: [jupyter] trouble using local variable in shell scripts

--

You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAHAreOp5GV0LqXrQZPDSPpFnMRoMpsXuKDXfWJ0Ff-6dXG2Lxg%40mail.gmail.com.

[Fernando Perez]

On Wed, Jul 13, 2016 at 1:52 PM, Andy Davidson <An...@santacruzintegration.com> wrote:

# variable substitution AWK $2 does not work

Try typing

datasets = !aws s3 ls {baseURL} | grep PRE...

i.e., use {baseURL} instead of $baseURL.

Does that make any difference?