Shebang line limit
88 views
Skip to first unread message
Kelly Thompson
Mar 3, 2016, 10:34:25 PM3/3/16
to Spack
Because of spack's deep paths with long file names, I am running into the shebang line limit [1,2] in scripts installed by spack. For example, when I installed py-sphinx using spack. The main script 'sphinx-build' has this path as the first line in the python script:
#!/usr/projects/codes/vendors/spack.github/opt/spack/linux-x86_64/gcc-4.4.7/python-2.7.11-qgvqwaxahqyr2vhs47z3xhdgymuc4lzz/bin/python
This path is longer than the 79 char limit used by my system, making the script unusable with errors similar to this:
/bin/sh: /path/to/sphinx-build: /usr/projects/codes/vendors/spack.github/opt/spack/linux-x86_64/gcc-4.4.7/py: bad interpreter
Has the spack development team already considered this issue? Are there any solutions readily available (other than installing spack closer to /)? The references below provide various tricks to solve the problem that use intermediate shell scripts, but I think this would be difficult to implement.
Thoughts?
[1] http://stackoverflow.com/questions/10813538/shebang-line-limit-in-bash-and-linux-kernel
[2] https://github.com/pypa/virtualenv/issues/596
#!/usr/projects/codes/vendors/spack.github/opt/spack/linux-x86_64/gcc-4.4.7/python-2.7.11-qgvqwaxahqyr2vhs47z3xhdgymuc4lzz/bin/python
This path is longer than the 79 char limit used by my system, making the script unusable with errors similar to this:
/bin/sh: /path/to/sphinx-build: /usr/projects/codes/vendors/spack.github/opt/spack/linux-x86_64/gcc-4.4.7/py: bad interpreter
Has the spack development team already considered this issue? Are there any solutions readily available (other than installing spack closer to /)? The references below provide various tricks to solve the problem that use intermediate shell scripts, but I think this would be difficult to implement.
Thoughts?
[1] http://stackoverflow.com/questions/10813538/shebang-line-limit-in-bash-and-linux-kernel
[2] https://github.com/pypa/virtualenv/issues/596
Todd Gamblin
Mar 5, 2016, 7:38:44 AM3/5/16
to Kelly Thompson, Spack
Kelly,
Since this has come up something like three times now, and it's pretty irritating, I decided to see what I could do. I think I have a fix. Can you test out this PR?
It might need a little more polish, but basically it adds a script `sbang` at the top level, alongside spack. After install, spack searches through the $prefix/bin directory for any shebang'd scripts, and it inserts a line like this above the real interpreter:
#!/bin/bash /usr/projects/codes/vendors/spack.github/bin/sbang
#!/usr/projects/codes/vendors/spack.github/opt/spack/linux-x86_64/gcc-4.4.7/python-2.7.11-qgvqwaxahqyr2vhs47z3xhdgymuc4lzz/bin/python
...
`sbang`, when invoked, grabs the real interpreter and runs it. Linux seems like it is happy with interpreters that are themselves shebang scripts, but Mac OS X wants a binary, so you have to have the #!/bin/bash there too. In your case the first line is only 62 characters, which seems ok.
This might be of use to the virtualenv folks you mention, as they don't seem to have a real solution to the problem. Maybe I'll ping them if it works in Spack.
Opinions welcome.
-Todd
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://secure-web.cisco.com/12yqwu2RoOrv1oQ6kbdsf-ThnQm18VlJQ5PvR1Xqr-0cfrkYq72yxiU77g5iKAIKe9jzxhrTYk1QkAHffBJRYfDSAJvP2RasqORySoebaBFkLfQxpoIZ7vi5FlY1lCzRpmE5EOuAP0Y3Ls8Dm4ZDDsaOPi7Bxco5RqP-2wf-OcCpkLUJcsrqoSfpuNwu-PFne_1iHfNGXfEqW55G-4UCIvkihKt_jLBHyittUQjW7a1ppyCPfqsTmV_OFXrhwlLoIVkWvrcHhTEUbW-8YsiOvmz9SMQ9ASI02r3S-6sSUu178XaSJ36EKvW1xr5_GSTLv1JGl5xWL2-oDAljtuOZtSZxesKiIl-tvcAwTZmJP5Ri5m6IZUf4Ow0R_9AJXAqtA/https%3A%2F%2Fgroups.google.com%2Fgroup%2Fspack.
For more options, visit https://secure-web.cisco.com/1sbLkKTgKT3Fba66n-fHy1n8D35omu6vpa1UjsbY-B3bUpmAaCayK939xFqOxu-JJb1eHQND1I4nAYKDvsqWxx4PVbW-lZe-6NwSUvnGYFXgWnWSo0T4BEe5UWv_FP3J-QBT4sd2Ku_LJ93SlASVEHFeCv7di-vVe-tvTmZa-tD9en03dAh2EUxezjiZVSZFnjOiik5rGUbn_65ZSn77Dmj7YwwgpYG6rbdfdNuebjVD8C-OGfUTP82g0HXCs1WbCu_4x9tlE2U_x_GjoX4mCq2aLUMs-u9tL4X_Siy7YfOq0SOxmqXzBISyMQZa-lchwGWvdzy1Y3d_wgszT2XCYT5u2xS95g-HPIZn7QQONIQx3lUP8m8lcg7tC7gHGpCY7/https%3A%2F%2Fgroups.google.com%2Fd%2Foptout.
Kelly Thompson
Mar 6, 2016, 10:14:12 PM3/6/16
to Spack, k...@eklaj.org
Todd,
Your PR is working for my test case. I'm only working with Linux and will not be able to test on OSX.
Thanks for the quick turn around on this solution.
-kt
Todd Gamblin
Mar 6, 2016, 10:31:33 PM3/6/16
to Kelly Thompson, Spack
Great! Merged.
Visit this group at https://secure-web.cisco.com/1rirdXmamJ13AtnFGfcjPU8bYzdbCppcLqlEy-n2bIVo3scfYNemhEe8mSmy86MSoaWYORog7w7d1VbVMusMea-660bx2TPEOlUlZowPeNfY7Xq4ksxZVZSnEs6J-VgkN7lLgX-Mls3rda01tfcy8lH_slz78cfQV5bWGNwpWsRpRL1SjHk1e7OqnHJziQNXF0urh5bw2RfKYgBmwbTtXgoh6TLL20dVtvFa2sGHYJe_VT2N6yXz37GtDWH0z4qhjhTiNsBI0WK0MQMT30DjNA_eKsaVLUtmMTnLBkxOgrMpjt7PP4UafTk2SuZBs6GqXlHo589XSYLsxM2bTwlguSfHgr92nK7fwKynSCeCrMoiQH3aeKvrpYC1bijBHNLM3/https%3A%2F%2Fgroups.google.com%2Fgroup%2Fspack.
For more options, visit https://secure-web.cisco.com/1zg84GlDQ7jFQ_avFvabjXc696qlMcdYSdBLC8JaVrC3-qHmK9lX15zQsQ3be3-70YYWHOHiSeSCAXJx7EyuSp65vPu_tuDio_iLUk6suCbS9KpYu00BzLBiuaw1SCAwJ_iie1S4DsYEeiCu5QHoam3-KV0mf441i7xR33OD3RY2jls_pQshu_-xXhQT46041BNktWipKPzQsrjC4TbNFj6tydZi1ATlhHbov35RboqiNyPnt7kqLkiW_QFvj8kK6tRLyaTQVqgG-KE4gE0EhlFcUjiNpwd3cYYokSeUjHIz6CVY2f0F9O8C5OX9f9t3WH-gwIFo55gp0LMcCI68121RYuAdpN03PRw-vreQe2oC6JBGCgkDN1aoFb33q88I_/https%3A%2F%2Fgroups.google.com%2Fd%2Foptout.
Kelly Thompson
Mar 7, 2016, 10:46:02 PM3/7/16
to Spack, k...@eklaj.org
I think I may have given the green light too quickly on for feature/sbang. Using a clean 'develop' branch, I am seeing this error:
spack find
==> 0 installed packages.
spack install qt
[many packages are built...]
[Python-2.7.11 is built]
[xcb-proto1.11 is built]
libxcb-1.11.1 fails with the error:
==> Error: No spec file found at path /var/tmp/spack/opt/spack/linux-x86_64/gcc-4.4.7/libxcb-1.11.1-fcoxqkvmi3a2y2qvmcevwvlhbvh5wlvb/.spack/spec.yaml
It looks like the path /var/tmp/spack/opt/spack/linux-x86_64/gcc-4.4.7/libxcb-1.11.1-fcoxqkvmi3a2y2qvmcevwvlhbvh5wlvb/lib/python2.7/site-packages is created before spack expects /var/tmp/spack/opt/spack/linux-x86_64/gcc-4.4.7/libxcb-1.11.1-fcoxqkvmi3a2y2qvmcevwvlhbvh5wlvb to exist (see logic in lib/spack/spack/directory_layout.py).
Is there any chance that the sbang feature modified python in a way to causes these site-packages directories to be generated in each install directory? I haven't identified the source of the error, but decided it should reported sooner rather than later. Can anyone else confirm this behavior after updating with the feature/sbang update?
BTW - after the above error, if you blindly rerun 'spack install qt' (libxcb isn't actually installed, but spack thinks it is), the same error message shows up in qt before compiling starts.
-kt
Todd Gamblin
Mar 8, 2016, 6:07:36 AM3/8/16
to Kelly Thompson, Spack
Hi Kelly,
I made a bug for this:
And a PR that fixes it:
This wasn't a bug with sbang, but with some changes I made while merging a previous PR. The install prefix creation step was indeed moved too late.
-Todd
Great! Merged.
It might need a little more polish, but basically it adds a script `sbang` at the top level, alongside spack. After install, spack searches through the $prefix/bin directory for any shebang'd scripts, and it inserts a line like this above the real interpreter:#!/bin/bash /usr/projects/codes/vendors/spack.github/bin/sbang#!/usr/projects/codes/vendors/spack.github/opt/spack/linux-x86_64/gcc-4.4.7/python-2.7.11-qgvqwaxahqyr2vhs47z3xhdgymuc4lzz/bin/python...`sbang`, when invoked, grabs the real interpreter and runs it. Linux seems like it is happy with interpreters that are themselves shebang scripts, but Mac OS X wants a binary, so you have to have the #!/bin/bash there too. In your case the first line is only 62 characters, which seems ok.This might be of use to the virtualenv folks you mention, as they don't seem to have a real solution to the problem. Maybe I'll ping them if it works in Spack.Opinions welcome.-Todd
From: <sp...@googlegroups.com> on behalf of Kelly Thompson <k...@eklaj.org>
Date: Thursday, March 3, 2016 at 7:34 PM
To: Spack <sp...@googlegroups.com>
Subject: [spack] Shebang line limit
Because of spack's deep paths with long file names, I am running into the shebang line limit [1,2] in scripts installed by spack. For example, when I installed py-sphinx using spack. The main script 'sphinx-build' has this path as the first line in the python script:
#!/usr/projects/codes/vendors/spack.github/opt/spack/linux-x86_64/gcc-4.4.7/python-2.7.11-qgvqwaxahqyr2vhs47z3xhdgymuc4lzz/bin/python
This path is longer than the 79 char limit used by my system, making the script unusable with errors similar to this:
/bin/sh: /path/to/sphinx-build: /usr/projects/codes/vendors/spack.github/opt/spack/linux-x86_64/gcc-4.4.7/py: bad interpreter
Has the spack development team already considered this issue? Are there any solutions readily available (other than installing spack closer to /)? The references below provide various tricks to solve the problem that use intermediate shell scripts, but I think this would be difficult to implement.
Thoughts?
[1] http://stackoverflow.com/questions/10813538/shebang-line-limit-in-bash-and-linux-kernel
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://secure-web.cisco.com/12yqwu2RoOrv1oQ6kbdsf-ThnQm18VlJQ5PvR1Xqr-0cfrkYq72yxiU77g5iKAIKe9jzxhrTYk1QkAHffBJRYfDSAJvP2RasqORySoebaBFkLfQxpoIZ7vi5FlY1lCzRpmE5EOuAP0Y3Ls8Dm4ZDDsaOPi7Bxco5RqP-2wf-OcCpkLUJcsrqoSfpuNwu-PFne_1iHfNGXfEqW55G-4UCIvkihKt_jLBHyittUQjW7a1ppyCPfqsTmV_OFXrhwlLoIVkWvrcHhTEUbW-8YsiOvmz9SMQ9ASI02r3S-6sSUu178XaSJ36EKvW1xr5_GSTLv1JGl5xWL2-oDAljtuOZtSZxesKiIl-tvcAwTZmJP5Ri5m6IZUf4Ow0R_9AJXAqtA/https%3A%2F%2Fgroups.google.com%2Fgroup%2Fspack.
For more options, visit https://secure-web.cisco.com/1sbLkKTgKT3Fba66n-fHy1n8D35omu6vpa1UjsbY-B3bUpmAaCayK939xFqOxu-JJb1eHQND1I4nAYKDvsqWxx4PVbW-lZe-6NwSUvnGYFXgWnWSo0T4BEe5UWv_FP3J-QBT4sd2Ku_LJ93SlASVEHFeCv7di-vVe-tvTmZa-tD9en03dAh2EUxezjiZVSZFnjOiik5rGUbn_65ZSn77Dmj7YwwgpYG6rbdfdNuebjVD8C-OGfUTP82g0HXCs1WbCu_4x9tlE2U_x_GjoX4mCq2aLUMs-u9tL4X_Siy7YfOq0SOxmqXzBISyMQZa-lchwGWvdzy1Y3d_wgszT2XCYT5u2xS95g-HPIZn7QQONIQx3lUP8m8lcg7tC7gHGpCY7/https%3A%2F%2Fgroups.google.com%2Fd%2Foptout.
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://secure-web.cisco.com/1rirdXmamJ13AtnFGfcjPU8bYzdbCppcLqlEy-n2bIVo3scfYNemhEe8mSmy86MSoaWYORog7w7d1VbVMusMea-660bx2TPEOlUlZowPeNfY7Xq4ksxZVZSnEs6J-VgkN7lLgX-Mls3rda01tfcy8lH_slz78cfQV5bWGNwpWsRpRL1SjHk1e7OqnHJziQNXF0urh5bw2RfKYgBmwbTtXgoh6TLL20dVtvFa2sGHYJe_VT2N6yXz37GtDWH0z4qhjhTiNsBI0WK0MQMT30DjNA_eKsaVLUtmMTnLBkxOgrMpjt7PP4UafTk2SuZBs6GqXlHo589XSYLsxM2bTwlguSfHgr92nK7fwKynSCeCrMoiQH3aeKvrpYC1bijBHNLM3/https%3A%2F%2Fgroups.google.com%2Fgroup%2Fspack.
For more options, visit https://secure-web.cisco.com/1zg84GlDQ7jFQ_avFvabjXc696qlMcdYSdBLC8JaVrC3-qHmK9lX15zQsQ3be3-70YYWHOHiSeSCAXJx7EyuSp65vPu_tuDio_iLUk6suCbS9KpYu00BzLBiuaw1SCAwJ_iie1S4DsYEeiCu5QHoam3-KV0mf441i7xR33OD3RY2jls_pQshu_-xXhQT46041BNktWipKPzQsrjC4TbNFj6tydZi1ATlhHbov35RboqiNyPnt7kqLkiW_QFvj8kK6tRLyaTQVqgG-KE4gE0EhlFcUjiNpwd3cYYokSeUjHIz6CVY2f0F9O8C5OX9f9t3WH-gwIFo55gp0LMcCI68121RYuAdpN03PRw-vreQe2oC6JBGCgkDN1aoFb33q88I_/https%3A%2F%2Fgroups.google.com%2Fd%2Foptout.
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://secure-web.cisco.com/1IDtL1Aq_Hkb8-JnhAaRbw0vDn1f0EIrXMgMt3otZh1Qzg_WybkITKUbH_sALQvnU6n8JdpF_9Subw00EuMh42Bz3-YfBLySbOG_Ob1uaH5Dy9zQYIJBhyz6vxR4snlaa0h-WFNlPn21QNaw3iWM5khWX3kUktJUD8_YeBDHw6zJXCF1YkfulRynVd9Gk-DI7jYydSMI2LlfNSqGUTho842Cm4AViRyRIU-zf8-RPn-AQXqbffu1f90-nN7y74Fy_SVW3W77GwYxjHLDdR8CT70GkPnybn-jT_xBAyyG2H86ZpfeqhTvG6SUAr242Koqxz5YTVWY9H1gwpWVgWRhkHF5FrwEeYP8g7SaHWCebLg43vTvCRDGw3AtVNp6RmMkD/https%3A%2F%2Fgroups.google.com%2Fgroup%2Fspack.
For more options, visit https://secure-web.cisco.com/13sJQyDA-PYNYhapndNsvQJgng50fA8LTNfStBqyzOyXWc8Gh5PKwNbg1sm1ye1O9q0YKsyyEdDdCqHEHaWYhmIjoDPzdlVA42v3cm2hD-Qar0Ce2sMaveNc4lqY2A8HvNeP4-GQvKLSMxjiVc9qj6_vU8QgyEZHtc9n2XsPKTNjdt8QTAccyq9qOMs33SAzSK-DNCgTWzavW6eSBRxhNcOgjTnC1jzq-3hX4btfs0Ggjl2jK5NTluKmVqZ-2lrIPr2bLSimF2feIOBWA8XzABIjECnJoeT3Sn09wvUtUl2iofSwR9Hjp00xtftutOsbMsLMkUi6rX7aHe5PEk2xQTd6Kwd6emPwUOmT-ZndiQywLhFy6RaUCKvbO6QbsYZm-/https%3A%2F%2Fgroups.google.com%2Fd%2Foptout.
Kelly Thompson
Mar 9, 2016, 10:21:58 AM3/9/16
to Spack, k...@eklaj.org
PR504 fixes the issue I was seeing. Thanks.
-kt
Reply all
Reply to author
Forward
0 new messages