I really liked the response Kurt gave; full of gold. I doubt I can add much. I did want to share with you that the situation you are experiencing, is in my experience more common than you might think. There are perfect agile shops, and there are clients all around the world learning and growing through the hard work of the agile agents of change that have taken their own journey towards helping to educate them. But there are still a lot of traditional places too. I am sure you could leave your place (current econmic client aside), and of course you could stay and become an agile agent of change.
The different roles of a software development effort have their roots in traditional ideas of software engineering as a discipline. Some teams/companies follow the traditional because of what they know and because they have experience and some internalized level of success with it. Depending on the situation, the skills of the team, the topology of the whole team structure, traditional might seem like the better approach over other neo-disciplines. That may or may not be true, but its often not black and white.
I dont think craftmanship is about just being agile either. I think agile values and principles exude a heigtened appreciation for the discipline, but they do not define the discipline. And while a coder with thousands of hours of practise will surely be on the unending road to craftmanship and mastery, software craftmanship is much more. Cory Foy wrote a good piece early this year about focusing on other aspects of craftmanship, from team work to collaboration. I share this view. A master woodworker, while a craftsman, does not fully map to software in my opinion. So just focusing on the code, while rewarding, is limited IMHO.
And therefore the answer is in the question - if we are craftsman that have answers to solve this problem you face, it implies that it is more than just coding, being agile or something else more focused. Your situation requires an agent of change, be it agile or some other approach. You could walk away, and I dont deride anyone who does that, as I have walked that path once or twice myself. I would say the problem you face is in how do you become that agent of change, how do you model, imbue and exude the values and principles, practices and disciplines that can lead the minds of your team to a better place? You are not faced with a simple problem, because in truth you need to slowly change the beliefs of others, which is not easy. But I do believe a craftsman/journeyman harbours the qualities that can help with this problem.
I am in a place that is traditionally chaotic, and they have had a lot of success with this approach. The more success with something, the greater the challenge in changing someones beliefs about that artefact of success. But almost always, there are little opportunities to model something. Such opportunities were not apparent when I first got here, and my puer aeternus bubbled underneath the surface. Eventually I saw some opportunities to demonstrate a different approach to the collaboration required between the customer team and development. The requirements provided to use by our BA were only half complete. So I started split the work up for myself into excel, with user stories and then the acceptance tests, which we all call executable specifications now. I kept this to myself, but referred to it in meetings, dropping in nomenclature when I could. I even started creating my own burndowns and putting them up above my monitors. Slowly people asked me what the graphs were, and I showed them my own personal plan.
Dont get me wrong, a personal plan, my own burndowns, my own version of requirements are all heresy in some respects, depending on the motivation for their creation. I am early in the process of guiding some semblance of change, and all of these artefacts are nothing but nudges towards slowly developing experience with different concepts in the minds of my team members. Unless a team/company has willingly requested someone to come in and mentor them on a different approach, other devices of change are required. Patience is a necessary quality for an agent of change. I have had to let small battles be lost, much to my chagrin at times.
Progress is being made. Slow or fast? These are just concepts. I let go of a time, and let go of any need for this situation to be fixed immediately. In only a few months however, we now talk about user stories, scenarios, velocity, test driven development, contiual integration, iterations. The BA has been the greatest challenge and I would say he is still the anchor in the sea. I honour his fear of change, and take it upon myself to adapt and develop the appreciation and understanding of him as a person, so that i can work more effectively with and develop trust. Relationship building has been key. The quality of work I and the team are putting out now has gone up dramatically, and the ability to make changes is much quicker with little fuss of re-engineering. But all of these skills that in my opinion do embody craftmanship, do not come even close to the impact relationships and collaboration has had.
In the words of Gandhi: Be the change you want to see in the world.
Whatever that may look like.
Warm regards,
Nick.