Help me develop a Job Tracker (?)

[Softeisbieger]

Hi,

maybe first, a few things about me: I have no experience in websites / django. I have a background in scientific computing and I especially enjoy using python. I did some GUI programming in .NET and SQL.

So here is my project: I want to develop a 'Job Tracker' (approx. like an Issue Tracker), with a central twist: In traditional issue trackers the work flow is simple / unique / not adaptable: For example 'Open -> Working -> Closed'. What I have in mind is, to enable the users to define one or more such work flows and to manage these using a single applications. If some program / website already provides this, a note is welcome.

Why? I am developing this for a landscape gardening company, where
 Person A gets a task from some customer.
 -> Person B is responsible for ordering people to solve the task.
 -> Person B is responsible for collecting information to enable billing
 -> Person C creates the invoice -> Done
This is one example work flow. There are many other possible workflows, for example for creating a cost estimate for a customer. In the past this all occured on paper. The company reached a size where this is not feasible anymore.

So one thing that is needed is a customer management. This is straightforward. I am uncertain how to handle different work flows, though. The user should be able to define these 'on the fly' and I have no idea how to express this in terms of django. What I know about a workflow:

• it has 1..n steps
• it can be connected to a customer
• each carries some information: date created, creator, description, ...
  
• at all times the job is assigned to someone
  
• on definition of the workflow, one must be able to preset each step: Who is (typically) responsible, when should this step be completed.

I can imagine two possibilities: First, hide the different workflows from the database. Use an 'intelligent' string (json) to store different workflows with the same interface. Second, somehow create new tables for new workflows on the fly. The second seems to be the more elegant solution.

So this is my setup. Any idea / hint / solution is welcome.

Thanks!

 Johannes

[Javier Guerra Giraldez]

On Fri, Jun 26, 2015 at 3:01 AM, Softeisbieger <j.kit...@gmx.de> wrote:
> I am uncertain how to handle different work flows, though. The user should
> be able to define these 'on the fly' and I have no idea how to express this
> in terms of django. What I know about a workflow:

there's a thing known as a workflow engine, with extensive theory,
definitions, terminology, research and implementations. Check some of
theory in wikipedia[1].

as is common in Django, a very useful first stop is
djangopackages.com, where you can find a comparison grid of workflow
engines already created for Django[2]

some of them are quite capable, and even if miss something you would
like, they're not too hard to adapt, once you have the theory well
understood.

Of course, if you're new to Django, first you _have_ to do the
tutorial, or lots of things might not make sense.

[1]: https://en.wikipedia.org/wiki/Workflow_engine
[1.1]: https://en.wikipedia.org/wiki/Business_process_modeling
[1.2]: https://en.wikipedia.org/wiki/Petri_net
[1.3]: https://en.wikipedia.org/wiki/Kahn_process_networks
[2]: https://www.djangopackages.com/grids/g/workflow/

--
Javier

[Softeisbieger]

I knew there had to be something like a workflow engine, thanks for the hint.

I had a look at django-viewflow which looks pretty promising. Still, as far as I'm aware, some points are beyond the scope of this framework:

• It would be nice if users could set up their own workflows.
• I would like to have the flexibility of a workflow engine and some ingredients from standard bug trackers: It should be possible to comment on the different workflows.
• I'd like to maintain a history of each workflow: Creation date, status change date, ... (who did what and when?)
• User can subscribe to a workflow. They can choose when they want to get an update: Status change, someone commented, ...
• when defining a workflow, for each activity one should be able to preset a user who is typically responsible and a due date for this event.
• a comprehensive list of all workflows: filter based on workflow type, responsible user, in time / overdue, type of work, customer...
• user should be able to change some properties of a workflow that is currently executed: For example to postpone a deadline. Users need to give a reason for this.
  

I'll dig a bit into django-viewflow to see how that works. django-fsm may work as well...

[Softeisbieger]

I am currently thinking about the database representation. I don't think I can solve this with django-viewflow: I need to be able to specify in advance a deadline and an assignee for each task of a work flow.

Maybe consider the following: A work flow consists of a number of tasks. Each task is followed by another task or 'end'. Each task has a predecessor which is either another task or 'start'. Now model a task which contains some information and the primary key of it predecessor and successor. A work flow contains a description and the primary key of the currently active task. The advantage is that in this way a work flow can consist of any number of tasks. The user then could define a template work flow where for example the assignee of some task (or all) is predefined. New work flows can then be started using either a template or a new definition from scratch.

Maybe you could give me some input. I am not to familiar with all this stuff...

[Javier Guerra Giraldez]

On Thu, Jul 2, 2015 at 10:26 AM, Softeisbieger <j.kit...@gmx.de> wrote:
> I am currently thinking about the database representation. I don't think I
> can solve this with django-viewflow: I need to be able to specify in advance
> a deadline and an assignee for each task of a work flow.

first of all, its important to note that a full-featured workflow
engine is Turing-complete, so it's theoretically undecidable. There
are limits on what kind of tools an automated system can provide.
specifically, it's easy to write workflows for which it's impossible
to calculate a detailed schedule.

if you want your system to be able to define deadlines for each task
given a final deadline, or some useful "%-done" indicator, then you
might be better served by a "limited shape" workflow model. The
simplest case (yet widely used) is a linear pipeline of tasks, which
can look similar to what you describe.

with a linear pipeline, it's easy to set time estimates for each step,
to calculate proposed milestone dates backward from the final deadline
date. If you get behind schedule, it could even estimate "tight
goals" by proportionally distributing the remaining time between the
remaining tasks.

some limitations of the linear model can be relaxed by defining some
steps as "optional", or allowing some authorizations to skip a task,
or by specifying the prerequisites of a task as not necessarily "all
the previous steps".


if you still want the wide flexibility of a full-featured workflow
engine, recognize that it's not easy for most people to think in terms
of abstract states, and planning all the interdependences between
steps of any slightly complex process is in fact a programming
activity, even if done graphically. IOW, if you ask users to
repeatedly one-shot workflows, they will naturally limit themselves to
simple lists of tasks and manage any unexpected circumstances by
(ab)using 'override' permissions.

workflow engines shine if you have a mid-to-large group of people
doing some complex process repeatedly. a good example (and one where
i've found them quite a lifesaver) is in the publishing industry: you
have several editing steps, with well-defined approval points, and
rules of when some work must be redone, discarded, or rushed. and it
must be done again and again, for each campaign, so it pays up to do a
very detailed model.


note that a state-transitions diagram is usually drawn as several
circles connected by arrows. The tasks are associated to the arrows
(transitions), not the the circles (states). Also, in most engines,
each transition has some permission restrictions and maybe a way to
specify preconditions, which makes it easy to show, for a given user,
which are the transitions available for a specific process.


good luck!

--
Javier

[Softeisbieger]

On Thursday, July 2, 2015 at 6:04:57 PM UTC+2, Javier Guerra wrote:

good luck!

--
Javier

Thanks for your input! Indeed, a linear pipeline should be sufficient. What do you think of my idea of organizing this in two database tables? One table models a work flow. A work flow points to the currently active task in another table. Each task has information about its predecessor / successor. In principle like a doubly-linked list. This solution feels a bit brittle / hacked to me. Again, my reason is, I want to store all the work flows with this standard interface. Also, users can easily create new work flows with any number of steps. Maybe there is a better way?

In case you are interested: I wrote earlier that I am developing this for a landscape gardening company. Since the company grew quite a bit recently and more people are working in administration, it is necessary to organize the flow of jobs: Quickly see who is responsible, what is currently happening, which jobs are overdue. Also each job is touched by different people (something like acquisition -> execution -> accounting) and the framework should organize this as well. There are different work flows for example to execute work and to make a quotation. Until now this is organized using a combination of paper and memory which is not feasible anymore.

[Javier Guerra Giraldez]

On Fri, Jul 3, 2015 at 2:50 AM, Softeisbieger <j.kit...@gmx.de> wrote:
> Thanks for your input! Indeed, a linear pipeline should be sufficient. What
> do you think of my idea of organizing this in two database tables? One table
> models a work flow. A work flow points to the currently active task in
> another table. Each task has information about its predecessor / successor.
> In principle like a doubly-linked list. This solution feels a bit brittle /
> hacked to me. Again, my reason is, I want to store all the work flows with
> this standard interface. Also, users can easily create new work flows with
> any number of steps. Maybe there is a better way?

i don't think a linked list is appropriate for databases. the
simplest design would be: a Workflow model to identify it (with id,
name, creator, that kind of things), and a Step model that 'belongs'
to the Workflow something like this:

class Workflow(models.model):
name = models.charfield()

class Step(models.model):
workflow = models.ForeignKeyField(Workflow)
name = models.charfield()
order = models.SmallIntegerField()
taskdescriiption = models.TextField()

class Meta:
ordering = ['workflow','order']

def previous(self):
return self.workflow.step_set.filter(order_lt=self.order)[-1]

def next(self):
return self.workflow.step_set.filter(order_gt=self.order)[0]


those two models keep the definition of a workflow. then, define the
'working' models, one for a whole project, and other for each task to
perform:

class Project(models.model):
name = models.charfield()
workflow = models.ForeignKeyField(Workflow)
responsible = models.ForeignKeyField(User)
deadline = models.DatetimeField()

def started_tasks(self):
return self.task_set.filter(start__isnull=False)

def finished_tasks(self):
return self.task_set.filter(finish__isnull=False)

def in_process_tasks(self):
return self.task_set.filter(start__isnull=False, finish__isnull=True)

def finished(self):
return self.started_tasks.exist() and not self.in_process_task.exist()

class Task(models.model):
project = models.ForeignKeyField(Task)
step = models.ForeignKeyField(Step)
deadline = models.DatetimeField()
assigned_to = models.ForeignKeyField(User, blank=True, null=True)
start = models.DatetimeField(blank=True, null=True)
finish = models.DatetimeField(blank=True, null=True)


in most cases, it's better not to have a 'current task' pointer. just
some tasks that have been started but not finished(yet). it's up to
you to decide if it's appropriate to have more than one task in
process for a given project, or if it's valid to start a task before
some of the previous tasks.

usually it's also nice to add action methods to some models, maybe a
project.start(user) that creates the first task and starts it, and a
project.next(user) that finishes a task and creates/starts the next
one (if there's any more step(s)).

--
Javier

[Softeisbieger]

Let's see if I understand your proposal correctly: Workflow and Step act as template and are used to define the structure of different workflows. Project and Task 'implement' the template and are used as for actual work flows. So I formulate things that are fixed using the template and for the rest I use Project and Task models.

I adapted your code a bit: For each Step one can specify an assignee and a deadline that act as suggestions for the same data of the task model. So when a task is created, it is at first filled using the information in the Step model. In addition I added some more information to the Project.

from django.db import models
from django.contrib.auth.models import User

class Workflow(models.Model):
    name = models.charfield(max_length = 200)
    description = models.TextField()

class Step(models.Model):

    workflow = models.ForeignKeyField(Workflow)
    name = models.charfield()
    order = models.SmallIntegerField()

    taskdescription = models.TextField()

    # suggestions for deadline / assignee
    deadline = # implement as number of days from project.start_date or as number of days from start of current step. How?

    assigned_to = models.ForeignKeyField(User, blank=True, null=True)

class Project(models.Model):

    name = models.charfield()
    workflow = models.ForeignKeyField(Workflow)

   
    creation_date = models.DatetimeField(auto_now_add = True)
    creator = models.ForeignKeyField(User)

    start_date = models.DatetimeField() # in case start of project differs from creation date, default: creation date

class Task(models.Model):
    project = models.ForeignKeyField(Project)
    step = models.ForeignKeyField(Step)
    deadline = models.DatetimeField() # initially computed from project.start_date, may be postponed by the user
    assigned_to  = models.ForeignKeyField(User, blank=True, null=True) # initially as given by Step, may be changed later

[Softeisbieger]

So I made some progress on this project. I set up the means to manage customers, which is pretty basic stuff: I usedCreateView, ListView and the like.

Now I have a more complex situation where I am stuck: I am now working on creating projects (see models above). Here I have to implement the following:

1. Choose a work flow
2. Based on choice in 1. show forms for the project and the different tasks (as defined by Workflow and Step). Show presets as defined in Workflow / Step.
   

My Problem: I don't know how to approach this. Somehow I have to create Project / Jobs as indicated by the chosen Workflow. Unfortunately this a bit over my head at the moment: Could you give me rough directions on how to approach this?  Drop some keywords what is needed to make this work?

[Aaron C. de Bruyn]

You might be looking for the FormWizard.

In the latest version of Django, it has been moved to an external
application (https://github.com/django/django-formtools/blob/master/docs/wizard.rst).
In older version of Django, it is part of the corp app
(https://docs.djangoproject.com/en/1.7/ref/contrib/formtools/form-wizard/).

-A

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/c3a735a5-8141-4c5b-bf1a-1ed064daf298%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Softeisbieger]

The FormWizard seems to be a good hint, since my input process consists of to steps: First create a Project and specify the Workflow to use. Second, show the Tasks for the Project (as indicated by the chosen Workflow / Step) and allow the user to tweak the Tasks (change assignee, deadline).

I can't wrap my head around the following: How do I do the second step: I have to show multiple forms for the different tasks on a single page (FormWizard: ' Define a number of Form classes – one per wizard page.') and I have to somehow know the chosen workflow in the first step... Is this possible with FormWizard? I could probably build a single Form for the multiple tasks, but how do I get the information on the chosen Workflow?

Essentially this is a two-step process: Based on the choice in step one, do something in step two...

Okay, I'll have a look at  WizardView.get_cleaned_data_for_step(step) . This seems to be the right deal...

[Softeisbieger]

First: one cannot use  get_cleaned_data_for_step(step) inside get_form_initial() (see: here).

Anyone familiar with FormWizard? I am a bit confused:
I want to use 'get_form_initial' to preset some values based on the choice in the previous step. ATM my code looks like this:

class JobWizard(CookieWizardView):
    form_list = [forms.JobForm, forms.TaskForm]
    def done(self, form_list, **kwargs):
        do_something_with_the_form_data(form_list) # todo
        return HttpResponseRedirect('/jobs/')

    def get_form_initial(self, step):
        current_step = self.storage.current_step
        print(current_step)
        print(self.steps.current)
        print('\n')

        if current_step == '1':
            return self.initial_dict.get(step, {'deadline': timezone.now()})

        return self.initial_dict.get(step, {'street': self.storage.current_step})  

The problem is that self.steps.current or self.storage.current_step don't work as I would expect: When working on the first form (forms.JobForm) it is '0' (as expected), but when submitting the first form and advancing to the second, it is still '0' (why?). So the above if clause if current_step == '1' does not work as I would like it to. Once I submit the second form self.steps.current is incremented. So maybe I'm using the wrong variable: How do I determine if I'm in the second step? How can I rename the step names?

My goal at first is to load data from the first step and use it to preset some of the fields of the second step. Once that works I need to somehow create a formset for the different tasks...


[Softeisbieger]

Ok, so I set up the FormWizard, but it has some strange side effects. I implemented the process in three steps:


1. Choose a workflow
2. Specify information for the new Job (Adress, deadline, customer...)
3. Review deadlines and assigness for the different steps.

Problems occur in the third step:

• for some reason additional, unrelated tasks from the db are shown
•  even though values for the mandatory fields in the form are chosen, an error is thrown that these are missing
• if I specify a non-mandatory field (assignee) suddenly the error above does not occur anymore an the process finishes without error (but not necessarily the right result)

I am thinking that FormWizard cannot handle the 'dynamic' changes I am doing: The third step is adjusted based on the chosen workflow in the first step. I read somewhere that this is can be an issue.

Maybe one of you can tell me where the mistake is? Any hint is appreciated.

My FormWizard:

class JobWizard(SessionWizardView):
    form_list = [
            ('first', forms.WorkflowSelectForm),
            ('second', forms.JobForm),
            ('third', forms.TaskFormSet)
            ]

    def done(self, form_list, **kwargs):
        # store Job in database
        # how do I access data in ValuesView?
        forms = []
        for form in form_list:
            forms.append(form)
       
        job = forms[1].save()
        for form in forms[2]:
            current_task = form.save(commit=False)
            current_task.job = job            
            current_task.save()

       

        return HttpResponseRedirect('/jobs/')

    def get_form_initial(self, step):

        if step is None:
            step = self.steps.current

        if step == 'second':
            prev_data = self.storage.get_step_data('first')
       
            return {'deadline': (timezone.now() + timezone.timedelta(days=30)).replace(hour=12, minute=0, second=0),
                        'workflow': Workflow.objects.get(pk = prev_data.get('first-workflow')),
                        'creator': self.request.user
                        }
       
        if step == 'third':
            first_step_data = self.get_cleaned_data_for_step('first')
            second_step_data = self.get_cleaned_data_for_step('second')
            steps = Step.objects.filter(workflow=first_step_data['workflow']).order_by('order')
            # get total time
            t_total = 0.
            for step in steps:
                t_total += step.time_fraction
            deadline = second_step_data['deadline']
            start_date = second_step_data['start_date']
            job_duration = (deadline - start_date).total_seconds()
           
            initial_data = []
            task_deadline = start_date
            for step in steps:
                fraction = step.time_fraction / t_total
                task_duration = fraction * job_duration
                task_deadline = task_deadline + timezone.timedelta(seconds = task_duration)
                assignee = step.assigned_to
                data = {'deadline':  task_deadline,
                        'assigned_to': assignee,
                        'step': step}
                initial_data.append(data)
       
            return initial_data
           
        return self.initial_dict.get(step, {})

    def get_form(self, step=None, data=None, files=None):
        form = super(JobWizard, self).get_form(step, data, files)
        if step is None:
            step = self.steps.current

        if step == 'third':
            first_step_data = self.get_cleaned_data_for_step('first')
            steps = Step.objects.filter(workflow=first_step_data['workflow']).order_by('order')
            form.extra = len(steps)
        return form

The forms:

class JobForm(forms.ModelForm):
    start_date = forms.DateTimeField(widget=forms.DateTimeInput, initial=timezone.now())
    workflow = forms.ModelChoiceField(queryset=Workflow.objects.all(),widget=forms.TextInput(attrs={'readonly':'readonly'}))
    creator = forms.ModelChoiceField(queryset=User.objects.all(),widget=forms.TextInput(attrs={'readonly':'readonly'}))
    class Meta:
        model = Job
        exclude = ('creation_date', 'paused', 'paused_until', 'completed')


class TaskForm(forms.ModelForm):
    class Meta:
        model = Task
        exclude = ('job', 'finished', )

TaskFormSet = modelformset_factory(Task, form=TaskForm )




[Softeisbieger]

I could fix part of my problem: The formtools formwizard doc states for the us of modelformsets:

'WizardView supports ModelForms and ModelFormSets. Additionally to initial_dict, the as_view() method takes an instance_dict argument that should contain model instances for steps based on ModelForm and querysets for steps based on ModelFormSet.'

For me this does not make to much sense: Why should I specify model instances / querysets for steps basend on ModelForm / ModelFormSet? In the case of ModelFormSet, FormView seems to prepopulate the formset with all model instances from the database, causing half of my error above.
Why has this behavior been chosen? I don't get it...

Nevertheless, to fix this I added 'instance_dict = {'third': Task.objects.none()}' to the JobWizard.

Still half of the error persists: Upon submitting the final formwizard-form, the last form in the TaskFormSet produces a 'RelatedObjectDoesNotExist' error for Step ('Task has no step'), even though it was preset by 'get_form_initial'. All other forms are saved without a problem. So what is special about the last form in the formset? Ugh...