Rapid feedback

[Rachael Kenney-Reid]

Hello fellow evaluators! 

I'm wondering if some of you might be able to give me feedback on something in thinking about.  

My team is working on an evaluation for an office within the Veterans Health Administration. I would like to get information to that office quickly (maybe a weekly or monthly list of a few things we've heard on interviews) but the qualitative methodologist on the project (who comes from a research background and isn't formally trained in evaluation) advises that we should wait until the data is analyzed. We are taking a formative approach to the evaluation so my sense is that the context is different than research and we should share.

What have your experiences been? Is it a bad idea to share preliminary findings? And, if you've rapidly shared information, what formats have been useful?

Thank you for any feedback!

Best,

Rachael

[Laib - CDPHE, Abby]

Hi Rachael, 

While I believe it can be helpful and a good practice to share preliminary results with stakeholders, especially during a formative evaluation to inform rapid cycle improvements - I may err on the side of caution of sharing a few comments from a few interviews conducted that week or month. Qualitatively, you won't know that those one off comments are necessarily themes until more interviews are completed and the thematic analyses have been completed. 

Without knowing anything about your evaluation or data collection timelines.... Perhaps, depending on the number of interviews being conducted on a monthly basis, you could compromise and share qualitative "preliminary results or rolling results" on more of a quarterly basis and be sure to explain that these are only the themes that emerged from the past 3 months, and that you'll be adding to these themes each quarter as more interviews are conducted. Be sure though to give your qualitative analyst enough time to conduct those quarterly interviews and analyze the data (so these may be actually shared the 4th month or month after each end of quarter). 

As far as formats, we've used brief one-two pagers (executive summary style or infographic style), slide decks (include graphs and key bullets/takeaways), and dashboards (although this wouldn't lend itself to qualitative data).

Sincerely,

Abby

--
You received this message because you are subscribed to the Google Groups "Colorado Evaluation Network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to colorado-evaluation...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CABOpQsWapewPS0_%3Du%3D5HTw6UftHDwpAvJhjMFTe7xCf%3D_b_KhA%40mail.gmail.com.

--

Abby Laib

Health Promotion and Chronic Disease Evaluation Unit Supervisor

Health Statistics and Evaluation Branch

4300 Cherry Creek Drive South, Denver, CO 80246

O: 303-691-4035 | C: 920-359-0640 | F: 303.691.7821

abby...@state.co.us | www.colorado.gov/cdphe

Preferred Pronouns: She/Her/Hers

    

[Matthew Goldwasser]

Rachael,

I would like to add just a few small additions to this dialogue. I think sharing preliminary findings depends, in part, on how long your budget says you have to complete the work and where this current request fits within that time span. 

Fundamentally, I think that the evaluation approach and the expectations of your client(s) is an important consideration. It is bundled up with what is your working relationship. By that I mean, if the expectation is for them to be active participants in the evaluation or if the expectation is that the evaluation is something they contracted to be done for and about them. The former opens the door wider for sharing preliminary findings than the latter. My experience has been that oftentimes, even when participants agree to be participants, clients have very limited time to reflect carefully on the formative data. Having said that, I think as long as your team can agree on what to share and then to present findings with the appropriate and unambiguous caveats that it can be good practice to share what you've learned so far and/or what questions--old and new-- you are working out.

I hope something in this is helpful.

Matthew

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CAN7jnTfZXzPJGYEX%2B8zDnpS3xEU1YacW834pNnoXigTLyVyJkQ%40mail.gmail.com.

[Aisha Rios]

I will add a layer on to Rachel‘s feedback. I do come from a research background (anthropology) and have been practicing evaluation since 2014 after I transitioned out of applied research. So, I can speak from multiple perspectives.

I also cannot tell from your email if you have had a conversation with your qual methodologist about their rationale, but if you have not I think you should because there might be a great opportunity to learn from each other. And a great opportunity to build rapport on the team.

 Without knowing their rationale, I can guess based on my own knowledge and experience. I don’t think the concerns about sharing findings before analysis is complete is about the contextual difference between evaluation and research. I believe it’s an ethical stance relevant to people who practice evaluation and conduct research. Sharing preliminary results before all the analysis is complete runs the risk of sharing something that does not actually lineup with the final findings. Even if you carefully explain that preliminary findings are not final when sharing, that doesn’t mean that the information shared isn’t going to be interpreted/perceived as valid and potentially acted on which is a risk. As Rachel pointed out, you want to avoid sharing impressions based on a subset of the data to be collected because you risk sharing biased information based on what is standing out to you or your team at that point in time. 

Did the VHA request preliminary feedback quickly or is that something that you/your team proposed? I ask because I’m curious who is driving this request. If the push is coming from VHA, explaining the ethical concerns could be a great learning opportunity for them.

Hope that is helpful!

Aisha

On Thu, Jan 28, 2021 at 7:24 AM Laib - CDPHE, Abby <abby...@state.co.us> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CAN7jnTfZXzPJGYEX%2B8zDnpS3xEU1YacW834pNnoXigTLyVyJkQ%40mail.gmail.com.

--

Sent from Gmail Mobile

[Natalie De Sole]

Hello!

What a fun question, thank you for asking. Just want to second what Aisha is saying. I've worked on a many 5 year, 3 year, 1 year long projects, and usually we thought about the difference between programming or internal knowledge vs. external reports. As a the third party evaluator we know internal staff are itching to hear more about the key ideas. There was a huge study done at North Caroline U in the education department at some point that showed deep level qualitative analytical analysis did not necessarily show better findings than rapid assessments, because when we report quickly on data we also keep in mind things like tone, and impressions made by how people shared the data. Rarely, do people keep a hold of that important data (though the anthro world Aisha is from is famous for doing it).

Sharing early learnings internally to help people understand impressions is pretty normal from my knowledge, a quick list, easy summary, all in the context that is not VALIDATED data and is to help keep people abreast of context rather than to report to the official and external funders  has been a pretty normal practice for many large foundations, institutions, and nonprofits. 

Good luck! Aisha's a great person to follow up with... I'm sure Abby is too (I just don't know her :).

Warmly,

Natalie

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CAOcER72T_o_pK3Q%3DWVdZp3Cp_7B-TUKqPAHnrcGY5OAy8sAmqw%40mail.gmail.com.

--

Natalie De Sole

(she/her/hers)

Learning Forward Evaluations and Knowledge Facilitation

nat...@rooted-growth.org

https://rooted-growth.org

720.372.6116

[Stephanie V. Chasteen]

I agree with Natalie - sharing these early findings informally is fairly typical. For me, when I share preliminary findings I also change my communication strategies to frame them as preliminary: I use a slide deck, not a written product; I use bullet points and not paragraphs; I use questions and hedge language (What do you think about this idea that is surfacing?) rather than statements (findings indicate that…); I have a discussion, rather than submitting a report. I make the purpose of that discussion to generate ideas and inform me as evaluator about what might be appropriate responses to the evaluation, and then those ideas make their way into the final report. It’s usually a win-win.

Stephanie

............................................................................................................

Stephanie V. Chasteen, Ph.D.  
STEM education and evaluation consulting

chasteenconsulting.com 

stephanie...@icloud.com (note new email)

I will sometimes send emails outside of normal business hours; I do not expect others to do so.

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CANAACuuWNyn89Pkd8_-UfvUSGF03qbkOMXeKgNFe%2Bb2RXJmCEg%40mail.gmail.com.

[Patrick Dorion]

Aisha's last question is a good one--whether or not the VHA requested it. For me, there could be a simpler answer: will the data be used by VHA along the timeframe you are proposing? In my experience (even if they did request it), many clients do not have the capacity to actually put it into use. I suppose that gets back to the ethical piece as well--if they will use it, you want the data to be usable (representative). 

What a great conversation! 

Patrick Dorion

Pronouns: he/him/his learn more

Senior Consultant, Smith & Lehmann Consulting 

(970) 818-9309

pat...@smithlehmann.com 

On Thu, Jan 28, 2021 at 8:10 AM Aisha Rios <aisha....@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CAOcER72T_o_pK3Q%3DWVdZp3Cp_7B-TUKqPAHnrcGY5OAy8sAmqw%40mail.gmail.com.

[Aisha Rios]

Great conversation indeed! So much wisdom and great questions shared :)

------------------------------------------

Aisha Rios, PhD (she/her/hers)

aisha....@gmail.com

404-884-5751

"All of the possibilities for real, lasting, and meaningful change are downstream of community power. Until we demand and organize for power itself—rather than pleading for those who have it to take the actions we’d like—we will never get it. And until we get it, we will always be at the mercy of those who have it. They have shown us over the decades whose side they are on: the side of profit, racial hierarchy, and colonial domination and control. It’s time we chose ours.” - Olúfẹ́mi O. Táíwò, philosopher and organizer

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CAJGDWGvWZSfL7-Gj%3DWpr4mf7%3DAvHFg9OxFdOoGaD6Q3FLi57kg%40mail.gmail.com.

[Mariana Enriquez]

Adding my two cents to the conversation.

• There is a great point already mentioned about who is requesting the findings: are you offering them? Is the project team asking? Why?
• Also important to consider is whether this schedule of delivery was agreed at the time of contract signing, or something that has come up as the project takes place, and what is the motivation for such request
• I believe that weekly updates are a killer. How much are you really learning to make it worth sharing with such frequency? Monthly "findings" may be more appropriate
• Would you have a different opinion if the data were quantitative? What is your rationale for one or the other?
• You mentioned that the analyst wants to wait until the data are analyzed, which makes me think that you are considering sharing "raw data." Not sure this is the intention, but I would not share raw data. I may share bullets with preliminary findings, as someone already suggested, but not raw data
• I would share preliminary findings if what you are "hearing" is something to be used as formative evaluation. Years ago I evaluated a program that was definitely not moving in the expected direction as shown by relevant indicators. At that time I was collecting panel (mixed, qual/quan) data. Because this was such an important issue for the project, I collected (non-scheduled) data to inform the problems faced by the project, and reported those findings within a couple of weeks to the project team, which then implemented changes in the project to mitigate the negative tendency. It was a book definition of formative evaluation
• I recently worked in another project where I'd provide survey feedback to the project team within 8-10 days of data collection, because they were using these findings to adapt the project to the needs of the participants. The team knew the limitations of the data within the context of the full project.

In short, yes, I am pro sharing preliminary findings to be used formatively for the project, but I would be very careful in also providing the context of the findings, making sure to indicate that final findings might be different once all the data are collected and analyzed.

Hope this is helpful. And would love to hear what is your final decision.

Mariana

Mariana Enríquez, Ph.D.

Program Evaluation Consultant

enriquez.mari...@gmail.com

Ph: 303.548.6132

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CAOcER71tDWowSnPFdKQqXMWJfnRSAq%3DdyAhqcccDtQsYCoMCDg%40mail.gmail.com.

[Kerry]

I think my feedback will complement much of what you've already read - note I have a background in participatory/community-based research and also 30 years in eval...

I'm wondering if you could share a "quote of the week" (or month or whatever makes sense given your staff time/capacity and theirs to make use of the info per other folks' comments) as a conversation starter - framed to make it clear that it is not necessarily representative but raises important considerations for the organization. IMHO outliers (either positive or negative - or neither, just coming from a unique perspective/sharing unusual observations) are often at least if not more important than typical responses for an organization to be aware of and to take into account in developing strategy and improving practice - but that it is important that everyone understand that is the case when those are the perspectives represented by the selected quote(s). Again, make it clear that these are not selected because they are typical but because they raise points or reveal experiences the organization may not already be aware of that can inform beneficial conversations while they await your more comprehensive annual reporting...

Feel like I may have said the same thing 3 different ways there (lol!) but hopefully at least one of them is useful. :)

Kerry

kv

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CAOcER71tDWowSnPFdKQqXMWJfnRSAq%3DdyAhqcccDtQsYCoMCDg%40mail.gmail.com.

--

[Rachael Kenney-Reid]

This is such a wonderful conversation. Thank you, everyone, for being so generous with your thoughts and bearing with me as my question blew up your inboxes! I hope it was okay to use the list in this way.

I took ideas away from each and every one of your emails. I would love to continue the conversation with anyone who is interested but maybe we can take it off of the listserv so we don’t bombard folks who aren’t interested? Please email me directly if you would like to continue "noodling" on this and if there is interest I can start an email string or maybe we can have an informal zoom? 

My next step is to have a more targeted conversation with my methodologist about how to move forward. I wanted to get a sense of what other evaluators thought before I broached the subject again and you all have given me a lot of food for thought.

To answer some of your questions:

The partner requested more rapid feedback so I’m trying to balance that with the fact that we don’t have “results” or “themes” yet. They asked that we send “good quotes” (meaning compelling positive quotes about the program). They want to use the quotes as supplemental examples in their status updates to leadership in the context of "here's what one Veteran said." They also want actionable information to make iterative improvements to the program. 

In terms of the schedule of delivery, this project is pretty mushy in that regard. My center has a reputation for being responsive to the needs of our partners (within reason). We try to be flexible and provide products that are useful and actionable. So our schedule is often changing depending on their needs.

We’re only considering qualitative data because the program is off to a really slow start. At this point, only a handful of Veterans a month are seen so the quantitative data won’t tell them much. The interviews are trickling in so we do have some qualitative data.

We would not share raw transcripts, just quotes to show some of the things that are coming up. This would not be framed as findings, but some of you have brought up the good point that even if we tell them that they may interpret it differently.

Abby (and I think others mentioned this, too): That’s a great point that we don’t know the themes yet. Anything we share at this point would be framed as “here is something that we heard from a Veteran” because any themes or findings are still several months out.

Kerry: I think your suggestion of a “quote of the week” (or less frequent in this case) is a neat idea. And stressing that they aren’t selected because they are typical but because they represent an interesting, unique experience.

Lynn, Stephanie: I like your suggestion of doing this in a more conversational manner and not present it in a formal format. That would keep with the messaging that these are not findings.

Natalie: Thank you for sharing all of those resources! They will come in very handy as we move forward.

Aisha: That is a great point about it being a learning opportunity for our partner. I’m thinking back to the “evaluation competence” of Patton’s site visit standards (although we aren’t doing site visits on this project) and teaching our stakeholders about evaluation as part of our role.

Marina: Thank you for sharing that compelling example of how you collected off schedule data to inform a program. Formative work offers so much opportunity to really make a difference in the programs that we are working with.

Patrick: That was a great question about whether they can use the information in a timely manner. It depends on what we hear. It is the government after all, “timely” takes on a whole new meaning J Some things may be able to be changed quickly while others may not.  

Natalie: That is so interesting about the research on reporting out of rapid findings! Do you remember anything else about the report? I tried to look it up without much luck. I was able to attend one of their sessions with Alison Hamilton this summer, they have a lot of great work going on so I'm sure it's an excellent report.

Matthew: That is a compelling question about the relationship with the partner. The evaluation I primarily do in the VHA is particularly interesting because my team is all VHA employees, but we are external to the office that is implementing the program. That said, we’ve worked very closely with our partners and, for better or for worse, the lines are sometimes blurred and we almost become a part of the team that we are doing the evaluation for.

Thank you again for your insights. I'm so glad you indulged in this question with me!

Best,

Rachael

Rachael R. Kenney-Reid, MA, PMP

she/her/hers

978-395-1341 | kenn...@gmail.com | www.linkedin.com/in/rrkenney

On Wed, Jan 27, 2021 at 9:50 PM Rachael Kenney-Reid <kenn...@gmail.com> wrote:

--

[Krystina]

Hi Rachel,

I've been watching this conversation all day.  I'm glad you started it.  As a social psychologist, I just wanted to say that regardless of whether you are an evaluator, a researcher, or a client, there is no way of getting around being a human being.  All of us are biased when we get preliminary results.  The results either excite us or we dismiss them as biased on our owned heuristics.  The "results" you get and offer prematurely will be what they remember.  Be careful with what you share.  People are going to take away what you share and there won't be a way to explain away that it is preliminary.  

Best, Krystina

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CABOpQsUQpL4PFGTv0TNuFrGhmRT-XEzjbhbPpv897F8RS72%3DuA%40mail.gmail.com.

[Natalie De Sole]

Hey Rachael,

I know it's been a while, sorry, this has been in the back of my head but other things took precedence. Rather than the field of education (which is the field from I learned from, and in which I was trained to use rapid assessments) the most published field seems to be the anthro-medical world. Of course, the medical field is very motivated to figure out answers quickly!

Here a free published resource on rapid vs. in-depth qualitative analytic methods; what's also great about it are the amount of citations that exist on other studies which published on this subject so you can dive in deeper on the value of rapid learning: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-019-0853-y 

If you follow 3ie ....  ... they talk a lot about rapid evidence reviews, another way to think about framing things.

Anyways, there are a lot of experts in this group on rapid assessments, but figured I'd follow up.

Happy Spring!

Warmly,

Natalie

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CABUym6d7XYDX%3DPkv_KbGOk8GmNUZvEqT-zH7CYEgtTTiokQ00A%40mail.gmail.com.

[Natalie De Sole]

ps. if it helps, what I like about it is I think many people get trained on deep qual methods stuff from Miles, Huberman, and Saldana... this article directly cites Saldana (one of the most established) as the comparison.

To view this discussion on the web visit https://groups.google.com/d/msgid/colorado-evaluation-network/CANAACusdEtLO2gapdJ%3Dm0-Yfu3xGJkQh3XMxR9egmFXgSbG%2BYA%40mail.gmail.com.