Designing a study on outreach method

Vegan message board for support on vegan related issues and questions.
Topics include philosophy, activism, effective altruism, plant-based nutrition, and diet advice/discussion whether high carb, low carb (eco atkins/vegan keto) or anything in between.
Meat eater vs. Vegan debate welcome, but please keep it within debate topics.
carnap
Anti-Vegan Troll
Posts: 414
Joined: Wed Feb 01, 2017 12:54 pm

Re: Designing a study on outreach method

Post by carnap »

esquizofrenico wrote: Thu Mar 08, 2018 5:43 am The biggest problem I see with this kind of experiment is that you are judging your results from what people are willing to say, this could give a lot of problems. For example, it could be that a harsh approach make a lot of people angry that will say they will consume the same amount of meat or more, while to a few it causes a deep impact and makes them to really take the matter seriously; while a "nice" approach make people like the speaker and sympathetically say they will reduce their meat consumption, while actually not feeling any drive to do it in the real life. In that case this experiment would determine that the "nice" approach is more efficient, wrongly.
That's a good point, generally people won't report that accurately if they think there is any judgement. Hard to do this sort of survey without there being implied judgement. Also while whether someone has "gone vegan" is fairly clear, its not clear when someone has actually reduced meat intake because few people are really adding it up. Many people in the US report that they are "reduce meat" yet industry data shows increased consumption.

What is somewhat amusing is that a good number of self identified "vegetarians" still eat meat, people often use labels loosely. So what you'd really want to do is do a dietary survey that asks concrete questions about foods, asks people to summarize their last 1~2 days of meals, etc.
I'm here to exploit you schmucks into demonstrating the blatant anti-intellectualism in the vegan community and the reality of veganism. But I can do that with any user name.
User avatar
Jebus
Master of the Forum
Posts: 2379
Joined: Fri Oct 03, 2014 2:08 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by Jebus »

carnap wrote: Thu Mar 08, 2018 10:20 pmWithout a control group you couldn't know whether either strategy actually worked, only that one was worse than the other or they were comparable.
Since you would rather use available resources to include a control group (rather than increase the sample size in the experimental groups which could possibly lead to more knowledge about significant demographic differences) I am curious as to what exactly you would do with the control group.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.
Jamie in Chile
Senior Member
Posts: 399
Joined: Tue Apr 11, 2017 7:40 pm
Diet: Vegetarian

Re: Designing a study on outreach method

Post by Jamie in Chile »

Control group, as I said at the outset, is people that filled in surveys but never received any workshops or talks. Their surveys would be different because some questions about the talks would be removed but the questions about the actual amount of meat consumption, whether you define as a vegetarian etc, would be exactly the same questions at the same three points, at the start, after 6 months (for example) and after 2 years (for example).

On balance I think my original suggestion of a control group is the right one with more useful info. For instance instead of knowing which approach was better you might learn if one approach was neutral or even negative, in which case it might be more worth trying to persuade people using that approach to stop.

I think you would survey immediately at the end of the workshop or literally hours or one day afterwards. That would be a closed survey. Everyone would be a sat in a room doing it so everyone on the course would do it, so you would have 90%+ participation.

Then you would do follow up surveys at the 6-month mark which would also be within the school/University framework and part of the course. And again at 2 years as I mentioned in the original post. These would include questions about actual behavior. Like “What % have you reduced your meat consumption by in the last two years since the workshop?.” As well as questions like “When did you last eat meat?” “How many of your meals contains meat per week?” The same questions should be asked immediately before the workshop/talk. The control group comes in handy here because there may be an underlying trend decline of meat consumption per person in developed countries anyway.

The 2-year follow up surveys get a financial incentive (a cash payment) that goes direct to each participant because these will no longer be dealt with as part of the course but by email, people having moved on from their courses. Obviously, you collect email address and mobile numbers during the workshop.

Contamination is a risk but reduces with sample size I think. You might want to make sure the school doesn’t decide to counter your approach by inviting the meat industry to give a talk with yours for even handedness. You would need to explain to the school that please can they do this after all follow up surveys have concluded.

Another risk I can see is that some people will want to report results they know the asker is asking for (human nature) especially if they know the ethics of veganism are right but they are are struggling in practice. So they may report that they are more vegetarian than the reality when being asked say how much meat you eat per week. So, I think you counter this by stating at the beginning of the survey. “Please be honest. We don’t mind if you still eat meat.” And just not including any ethical messaging or any influencing of any sort with the follow up surveys, which need to be neutral. The person who did the original workshops and talks should perhaps not be present at the follow up survey (not even at the follow up survey immediately after their talk), and the respondents perhaps simply fill in the survey and stick it in a box without discussing their answers. Actually you could still ask for feedback and have a focus group type qualitative discussion - perhaps after the surveys are in the box. I think Esquizofrenico has a good point re the initial surveys especially but I think what people are willing to say is less of an issue on private surveys months/years later.
Jamie in Chile
Senior Member
Posts: 399
Joined: Tue Apr 11, 2017 7:40 pm
Diet: Vegetarian

Re: Designing a study on outreach method

Post by Jamie in Chile »

Actually I suppose you would need to do 4 surveys not 3

Survey 1 at day zero
Then immediately after that talk and workshop on day one
Survey 2 on day two
Survey 3 after 6 months
Survey 4 after 2 years

Surveys 1,3 and 4 would contain questions about actual consumption. Survey 2 would not since no-one has changed in the time since survey 1 which is a day or two later

Surveys 1, 2,3 and 4 would contain questions about planned changes, if you plan to eat more vegetarian in the future. This should even be on survey 1 before the talk since that in itself is a form of control.

Survey 2 would contain questions about "how did the talk make you feel?" "Do you think you plan to change as a result?" These questions could also be repeated in surveys 3 and 4.

Keep an eye on the thread as I have thought of 2 possible reason why the survey might not be a good idea (or at least issues that would need to be considered) that I'd like to discuss, but I'll hold off for now so as not to have too many competing threads of argument at once.
User avatar
Jebus
Master of the Forum
Posts: 2379
Joined: Fri Oct 03, 2014 2:08 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by Jebus »

Jamie in Chile wrote: Fri Mar 09, 2018 9:14 amFor instance instead of knowing which approach was better you might learn if one approach was neutral or even negative
You don't need a control group for that. Example.

Item 3 (administered before the talk):

What is your attitude towards veganism in general?:

Very negative
Negative
Neutral (Response Chosen)
Positive
Very positive

Item 3 (administered after the talk):

What is your attitude towards veganism in general?:

Very negative
Negative (response chosen)
Neutral
Positive
Very positive

If this were the predominant reply in both the soft and harsh subject groups, we could conclude that no intervention at all would be preferable. We could make that conclusion without using a control group.
Jamie in Chile wrote: Fri Mar 09, 2018 9:14 amThe control group comes in handy here because there may be an underlying trend decline of meat consumption per person in developed countries anyway.
So what? The results from the study will tell us:
Whether or not a soft approach is more effective than a harsh approach
Whether or not a harsh approach is more effective than a soft approach
Whether or not there is no significant difference between the two
Whether or not doing nothing is better than any intervention

Great if the "trend decline of meat consumption per person in developed countries anyway" but that doesn't affect this study or the strategy we adopt following the survey results.
Jamie in Chile wrote: Fri Mar 09, 2018 9:14 amAnother risk I can see is that some people will want to report results they know the asker is asking for
Yes, but this can be significantly reduced if the study is anonymous and confidential.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.
User avatar
Jebus
Master of the Forum
Posts: 2379
Joined: Fri Oct 03, 2014 2:08 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by Jebus »

Jamie in Chile wrote: Fri Mar 09, 2018 9:20 am Actually I suppose you would need to do 4 surveys not 3

Survey 1 at day zero
Then immediately after that talk and workshop on day one
Survey 2 on day two
Survey 3 after 6 months
Survey 4 after 2 years
Three surveys are enough. Whatever happens between 6 and 24 months will have little to do with the intervention technique.
Jamie in Chile wrote: Fri Mar 09, 2018 9:20 amSurvey 2 would contain questions about "how did the talk make you feel?" "Do you think you plan to change as a result?"
I'm not sure if you are suggesting open-ended questions here. If so, I would strongly recommend against. For statistical purposes, each item response needs to have a numerical value.

Either way, we need to be careful about formulating questions in a manner where the subjects' personal feelings about the presenter may be manifested. For example, the subject may dislike the harsh presenter although the message s/he presented hit home. In this case, a question like "how did the talk make you feel?" could be ambiguous.

Also, as I mentioned before, the questions asked before and after the intervention (as well as the response options) need to be identical.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.
Jamie in Chile
Senior Member
Posts: 399
Joined: Tue Apr 11, 2017 7:40 pm
Diet: Vegetarian

Re: Designing a study on outreach method

Post by Jamie in Chile »

I don't feel strongly enough about the control group to have a long debate about it. The value of the control group would be reduced if you omit the 2-year response and only did a follow up after some months, because you could reasonably assume very little substantial trends over a period of months for a typical population.

I do think behavioural tchanges between 6 and 24 months is worth considering, however. You sow a seed but not enough to get action. Then maybe at the 18 month mark a second thing happens. Perhaps the person's friend goes vegan and instead of thinking "vegan? aren't they just wierdo animal activists" and thinking nothing of it and just thinking your friend is going through a phase you think. "Hey, I remember that talk I heard and I thought at the time it was sort of convincing and I ought to give that some thought and watch that earthlings movie he mentioned and I never did that, hey maybe now I will."

Also:

June 2014 - I went on a trip which including fishing. I decided that I didn't want to fish myself but I would sit in the boat with my friends. The guide handed me a fish one of my friends had caught and a knife and asked me to kill the fish. I had a knife touching against the live fish's head, the knife in my hand, it was a very small fish, all I had to do was exert a slight pressure and split the tiny head, no blood or gore or effort, but it didn't feel right to me, so I passed the guide back the fish. I did not kill it. Later, back at the place we were staying, they bought the cooked fish and I ate it. I realised at that point there was a contradiction that I needed to address at some point.

6 months after that I was still eating fish (and meat). 2 years after that I was vegetarian and mostly vegan. The whole fish incident was a factor, because it prompted me to buy several books relating to the ethics of eating animals, even though for some reason I didn't get round to it until way later.
carnap
Anti-Vegan Troll
Posts: 414
Joined: Wed Feb 01, 2017 12:54 pm

Re: Designing a study on outreach method

Post by carnap »

Jebus wrote: Fri Mar 09, 2018 12:58 pm If this were the predominant reply in both the soft and harsh subject groups, we could conclude that no intervention at all would be preferable. We could make that conclusion without using a control group.
Not necessarily, that assumes that their negative response right after the talk had a persistent (negative) impact on their thinking overtime but that may not be true.

Any topic that is likely to produce cognitive dissonance may result in initially unfavorable responses (the dissonance of conflicting views) that may resolve themselves as the person changes their thinking (resolving the conflicting views). So you need to know how the person processes the information long term, not just their initial reaction after the lecture.

Without longer term follow-up and a control group that study would have very little value.
I'm here to exploit you schmucks into demonstrating the blatant anti-intellectualism in the vegan community and the reality of veganism. But I can do that with any user name.
User avatar
Jebus
Master of the Forum
Posts: 2379
Joined: Fri Oct 03, 2014 2:08 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by Jebus »

carnap wrote: Sat Mar 10, 2018 3:22 pmWithout longer term follow-up and a control group that study would have very little value.
You just explained why long term follow-up is important. No one has yet given a good reason why a control group is necessary. To clarify, I would like someone who is pro control group to clarify exactly what they would do with the control group, i.e. what questions would they be asked? What would be the time interval between questions etc.?
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.
Jamie in Chile
Senior Member
Posts: 399
Joined: Tue Apr 11, 2017 7:40 pm
Diet: Vegetarian

Re: Designing a study on outreach method

Post by Jamie in Chile »

I'm wondering if we've had enough discussion about the control group.....agree to disagree?....relevant points have already been made....

On another topic:

I already said: Schools and universities would mean producing a certain result favouring a certain approach that wouldn't necessarily be applicable to older people.

To expand on that:

One danger I can see with doing it as part of a University course was that if you targeted philosophy students you will get people that are disproportionately open minded and logical and it would be difficult to convince a course on an unrelated subject (Chemistry for instance) to get their students to participate or build this into the curriculum. So perhaps schools would be better since they contain a wide variety of different people. Even a Chemistry course at Uni may contain people who are simply more intelligent and open minded than the average person, so could produce results that disproportionately show a willingness to change amongst the general population.

The same risk might be there with students at a school due to the open minded-ness of youth. But I think targeting 17-year olds at school is the best thing I can think of - and possibly this is the generation that will more likely drive change that activists need to target? There are also so many schools that you could make a list of 50 of them and keep calling them with a slick sales pitch sooner or later one would agree if you pitch it as giving free classes to your students and the teacher can sit in the class just marking work. The results would be more valuable to an organization like Vegan Outreach that targets youngsters at University, but having not included old people would be an issue for the wider applicability of the results.

However it might be possible to carry out the exercise in an old people’s home. Let’s face some of them have time on their hands and I think people would agree to it. So again you make a list of 50 old people’s homes and just keep calling until one agrees to it.

Getting people of the age group 30-50, which is an important age group with substantial power in purchasing and general however, is more tricky to think of a closed environment where the study could be done.

Perhaps call companies and offer free lunches alongside the talk. This does then lead to a question about whether to bring vegan food (possibly biasing the results) or non-vegan food (which might be morally unacceptable to the people carrying out the study). So perhaps if we want to expand the study to different age groups we just pay money to the 20-60 age group which is the age range in offices or other working environments. You go into an office environment and hold the talks at 6pm offering a fee.

Perhaps that's how to do it if you want to get the results to be viable for all age group.s
Post Reply