Designing a study on outreach method

Vegan message board for support on vegan related issues and questions.
Topics include philosophy, activism, effective altruism, plant-based nutrition, and diet advice/discussion whether high carb, low carb (eco atkins/vegan keto) or anything in between.
Meat eater vs. Vegan debate welcome, but please keep it within debate topics.
carnap
Senior Member
Posts: 340
Joined: Wed Feb 01, 2017 12:54 pm
Religion: Other

Re: Designing a study on outreach method

Post by carnap » Mon Mar 12, 2018 3:22 pm

Jebus wrote:
Sun Mar 11, 2018 8:23 am
You just explained why long term follow-up is important. No one has yet given a good reason why a control group is necessary. To clarify, I would like someone who is pro control group to clarify exactly what they would do with the control group, i.e. what questions would they be asked? What would be the time interval between questions etc.?
That as already discussed, you need a control group to know whether either of the groups were effective at all. You can compare them without a control group but you cannot know whether they are effective compared to "doing nothing". You need to know the baseline, some percent of people are going to report eating less meat, going vegan, etc without viewing either lecture. You need to know these baseline rates to know whether the lecture improved or worsen them.

Also just surveying people's impressions on veganism won't tell you much anyways, you won't to know whether it impacted people's behavior. For example after a lecture given by a Jehovah Witness people may have a more positive view of them...but they may be no more likely to join the religion or agree with some aspects of it.

carnap
Senior Member
Posts: 340
Joined: Wed Feb 01, 2017 12:54 pm
Religion: Other

Post by carnap » Mon Mar 12, 2018 3:39 pm

Jamie in Chile wrote:
Sun Mar 11, 2018 10:33 am
Getting people of the age group 30-50, which is an important age group with substantial power in purchasing and general however, is more tricky to think of a closed environment where the study could be done.
People study this age group all the time, what you want is a random sample of people and there are methods for doing this. A college campus provides a really poor random sample, the college population is obviously not representative of the general population even within the 18~25 age cohort. Going into an "office environment" would also be a really poor random sample.

Honestly...just grabbing people randomly (ask every 5th person, etc) from the street and giving them $5 to watch a short video and then giving them another $5 if they follow-up via online or the phone would give you a better sample than going to colleges, offices, etc. This strategy would suffer from selection bias but so would going to a college, but this sample would be more random.

There is plenty of resources and money in the vegan community to do research, the reason why its not done is that very few in the vegan community seem to have any interest in confirming their pet theories. There is actually little incentive to do it, most of the money in the vegan community is in various non-profits and each is based on some pet theory about activism, animal rights, etc and none have a financial incentive put their theories to the test. If they were wrong it will negatively impact their funding. Actually it was pretty funny.....a bit ago Mercy for Animals did fund a study to study the effectiveness of showing graphic videos, etc to people...guess what it found? That the strategy was actually negative. What did Mercy for Animals do? Made a bunch of excuses about the study and ignored the results....despite the fact that they funded it and were involved.

For-profit businesses are likely to be the group that is going to conduct good research but its going to be private. They need an accurate understanding of the market if they are going to succeed. Non-profits, on the other hand, succeed by merely convincing people to give money to them regardless of what they do actually works.

User avatar
Jebus
Master of the Forum
Posts: 1781
Joined: Fri Oct 03, 2014 2:08 pm
Religion: None (Atheist)
Diet: Vegan

Post by Jebus » Mon Mar 12, 2018 4:17 pm

carnap wrote:
Mon Mar 12, 2018 3:22 pm
That as already discussed, you need a control group to know whether either of the groups were effective at all.
Whether or not they are effective can be seen by comparing the before and after responses and checking any effect with a follow up behavioral study. No control group needed for that.
carnap wrote:
Mon Mar 12, 2018 3:22 pm
You can compare them without a control group but you cannot know whether they are effective compared to "doing nothing".


I think the suggestion that doing nothing could be better than an intervention that has proven to have positive results is a bit ridiculous. But OK, let's assume there are simultaneous forces that compel people to eat less meat. Such forces would not in any way relate to the study results. In other words, unless the study subjects are sequestered from the outside world, any such forces would not subtract (and likely not even replace) any forces brought on by the intervention. We could live in a world where people are turning vegan left and right but whichever is deemed the most effective intervention method would still be effective in moving people towards veganism.
carnap wrote:
Mon Mar 12, 2018 3:22 pm
Also just surveying people's impressions on veganism won't tell you much anyways, you won't to know whether it impacted people's behavior.
Yes, which is why we need a behavioral survey some time after the intervention. I think everyone is in agreement on that.

I feel like this discussion isn't moving anywhere with the same arguments being repeated over and over. A few people in this thread have expressed a need for a control group but, as of now, no one has explained exactly what they would do with the control group.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.

Jamie in Chile
Full Member
Posts: 197
Joined: Tue Apr 11, 2017 7:40 pm
Religion: None (Atheist)
Diet: Vegetarian

Post by Jamie in Chile » Mon Mar 12, 2018 7:18 pm

The office, the school, the University, old people's home, none of them on their own are representative of soceity, but all combined together may be. Office could include some other working environments like factories, warehouses but offices are likely easy to stage (big rooms with tables or space for a talk and a large audience). So you either combine together a number of different environments or you conclude that the results are only valid in relation to that type of people.

I think there is a disadvantage to stopping people in the street, you can't get people to do the long workshop or longer talk, it is much harder to get the follow up, and more financial incentive is needed. Also, a 5-minute video (the realistic amount someone might agree to watch if stopped in the street - maybe 20 with decent financial incentive) might not be remembered 6 months or 2 years later so the behavioural tracking is going to be difficult. A short video is not enough to change behaviour - the effectivess of the video would be lost in the noise -e.g. if it reduced meat eating by an average of 3% but the margin for error was 10%.

I do agree it has the randomness advantage though.

Jamie in Chile
Full Member
Posts: 197
Joined: Tue Apr 11, 2017 7:40 pm
Religion: None (Atheist)
Diet: Vegetarian

Post by Jamie in Chile » Mon Mar 12, 2018 7:28 pm

carnap wrote:
Mon Mar 12, 2018 3:39 pm
Jamie in Chile wrote:
Sun Mar 11, 2018 10:33 am
Getting people of the age group 30-50, which is an important age group with substantial power in purchasing and general however, is more tricky to think of a closed environment where the study could be done.
There is plenty of resources and money in the vegan community to do research, the reason why its not done is that very few in the vegan community seem to have any interest in confirming their pet theories. There is actually little incentive to do it, most of the money in the vegan community is in various non-profits and each is based on some pet theory about activism, animal rights, etc and none have a financial incentive put their theories to the test. If they were wrong it will negatively impact their funding. Actually it was pretty funny.....a bit ago Mercy for Animals did fund a study to study the effectiveness of showing graphic videos, etc to people...guess what it found? That the strategy was actually negative. What did Mercy for Animals do? Made a bunch of excuses about the study and ignored the results....despite the fact that they funded it and were involved.

For-profit businesses are likely to be the group that is going to conduct good research but its going to be private. They need an accurate understanding of the market if they are going to succeed. Non-profits, on the other hand, succeed by merely convincing people to give money to them regardless of what they do actually works.
I think you make some good points here. This is what I was referring to earlier actually when I said I wanted to make some other points later on. I was going to say you would want to try and ask vegan activists in advance if they would change if the studies suggested they should. I mean, do you think if the results suggestive a more soft approach rather than the direct one of say Gary Francione or Gary Yourofsky (OK, I know he retired but just making a general point) and they were told this, would they change their approach? Probably not. They'd probably have some excuse or other argument or challenge the results. However, I do think if the results were put out there if they were rigorous enough, and the findings were clear, they would steadily cause change over time with new and existing advocates.

I think my proposed study really tests style of approach rather than exactly which approach - for example if leafleting ogranization Vegan Outreach became a funder of it, they would not be at risk of the results telling them that leafleting doesn't work. Instead, the results would guide them about what type of message to put on the leaflets.

I think the softly softly activists and the in your face activists both do the approach that both fits their personality, and then they retrospectively try to justify it, so that is more of an issue.

carnap
Senior Member
Posts: 340
Joined: Wed Feb 01, 2017 12:54 pm
Religion: Other

Post by carnap » Mon Mar 12, 2018 11:40 pm

Jebus wrote:
Mon Mar 12, 2018 4:17 pm
Whether or not they are effective can be seen by comparing the before and after responses and checking any effect with a follow up behavioral study. No control group needed for that.
You cannot see that without a control group, you need to know what the baseline if you're going to say there was an improvement. If you randomly selected a group of people some of them over 12 months would decide to eat less meat, become vegetarian, etc. You need to know those baseline rates so you can compare them to the rates from the groups that saw the lectures.
Jebus wrote:
Mon Mar 12, 2018 4:17 pm
I think the suggestion that doing nothing could be better than an intervention that has proven to have positive results is a bit ridiculous.
And how has it "proven" to have positive results? That is just the issue, without a control group you cannot demonstrate that the intervention has "positive results". You can only show that one lecture was better than the other.
Jebus wrote:
Mon Mar 12, 2018 4:17 pm
But OK, let's assume there are simultaneous forces that compel people to eat less meat. Such forces would not in any way relate to the study results.
There are such "simultaneous forces", you're not isolating people from their social environment. And these forces would relate to the study because they would create a baseline for interest in reducing meat, going vegan, etc.

If you took a random sample of 10,000 and surveyed them once and then again in 12 months there would be some percent that would go vegetarian, vegan, reduce meat or make some other dietary change. You need to know those rates to claim that some intervention has "positive results" since the intervention would have to increase the percentages for you to claim its effective.

What you would do with the control group (how you would create, etc) is a different topic than whether or not you need one. I'm just discussing the need for a control group, its a purely statistical/logical point.

User avatar
Jebus
Master of the Forum
Posts: 1781
Joined: Fri Oct 03, 2014 2:08 pm
Religion: None (Atheist)
Diet: Vegan

Post by Jebus » Tue Mar 13, 2018 1:34 am

carnap wrote:
Mon Mar 12, 2018 11:40 pm
without a control group you cannot demonstrate that the intervention has "positive results". You can only show that one lecture was better than the other.
That's not true and I have explained why several times already.

Are you of the opinion that every type of survey needs a control group? If not, what would be the exception.

I am still curious as to how you would design the study with a control group.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.

carnap
Senior Member
Posts: 340
Joined: Wed Feb 01, 2017 12:54 pm
Religion: Other

Post by carnap » Tue Mar 13, 2018 3:25 am

Jamie in Chile wrote:
Mon Mar 12, 2018 7:28 pm
However, I do think if the results were put out there if they were rigorous enough, and the findings were clear, they would steadily cause change over time with new and existing advocates.
I think the key here would be "new" advocates, people are slow to change their mind and organizations often have financial incentives to maintain the status quo. Established advocates would likely find reasons/excuses to ignore the research and continue doing what they've always been doing.

A big problem with advocacy is that its not clear when you're being effective or not. So it takes a good deal of effort both conceptually and analytically to figure out how to track outcomes and to do it in mathematically/logically sound way. There just doesn't seem to be much appetite for this sort of analytic work with advocates as a whole but especially vegan advocates.
Jamie in Chile wrote:
Mon Mar 12, 2018 7:28 pm
I think the softly softly activists and the in your face activists both do the approach that both fits their personality, and then they retrospectively try to justify it, so that is more of an issue.
I think that just hints at advocacy being a form of self-aggrandizing behavior. Also while personality type is likely to be a factor, people are also self selecting into various entrenched organizations that focus on one approach or another. So an advocates personality type may make them focus on one set of groups rather than another once they join and identify with one group they will become more dogmatic.

carnap
Senior Member
Posts: 340
Joined: Wed Feb 01, 2017 12:54 pm
Religion: Other

Post by carnap » Tue Mar 13, 2018 3:37 am

Jebus wrote:
Tue Mar 13, 2018 1:34 am
That's not true and I have explained why several times already.

Are you of the opinion that every type of survey needs a control group? If not, what would be the exception.

I am still curious as to how you would design the study with a control group.
I've not seen you explain "several times", again, how can you know a method is effective if you don't know the baseline figures for the demographic in question? What exactly would the effectiveness be based on? Let's say that the survey found that 8% of one group and 10% of the other group reported eating less meat 12 months after the lecture how can you make any conclusion from that data other than the second group was more favorable? To say that the second strategy was effective in general you'd need to know how many people would have reported eating less meat within 12 months without seeing either lecture.

Not every survey based study needs a control group, it entirely depends what you're trying to conclude from the study. If you're trying to figure out "which strategy is better" then you need no control. But if you want to know whether the strategies are effective than you need a control.

User avatar
Jebus
Master of the Forum
Posts: 1781
Joined: Fri Oct 03, 2014 2:08 pm
Religion: None (Atheist)
Diet: Vegan

Post by Jebus » Tue Mar 13, 2018 11:39 am

carnap wrote:
Tue Mar 13, 2018 3:37 am
What exactly would the effectiveness be based on?

Let's say that the survey found that 8% of one group and 10% of the other group reported eating less meat 12 months after the lecture how can you make any conclusion from that data other than the second group was more favorable? To say that the second strategy was effective in general you'd need to know how many people would have reported eating less meat within 12 months without seeing either lecture.


Perhaps you are assuming a study design where two groups are tested after the interventions and then conclusions are drawn by comparing the results from the two groups. That would be a poor design and is not at all what I have suggested. I have suggested a design where the same questions are asked both before and after the intervention to see if there has been an effect. The baseline would therefore be the results from the first questionnaire.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests