Designing a study on outreach method

Vegan message board for support on vegan related issues and questions.
Topics include philosophy, activism, effective altruism, plant-based nutrition, and diet advice/discussion whether high carb, low carb (eco atkins/vegan keto) or anything in between.
Meat eater vs. Vegan debate welcome, but please keep it within debate topics.
inator
Full Member
Posts: 222
Joined: Sat Apr 04, 2015 3:50 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by inator »

Jebus wrote: Tue Mar 13, 2018 11:39 am I have suggested a design where the same questions are asked both before and after the intervention to see if there has been an effect. The baseline would therefore be the results from the first questionnaire.

Do you mean the questions would only be asked immediately after the intervention? If so, you could compare the attitudinal (not behavioral) effect that the intervention has had momentarily. In that case, the "before" answers would act as the baseline.
However, if you want to ask behavioral questions after 6 months, then the "before" answers would be a poor control for the behavioral effect of the intervention. The design wouldn't be able to isolate the effect of the intervention from the other experiences the test subjects have had during those 6 months.
It is entirely possible that someone who claims that they are not interested in veganism at t0 may have reduced their meat consumption at t1 for various other reasons.
Or that someone who was a vegan at t0 has gone back to eating meat at t1 simply because recidivism rates are generally high among vegans, and not necessarily because the intervention has put them off veganism.
carnap
Anti-Vegan Troll
Posts: 414
Joined: Wed Feb 01, 2017 12:54 pm

Re: Designing a study on outreach method

Post by carnap »

Jebus wrote: Tue Mar 13, 2018 11:39 am That would be a poor design and is not at all what I have suggested. I have suggested a design where the same questions are asked both before and after the intervention to see if there has been an effect. The baseline would therefore be the results from the first questionnaire.
As someone else just explained, the results before the lecture aren't the baseline because they don't account for the time that has past since the two surveys. Within some given time period some fraction of people will decide to reduce meat intake, go vegetarian, go vegan, etc and you need to know that rate for the demographic being studied.

This would be like suggesting that in a drug study you don't need a placebo group because you can just test people before they take the drug. But that doesn't work, you need to compare how the disease or condition progresses with and without the drug.
I'm here to exploit you schmucks into demonstrating the blatant anti-intellectualism in the vegan community and the reality of veganism. But I can do that with any user name.
User avatar
Jebus
Master of the Forum
Posts: 2379
Joined: Fri Oct 03, 2014 2:08 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by Jebus »

OK, since no one wants to explain what they would do with the control group I'm gonna take a guess.

Prior to the intervention subjects are given attitudinal and behavioral questions. The exact same questions are given to the control group. Examples of such questions include:

Attitudinal: How likely is it that you will ever become a vegan?
1. Highly unlikely
2. Unlikely
3. Don't know
4. Likely
5. Highly likely

Behavioral: Do you eat meat?

These same questions are asked 12 months after the intervention (to both subjects and control groups).

Let's pretend that both the soft approach and the harsh approach were somewhat effective although the soft approach was significantly more effective. However, the study revealed that although a significant number of people from the "soft approach group" had stopped eating meat even more people from the control group had stopped eating meat during these 12 months.

What did we learn from this?
Does the soft approach help people stop eating meat? Yes.
Does the harsh approach help people stop eating meat? Yes.
Is the soft approach more effective than the harsh approach? Yes.
Is it more effective to do nothing than to use the soft approach? Inconclusive. Even though doing nothing was more effective during these 12 months, there is no indication that these 12 months are indicative of the next 12 months or any future 12 month period?

Hence, the control group was only a waste of resources and added absolutely nothing to the study.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.
User avatar
Jebus
Master of the Forum
Posts: 2379
Joined: Fri Oct 03, 2014 2:08 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by Jebus »

carnap wrote: Wed Mar 14, 2018 10:32 am This would be like suggesting that in a drug study you don't need a placebo group because you can just test people before they take the drug. But that doesn't work, you need to compare how the disease or condition progresses with and without the drug.
That is completely different. In drug studies, researchers have the ability of hiding the independent variable from the subjects (i.e. they don't know if they are taking the placebo or the real thing). In our type of study, it is impossible to successfully hide the independent variable from the subjects.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.
carnap
Anti-Vegan Troll
Posts: 414
Joined: Wed Feb 01, 2017 12:54 pm

Re: Designing a study on outreach method

Post by carnap »

Jebus wrote: Thu Mar 15, 2018 2:45 pm That is completely different. In drug studies, researchers have the ability of hiding the independent variable from the subjects (i.e. they don't know if they are taking the placebo or the real thing). In our type of study, it is impossible to successfully hide the independent variable from the subjects.
Its not completely different, in both cases you have a similar issue in that you need to compare outcomes overtime. Drug studies are typically double-blind studies but that is irrelevant to the logical point being made here, namely, the need for a control group to compare outcomes to. And you could make this a blind study by having the control group listen to a lecture unrelated to veganism, meat, etc.

But whether or not the study is blind, it still needs a control group. Your method of determining the "baseline" is insufficient because it doesn't capture what happens over the time interval of the study.
I'm here to exploit you schmucks into demonstrating the blatant anti-intellectualism in the vegan community and the reality of veganism. But I can do that with any user name.
User avatar
Jebus
Master of the Forum
Posts: 2379
Joined: Fri Oct 03, 2014 2:08 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by Jebus »

carnap wrote: Fri Mar 16, 2018 1:24 pmYour method of determining the "baseline" is insufficient because it doesn't capture what happens over the time interval of the study.
That's true but

a. it tells us whether or not a soft approach is more effective than a harsh approach which is how this discussion started.
b. It tells us whether or not the approach has a positive effect

As mentioned in my previous post, a control group does not solve the time interval issue you have raised, since one can't determine if the control group time interval is representative of a "usual" time interval.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.
carnap
Anti-Vegan Troll
Posts: 414
Joined: Wed Feb 01, 2017 12:54 pm

Re: Designing a study on outreach method

Post by carnap »

Jebus wrote: Fri Mar 16, 2018 2:51 pm That's true but

a. it tells us whether or not a soft approach is more effective than a harsh approach which is how this discussion started.
b. It tells us whether or not the approach has a positive effect

As mentioned in my previous post, a control group does not solve the time interval issue you have raised, since one can't determine if the control group time interval is representative of a "usual" time interval.
a.) was agreed on early in the conversation. Its b.) that is the issue, you cannot determine whether it has a "positive effect" if you don't have a control group to know the baseline interest over the period of time studied. I'm not sure what you're trying to say about the time interval, the time internal should be the same for the control group and the lecture groups. If you follow-up on the lecture groups at 6-months and 12-months then you'd do the same with the control group. No different than what you'd do in a drug study.
I'm here to exploit you schmucks into demonstrating the blatant anti-intellectualism in the vegan community and the reality of veganism. But I can do that with any user name.
User avatar
Jebus
Master of the Forum
Posts: 2379
Joined: Fri Oct 03, 2014 2:08 pm
Diet: Vegan

Re: Designing a study on outreach method

Post by Jebus »

carnap wrote: Sat Mar 17, 2018 2:16 pmyou cannot determine whether it has a "positive effect" if you don't have a control group to know the baseline interest over the period of time studied.
Are you aware that I have suggested that a questionnaire is administered immediately before and after the intervention? Let's pretend subjects are significantly more favorable towards veganism in the second questionnaire and that these attitudes transfer into behavioral changes evidenced in a follow up study. How could you not conclude that this was a "positive effect"? Even if people are turning vegan left and right outside the study, the fact that the intervention had a positive effect still remains.


carnap wrote: Sat Mar 17, 2018 2:16 pmI'm not sure what you're trying to say about the time interval, the time internal should be the same for the control group and the lecture groups. If you follow-up on the lecture groups at 6-months and 12-months then you'd do the same with the control group. No different than what you'd do in a drug study.
Since the subjects won't be sequestered during six months, whatever forces affect the control group should also affect the experimental groups. Let's say these forces are favorable to veganism. Would you expect the experimental groups to have even more positive attitudes and behaviors towards veganism than they did right after the intervention, or would these forces just merge into the improvements from the post intervention questionnaire? How could you possibly measure this?

Again, I would be very curious as to how you would design the study. Let's say subjects are given the questionnaire one hour before and one hour after the intervention and the intervention lasts one hour. Are you suggesting that there is a control group that is given the same questionnaire at the same time interval (3 hours) except they don't get to take part of any intervention? Just seems ridiculous.
How to become vegan in 4.5 hours:
1.Watch Forks over Knives (Health)
2.Watch Cowspiracy (Environment)
3. Watch Earthlings (Ethics)
Congratulations, unless you are a complete idiot you are now a vegan.
carnap
Anti-Vegan Troll
Posts: 414
Joined: Wed Feb 01, 2017 12:54 pm

Re: Designing a study on outreach method

Post by carnap »

Jebus wrote: Sat Mar 17, 2018 4:38 pm Are you aware that I have suggested that a questionnaire is administered immediately before and after the intervention? Let's pretend subjects are significantly more favorable towards veganism in the second questionnaire and that these attitudes transfer into behavioral changes evidenced in a follow up study. How could you not conclude that this was a "positive effect"?
A survey right after the study doesn't tell you how the lecture impacts their behavior, to do that you need a follow-up study and then you have issues with controls. If the survey after the lecture is better you can claim a "positive effect" in the senses that sentiment was better on the survey but you cannot make any claims about a "positive effect" in their behavior which is what I've been discussing.


Jebus wrote: Sat Mar 17, 2018 4:38 pm Since the subjects won't be sequestered during six months, whatever forces affect the control group should also affect the experimental groups. Let's say these forces are favorable to veganism. Would you expect the experimental groups to have even more positive attitudes and behaviors towards veganism than they did right after the intervention, or would these forces just merge into the improvements from the post intervention questionnaire? How could you possibly measure this?
Right, the forces would impact both groups which is just the point. If there is a difference then you know the difference is likely do to the lecture and not the "outside forces" since those are roughly the same for both groups. The same holds true for drug studies.

In terms of the control group, I don't think its important that they view a mock lecture. You just need to define a group and take surveys of them within the same time frame as the lecture groups.
I'm here to exploit you schmucks into demonstrating the blatant anti-intellectualism in the vegan community and the reality of veganism. But I can do that with any user name.
User avatar
brimstoneSalad
neither stone nor salad
Posts: 10273
Joined: Wed May 28, 2014 9:20 am
Diet: Vegan

Re: Designing a study on outreach method

Post by brimstoneSalad »

carnap wrote: Wed Mar 14, 2018 10:32 am This would be like suggesting that in a drug study you don't need a placebo group because you can just test people before they take the drug. But that doesn't work, you need to compare how the disease or condition progresses with and without the drug.
There's a good argument to be made that drug studies should not use placebo groups if enough other studies have been done with placebo groups and the progression of the disease is well known. But this is mostly because it's unethical to leave people without treatment... also, people have a tendency to pool their pills outside the study so everybody gets at least some active ingredient. Now studies are more often done comparing treatments for well studied diseases.

But for veganism there are no serious ethical issues of leaving these students in the control without treatment, and the number of people out of these groups who would have converted or reduced otherwise isn't known, and we can't extrapolate from anything else because we don't have good data on that, so in this case I think carnap is right.

A control group is important. Hell, even without the lecture a control group would be valuable to compare to the efficacy of other interventions on similar populations without controls because we'd have some idea of how many would have gone vegan otherwise.

The lecture a vs lecture b would have some value, but only comparative value. If you're certainly going to give a lecture, it'll tell you which kind... but maybe a lecture isn't the best outreach approach period. We'd be able to determine that with controls which would let us compare lectures, leaflets, videos, etc.

For example, with controls:

Study on lectures: A, B, and control
Study on leaflets: A, B, and control
Study on internet videos, A, B, and control

Only nine groups.
And if the population studied was very similar, a mere seven groups would work (one control among all of them).

Now we can compare leaflets, lectures, and internet videos without more studies.

Otherwise, we would need:

Study on lectures: A vs B
Study on leaflets: A vs B
Study on lectures vs. leaflets: A vs B
Study on lectures vs. internet videos: A vs B
Study on leaflets vs. internet videos: A vs B

This requires ten groups. Just one control on that population could have saved a lot of research.
And the requirement for more study groups grows exponentially as you evaluate more strategies.

Controls seem like a waste of time, but they're very valuable.
Post Reply