+1(978)310-4246 credencewriters@gmail.com

Reliability, Validity, Generality, and Integrity
Reliability, Validity, Generality, and Integrity
Program Transcript
DR. STEVEN LITTLE: Hello, and welcome to Week 4 of Psychology 6735, research
methods for behavior analysis. Today, we’re going to be talking about reliability, validity- primarily, reliability and validity. There are some other things on generalizability and
integrity that will be integrated into it. But the focus is on reliability and validity.
Let’s start out by reviewing last week. Last week, we talked about behavioral
assessment, the purposes of behavioral assessment, types of behavioral assessment,
interviewing, observation, behavior rating scales, permanent products. We talked about
criteria when defining behaviors, and we talked about needed characteristics of
Well, this week our focus is on reliability and validity. We’re going to start with talking
about reliability, types of reliability, factors that affect reliability of observations. How do
we increase the accuracy of observations? And how do we calculate agreement when
we’re calculating interobserver agreement in doing reliability checks on the
The second big topic is validity. And we’re going to talk about internal validity, external
validity. Those are the two main types of validity. But also construct validity, and to a
lesser extent, statistical conclusion validity. Because that relates to statistics. We don’t
do much in the way statistics in ABA so I’m just going to cover that relatively
So let’s jump in and start talking about reliability. I actually love this stuff so, hopefully, I
won’t go too much off topic and keep you here longer than necessary. But reliability, first
of all, let’s define reliability. Reliability is a synonym for stability. So reliability refers to
the stability of measurement. That is, is it reproducible? Is it stable?
And we express reliability, especially when we are talking about norm reference tests,
which we use to a lesser extent in ABA, but we may use them. So I’m going to talk
some about that, and I’ll tests, and I will also talk about observations. But we express it
in terms usually of a reliability coefficient. This is just a range of zero. We don’t talk
about negative reliability. Basically, 0 to 1.
Or by what we call a standard error of measurement, which I will define in a little period
of time. A test should not be trusted if the reliability is low. Generally, when you look at a
test, be it a test of autism, be it a general behavior rating scale, whatever that test is, it
should have a reported reliability in every area of reliability of 0.8 or above. That’s
usually the standard that is considered acceptable.
So I’m going to start by talking about reliability with regard to testing, and then specific,
the single case methodology. So let me start by doing test, retest reliability. And this is
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
an index of stability over time. Giving a test once, you’re giving it again. At the two
points in time, how related are the scores on that test?
So you administer the same test to the same people at two different occasions. It’s
usually a short period. Two weeks to a month, in most cases, of norm reference tests. In
the correlation represents the extent to which the score is stable over time.
Now, the key thing on these tests is you don’t want taking the first test to change their
responses the second time they take the test. So that’s the key thing that we were using
in test, retest reliability. That’s assuming nothing has happened in that period of time. If
something has happened, then it throws off that test, retest reliability.
Now, you sometimes, not usually because we usually do observations of behavior as
behavior analysts. But occasionally, you may use a norm reference test in more of a
pre-post type of manner. In that case, you are looking for differences. It’s not the
reliability because you’re trying to see, did they change as a result of the intervention?
So when we talk about test, real test reliability, we are making the assumption that
nothing happened, such as an intervention, between testing one and testing two.
OK. The third type of reliability. I sort of say this. The second type of reliability is with
regard to tests is called alternate form reliability. Sometimes called equivalent form, or
parallel form reliability. And you get it by administering two equivalent tests to the same
people with very little change in the time.
If there’s no error in measurement between those two versions of the test, then
individuals should earn the same score, relatively the same score. So you should get a
high correlation between the two. There are some tests that have alternate forms. Most
of the ones that you would be using don’t. So it’s not a big deal for you, but it’s
something I just wanted to let you know about, you make sure you were aware of it.
The next form of reliability is called internal consistency reliability. Sometimes while one
form of it that is frequently used is split half reliability. And that you have a test. You
divide the test into two equivalent halves.
Usually, they do this by doing odd numbered items and compare that to even numbered
items. So item– let’s just say that there are 20 items on the test. That you have 1, 2, 4,
6, and so on on one side. And then you do 2, 4, 6, 8, and so on on the other side. And
you calculate a correlation between them.
If you get a high enough correlation, it would mean generally speaking that you have
high internal consistency. The test is similar in how they are comparing individuals
across the items within the test. Now, other times you may see when you’re reading
research where you’re looking at tests, probably the most common other type is called
Cronbach’s alpha. Cronbach was back in the ’50s and ’60s was a big test and
measurements person, and he’s the one who developed this. I’m not going to say how
to do it, but it’s basically a statistical calculation looking at intercorrelations between
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
them. Very easy to do when SPSS, but you just put the items in there and you ask it to
calculate alpha. But more difficult if you had to do it by hand.
There’s another thing called Kuder Richardson 20 you may see every once in a while
that also is a statistical version of that. But it’s internal consistency. Is the test consistent
from the beginning of it to the end of it? Is it measuring the same types of things? Are
people responding consistently?
Another thing that you’ll see, especially with norm referenced test is called the standard
error of measurement. Now, this is an ABA research class. This is not when I taught
cognitive assessments or tests and measurements to other people, which I required
people to calculate these things.
But the standard error measurement recognize that no matter what you are doing,
whether you’re using a standardized test, whether you’re doing observations, any type
of measurement, there’s always some uncertainty. There’s always some error
associated with whatever we do. Nothing is 100% perfect.
The error of the SEM is the amount of error usually attached to a person’s score. The
amount of error attached to somebody’s score. So if there is a large standard error, it
means it’s less precise measurement. You’re not as confident in it. There is more error
associated with it.
So the larger the error, the larger the SEM, standard error measurement, the lower the
reliability. For anybody who’s really interested in statistics, it is the standard deviation of
the distribution of error scores. I’m not going to go into more of that. Most of you could
just put it right out of your mind and don’t worry about that.
Again, it’s something I took some of my notes from when I last taught cognitive
assessment, which I did for a number of years in the clinical program here. So you don’t
have to worry about it. But if you really are a statistics nerd, that’s what it is.
The other thing that you will maybe even see more frequently than the standard error
measurement is called the confidence interval. The confidence interval is the range of
scores that contain the probability of including the examinee’s true score. So it’s a range
of scores that have the probability– and you’ll see in a minute– of containing the true
score without error.
So it’s formed on the basis of that standard error that I just talked to you about. The
standard error of measurements, the SM, what I just explained, is the 68% confidence
interval. So if you have an SEM of say, 3, then that score, plus or minus 3, you would be
68% confident that the true score falls within that range.
If you want to be more precise or more conservative, the 90% confidence interval is
1.65 times the SEM. The SEM is the 68% confidence interval. To get to the 90%
confidence interval, you would multiply the SEM by 1.65. If you want to get to the 95%
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
confidence interval, that you’re 95% confident that the score is going to fall in that range,
you would multiply the SEM by 1.96.
And if you really want to be conservative, you really want to be absolutely sure, the 99%
confidence interval– and we don’t go any more stringent than that– the 99% confidence
interval is 2.58 times the SEM. 2.58. Now, you don’t have to know these specific
numbers, but I wanted to get you an idea of it.
Now, in my psychometrics portion of my class when I was teaching cognitive
assessment, or any type of psychoeducational assessment, this is all based on that bell
curve, the normal distribution. And like in intelligence tests, cognitive measures, IQ
tests, whatever you want to call them, has a set mean of 100 and a standard deviation
of 15. So everything is based on that type of distribution.
Well, what I’ve talked about so far is– deals mostly with norm referenced test,
something that we don’t do that much of in ABA. If you also go on to become a licensed
psychologist, and you do diagnosis of autism, then you’ll be more likely to use those
measures. You will probably see these measures in reports that have been written by a
psychologist who may have diagnosed the individual on the autism spectrum.
If you work with individuals with an intellectual disability where they must receive a
measure of intellectual functioning, as well as adaptive behavior, you will definitely see it
there. But we don’t use it as much, so let’s talk about reliability as it applies to single
case methodology, single subject design.
What are we interested in? Well, we are interested in consistency. Now, I’m not talking
about consistency in the child’s behavior. Children’s behavior have variability. We’d like
to have a consistent baseline, but that’s not what we’re talking about reliability,
We are talking about the extent to which observers are observing the behavior, are
consistent. You want to make sure that you are consistent throughout the period of time,
from baseline through intervention and into follow-up phases. That you are consistent in
recognizing the behavior, in recording the behavior.
When we get into observer reliability, we want consistency between two individuals
observing the same child, that they are seeing the behavior at the same times. So that’s
the type of consistency we’re talking about with reliability. Consistency in observation.
I’m going to talk at length now about different aspects of reliability in single subject
design. First is interobserver reliability. Now, if you’re doing just your regular work as a
behavior analyst, you’re probably not going to get into observer reliability. You’re
probably going to trust yourself that you are consistent from observation setting to
observation setting, each time that you are consistent in what you do. And that baseline
is consistently recorded, as well as intervention is consistently recorded.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
But if you want to get something published, if you are doing something– doing– if you
decide to go on and get a PhD, or you’re doing something that is part of this program,
as a student, you’re probably going to have to get a measure of interobserver reliability.
And that means we want to make sure that basically there’s a quality control on the
main person doing the observation. By getting two people to observe the child at the
same time, you want to see how much they agree. Two people observing the same
child at the same time.
And it’s important when you’re doing this, like, let’s just say that you are doing a
momentary time sampling, that you are both on the same schedule of looking to see if
the behavior is there. You both might be accurate, but if you’re not looking at the
behavior at the exact same time, you’re not going to get consistency. So that’s what
we’re looking at primarily as interobserver reliability.
What factors affect the reliability of observations even with yourself? Well, when you’re
doing interobserver reliability calculations, first thing’s reactivity. We talked about this.
And that the simple act of observing may alter the child’s behavior. And that may
change over time.
We talked about that, about being a more participant observer, being familiar in the
environment. The more familiar you are, the less reactivity that you have. But it also
changes over time. And if you go in there and you’re unique or novel stimulus in the
environment when you first start, then your reactivity may change their– it will change
over time and may change the individual’s behavior– the target child’s behavior over
time. Not because of any intervention that happened. Nothing more than they become
used to your presence in the environment.
When you’re using– doing inter [? rate or ?] reliability, interobserver reliability, you get
the same problem. Let’s just say that your group– because usually you don’t have the
inter– the second person in there all the time. So you may be less reactive than the
other person that comes in. So you get to take a lot of different things in consideration.
So you may have changes in behavior when that individual comes into– to get that
second measure. So keep all these things in time– in mind and try to do what you can
to minimize them.
And I’ll talk about some things and I’ll try to minimize it in a little bit. A second aspect is
observer drift. Now, I’ve mentioned this before. And this is– drift is just a cognitive
phenomenon. It’s not specific to anybody. We all have a tendency to do it, in that there-it involves a gradual shift by the observer from the original response definition.
And as a result, the recording becomes inconsistent. OK. So another way we’re saying
is that the– as you go on in observing it, your criteria for the target behavior may
change, not necessarily consciously. When I grade exams, not multiple choice exam,
don’t worry about yours, yours are all being graded by a computer. And the multiple
choice exams are objective. You don’t have to worry about.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
When I’m grading essay exams, I realize that there may be some drift, that the fact that
my grading at the beginning may as I read more and more, my criteria may gradually
change. That’s why if I’m giving an essay test, I also always make sure that I, first of all,
put the pile of tests in some sort of random order. I try to shuffle them up so that I’m not
always looking at somebody’s the first time.
And then for question 1, I’ll start at the beginning and work towards the end. Question 2,
I’ll do it the opposite direction. Then I’ll shuffle them again, and I’ll go back and do the
same types of things. So that if there are any aspects of my lack of objectivity that I am
not consciously aware of, at least it will be distributed among people.
And I’ll be honest, I think I get more lenient as I go on. Because nobody is giving me
exactly what I want. I rarely get people give me on an essay exam giving me exactly
what I want. So I’m more stringent at the beginning because I’m going by what I think
they should be having as I read some and I see that they’re not– nobody is giving me
exactly what I want, I make it more lenient This is– I don’t– right now I don’t give essay
exams so you don’t have to worry about that in here in this course. And if for whatever
reason you got me in another non ABA course, I tried not to give essay exams anymore
because one of the reasons is because of that. Although, I do think I randomize the
error across all the students. But that’s an example of drift.
We’re not consciously aware of it. But recognize that the amount of drift also may vary
from observer to observer. And usually when you bring in someone to do interobserver
agreement, you don’t have them there every single time. You have them there some of
the time. So you may have already done a few observation sessions, and then they
come and you may have drifted a little, and they’re being very stringent on it, which may
lower your interobserver agreement.
So it’s complicated. And it’s things that we have to keep in mind as we go through it. I
hear a lot of times in studies, what you will do is you’ll look at those instances where
there was an agreement, and then the two observers will come back and discuss it and
try to reach a consensus as to what happened. Especially, if you have a video recording
of it, you can do that.
Why don’t we move on and talk about the recording procedure. And basically, this is
very simple. The more complicated the recording system, the more likely there will be
error. The more complicated the recording system. There are some very complicated
recording systems. I was trained in one a number of years ago to do observations in the
home that involved– oh, there must have been 20 to 30 different things that you were
looking for.
And you– because we were looking for ABCs also in there, and you’re trying to find
antecedents and consequences, all within the scope of this one recordings. As I said, I
was trained in it a long time ago. I don’t do it now because it was so complicated. It was- that there had to be error. Because– especially when you’re learning a system like
that. It’s really easy to make errors.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
So the procedure, the more complicated, the more error. But also in our observational
systems– and I tell you, we usually do some sort of time sampling– partial interval,
whole interval, momentary time sampling. And I’ve told you more than once that I tend
to prefer momentary time sampling.
So let’s just talk about that. And another reason why I like it is that momentary time
sampling has been found to be more accurate than either partial or whole interval
systems. So momentary time sampling tends to be more accurate. It’s not a huge
difference, but– and I told you before that you tend to overestimate the frequency of the
behavior with partial interval and underestimate with whole interval. So with time-momentary time sampling is better. You’re less likely– you’re more likely to get a more
accurate observation. You’re also less likely to have error.
So I talked about reactivity. I talked about observer drift. I talked about recording
procedure as factors. While the next one is the location of the observer. And this is
basically the more obtrusive you are, the more likely there will be reactivity. So this goes
back to the reactivity one. But your location, you want to be as unobtrusive as possible.
I think that’s pretty clear. I won’t elaborate on it.
Actually, I will elaborate on it. When I was working in the schools, I even tried to do
some things like observe through the window from outside. Some teacher knew about it.
I wasn’t like a voyeur out there. I was trying not to be reactive and not try– I did have to
do this in one case. My presence in the room kept changing this child’s behavior. There
was no question about it.
There were report after report after report about disruptive behavior by this one child.
And every time I was in the room, I was a discriminative stimulus through behave, I
guess I was. And the kid was fine. So I did it through bushes outside of the window
when I was– I hid in the bushes and I did see the problem behaviors the teacher had
reported over and over again. So your location can matter.
Next comes observer expectancy. Observers, you, me, we may be biased through our
expectations based on a number of different factors. It could be the gender of the target
child. It could be the behaviors of peers around them that give you a comparison effect,
that may alter how you view or behavior. The purpose of the intervention may change it.
There’s also a confirmatory bias. And I go into this in great detail with my clinical
psychologist students, when I was teaching the clinical program, in my school
psychology students when I was teaching the school psychology program. We had to
be so concerned about confirmatory bias. If you’re looking for it, you’re more likely to
see it.
Behavior is that if you weren’t specifically looking for confirmation, I’d say autism
spectrum disorder, you may just see as a normal variant of regular– of average
behavior. But when you’re looking for evidence of autism spectrum disorder, you will
say, hey, that’s evidence of it. So I always go into my students are going to be doing
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
diagnoses, clinical psych students, school psych students, to be so careful not to
conform– to have a confirmatory bias.
I’ve seen people using norm reference tests and give tests until they confirm that you
have enough evidence to support what they want to diagnose. One of my problems with
physicians doing diagnoses in the realm of psychology, mainly being autism spectrum
disorder, ADHD, those both– those being the prime ones, is that they are getting a
referral most likely for that. The parent is coming in and saying, the teacher thinks my
child may be on the autism spectrum. So they’re looking for it, and they’re looking to
confirm that.
And done– they’re not, I don’t think, as objective as we are. I know I’m putting them all
into a category. I am not being objective in that, because I’ve seen too often diagnoses
coming from physicians that we as psychologists would never do. Because there’s not
enough data to support it. But I won’t get into that. Now you see my biases.
So recognize that your expectations may change what you see and how you record
things. I’ve seen more than once– I give a good example. My wife’s dissertation in
which she was recording behavior in an elementary school classroom. And she was
videotaping, and back then there was literally a tape. So she was videotaping the
classroom, so she was able to see exactly what was going on.
And the teacher identified three child– three children who were pretty much obeyed the
rules in the classroom, and three children who didn’t obey the rules in the classroom,
who were less compliant with the rules. Interestingly, the three that the teacher picked
that were compliant were all girls, and the three that were least compliant were all boys.
But because there was a video of it, we saw exactly what happened. And there would
instances that the teacher for the exact same behavior would reinforce the girls and
punish the boys. So she definitely had expectations. Hopefully, we, as behavior
analysts, are less likely to have those expectations and to have it affect us. But with the
first step is to be aware of how possible biases could affect you. And don’t think that
anybody, yourself included, is free of bias. We all have our biases.
It may have been set through our behavioral history, but we all have biases. The best
way to not let them affect your performance is to be aware of what those biases are.
And if you think you can’t overcome your bias, you shouldn’t be working with somebody.
Next, is the predictability of the individual’s responses. Many behaviors occur in a
predictable sequence. You are– it’s very clear that there may be a discriminative
stimulus that may trigger a behavior. If that is reliable– if that as the is reliable, and
that’s kind of what [? nesta ?] is, it reliably predicts a behavior. And it’s clear that it
makes it easier to see the target behavior.
So it’s a predictable sequence. If you have a behavior that is more unpredictable that
doesn’t fall within a sequential pattern, if it’s more difficult as to record because you’re
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
not primed to look for it. I know you should be primed the whole time to look for it. But it
does make it easier if there are signs that something may be coming up. So
predictability of the response.
And the last one here is familiarity with the setting or the subjects, the participants.
When you first enter a new setting, reliability tends to be lower. It increases with practice
in that setting. Similarly, familiarity with the subject population may make the
observation easier.
Now, familiarity with children– working with children on the autism spectrum is going to
make observing children on the autism spectrum easier for you, because you have
practice. You have a level of comfort in working with those individuals. If you were then
working with somebody who is different, whose behaviors are different, you may be less
comfortable in dealing with them at first.
On the flip side, however, familiarity may lead to more expectations, which may
decrease accuracy. So we’re seeing it’s not as easy as it may sound. A lot of different
things that affect us, that affect the reliability of observations. Let me quickly review
what I talked about.
I talked about reactivity, observer drift, the recording procedure, complicated more
errors, the location. Again, that goes through reactivity. You want to be as unobtrusive
as possible. Observer expectancies and biases– we all have them. The predictability of
the individual’s responses and their familiarity with the setting or the participants. All of
those things affect the accuracy of the observation.
So what suggestions can I give to you to increase the accuracy of your observations?
How do we increase the accuracy of observations? Number one, it’s to be well-trained.
This is obvious. Observation is a skill. It needs to be taught systematically.
We talk about issues in these classes, in your VCS, that you’re getting some initial
experience in observing some behaviors. But their relatively short observations settings.
You’re observing some children in classrooms. But you will get more experience when
you do your fieldwork. You’ll get a lot of experience. Observation is such a key element
of being a behavior analyst.
You will develop over and over into better observers. The key thing is to recognize that
it’s a skill to be learned. And you get better with practice. So the first thing is being welltrained. The second is to use an uncomplicated code. I gave you an example before of
a complicated code. You want to keep it simple. The simpler, the better. You may have
heard that term KISS, keep it simple, stupid.
Or to be nicer, keep it super simple. The simpler is usually the better. You don’t want to
have an overly complex observation system. It just leads to more error. The third thing
goes back to operational definitions. You may say, is he ever going to stop talking about
operational definitions? Probably not. Because they are so important. Because if you
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
have a good operational definition, it will reduce ambiguity and it will increase the
accuracy of the observations. You’ll know exactly what you’re looking for. That’s the key.
Next, increase in the accuracy when you are doing interobserver agreements sessions.
You want to avoid interaction between the observers before the observation session.
That doesn’t mean you find somebody that you’d never met before that you don’t
interact with them at all. You don’t say hello to them. You can’t talk to you because
we’re going to be doing an observation session together. No, that’s not what I’m talking
You keep your interactions related to that client to a minimum before the observation
session. You don’t say, hey, I got a few things I want to tell you about this client. If you
want to get an accurate measure and an accurate measure of interobserver agreement,
you can’t say, hey, here’s a little clue– a cue for you. You know, before they, you know,
slap themselves on the head, you always see this little tail where they scratch their ear.
No, you’re don’t– they are doing the observation session. You may have things
described in the operational definition that may help them to see it, but you’re not going
to give them things that you may have come across in your observations. You may not-you don’t want to bias them. Say, OK, today you’re going to be observing Todd.
Todd is one big bully. He is horrible. I hate this kid. Similarly, you won’t going to say,
Todd is the sweetest guy you’ll ever– you’ll ever meet. He’s a wonderful human being.
Because either way, you could be biasing how the other person views Todd. So keep
that in mind. Just keep your interactions with the other observer as free of discussion of
the individual with who you’re going to observe as possible.
The next thing is to use an adaptation period. And that just means you just don’t come
in cold, never been someplace, and then start a formal observation. No, be present in
the environment for at least a little bit of time before that. Don’t just immediately start an
observation. Become more a part of the environment before you do the observation. It’s
going to reduce the reactivity.
If you are recording, and I gave an example of a study that I did at the Child
Development Center at the University of California Riverside when I was on the faculty
there, in which we put cameras in the room. And they said– we had them there, oh,
probably at least a month before we actually turned them on. Because we wanted to get
used to it. It’s the same idea. There’s an adaptation. Observe unobtrusively, of course.
Reducing after that, I’ve talked about that in many different ways. So I have it on lists
multiple times in multiple different ways.
If you are doing research, if possible, having the people who are doing the observations
unaware of the experimental hypotheses. Have your observers blind to the experimental
hypotheses. They don’t know what your hypothesis and what you’re expecting. They
may not know whether they’re doing baseline or intervention observations.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
That is the ideal way that there won’t be any biases related to those hypotheses that
may affect their observation. Not all easy to do. Not done that often. But ideally, that is
something you could do.
Another thing that you see more and more is using recordings. When possible, record
the behavior. It allows greater scrutiny of the behavior. It allows interobserver reliability
to be conducted more easily on time schedules. It allows you to go back and look at
things if you’re not sure. So it does– it can increase the accuracy of the observation by
having that and having the ability just to rewind and look at it again.
And next are systematic and frequent observations.
Use explicit response definitions– again, operational definitions– of the target behavior.
And have good, explicit criteria for scoring the behavior. In most cases, you want to
have short and numerous intervals. You don’t want to have real long intervals.
If you’re observing behavior of something that is infrequently occurring, then you doing
the observations directly may not be the best way to do it. Doing it using recordings or
having somebody in the environment record it for you may be the way that you have to
go. But systematic, frequent observations.
So let me just quickly run through these again. Increasing the accuracy of observations.
Being well trained, first of all. Use an uncomplicated code. Keep it simple. Good
operational definitions. Keep the interactions with other observers to a minimum related
to the target of the observation.
Use an adaptation period. Go in there and be there for a little while before you do the
observation. Be as unobtrusive as possible. If possible, have people doing observations
that are blind to the experimental hypothesis. They’re unaware of the experimental
Record the behavior. That’s another one. Record it using some sort of camera. Real
easy to do nowadays. Much more difficult back when I was starting out, when you had
to use actual tape, and they didn’t last that long. If you wanted to do it for any period of
time, you had to make sure that the tapes were being replaced over and over again. But
we don’t have to worry about that nowadays. And systematic and frequent observation
is best.
Still talking about reliability, we’re going to do measures of IOA. IOA. Interobserver
Agreement. How do we calculate that? I’m sure you have been asking that question to
yourself since the start of this lecture. Boy, he mentions IOA. I am dying to learn how it’s
calculated. It’s going to make my day to finally learn how IOA is calculated.
Hey, now you’re even using the term IOA instead of interobserver agreement. OK, I’m
getting a little bit weird now. So let’s go in and talk about calculating this. Total count
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
IOA. This is the simplest, and this is probably the most common form. This is what I
have used more often than not.
You use total count. And the formula is simply agreement divided by the total number of
intervals time is 100. So agreement, if you agreed– there were no disagreements, then
you’d have 100%. But in most cases, there are not going to be agreement on 100% of
the intervals. So the lower that agreement, the poorer the interobserver reliability. Very
simple to do, though. The total agreement versus the total number of intervals. Now
incidentally, you’re just looking at not the specific intervals, but rather the total times that
the two people agreed.
A more stringent type of interobserver agreement is exact count per interval
interobserver agreement. And that’s the percentage of total intervals in which the two
observers observe the same count. So now you’re taking the number of intervals of
100% interobserver agreements divided by n, the total number of intervals. So now
we’re looking at the specific number that they agreed on.
Did they agree on 80% or whatever? Looking at the specific intervals, not just the total
number of intervals of agreement. That’s exact count. Any of this stuff is confusing, both
the Kazdin book and the Cooper et al. book go into how to calculate these things. I
know with me here just talking to you, you don’t have the opportunity to ask me
questions right back. So if it’s confusing at all, go to your textbooks.
And a third type is total duration IOA. And this is for timing data, when you’re recording
duration of a behavior, as opposed to a momentary time sampling. You’re looking at the
total durations. So you take the shorter duration, play one person, say, half an hour
period. Recorded the behavior is occurring, 25 minutes of it. The second person
recorded 28 minutes of it.
So you take the shorter duration, 25, divide it by 28, and multiply it by 100. That’s when
you’re just looking at total duration interobserver agreement. But it doesn’t stop there.
There’s mean duration per occurrence interobserver agreement.
The total is real easy. You just take– OK, you say it happened 25 minutes out of 30? I
say it happened 28. And that’s it. But mean duration per occurrence IOA if you calculate
the percentage for each observation, and then you calculate a mean for all intervals. So
that means that it’s all like it happened 25 consecutive minutes and didn’t happen the
last five. No, that’s added up over the entire 30 minutes.
So mean duration per occurrence IOA is that each time you measure, this person
measured– OK, this happened 65 seconds. The other person said it happened 60
seconds. So you calculate that. And each time you calculate it, and then you get an
overall mean. So it is a more stringent version of duration interobserver agreement.
And finally is just calculating a Pearson product-moment correlation. Just r, a
correlation. Pearson– but it’s technically Pearson product-moment correlation. And this
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
is just the measure of a linear correlation between the two observers, observer x and
observer y. And you get a range of minus 1, which means every time observer x did it,
observer y didn’t. Every time observer y did it, x didn’t. Very predictable, but they’re
doing it in opposite directions. That’s not going to happen.
2, 1, which is every time observer x recorded it, every time the same was recorded by
observer y, and so on. And that would be flat. That would be a 1 if it was 100%. And it
would be approaching that. And again, using our criteria we talked about with tests, we
usually would be looking for at least 0.8.
So calculating. We talked about total count IOA, exact count, per interval IOA, total
duration IOA, mean duration per occurrence IOA, and calculating a correlation– a
Pearson product-moment correlation. And again, I can see that if you’re not familiar with
this it may be to the point where, huh? What’s all this about? So don’t worry too much.
Validity. So reliability is consistency in many different ways, but consistency. Validity is
basically, do things mean anything? I’ll go through different types of validity, starting with
internal validity. Internal validity reflects if the conducted research or study is sound. Is it
a sound study? How sure are you that when you change the independent variable, that
it resulted in changes to the dependent variable?
In ABA, can we say that the change in behavior is due to our intervention? We have to
have a sound experiment, first of all, before we can do anything else. We have to be
somewhat comfortable that changes in behavior can be attributed to our intervention,
what we did.
We’ll talk about this specifically in how we use your reversal design, ABAB design, or a
multiple baseline design. How we structure things so that we have good internal validity,
that we have confidence that it is our intervention that is changing the individual’s
behavior. So what we are trying to avoid against are confounding variables. Variables
that may contribute to the change in behavior that are not directly related to our
So the more confounding variables you have, the more your internal validity suffers. The
more the integrity of your study suffers. The less confident you are to make any claim
that is your intervention is causing the change in behavior. So you’ll see when we start
talking about specific designs how we design studies so that we reduce the confounds
and we have good internal validity. That we are confident in concluding that any change
in behavior was a function of our intervention. The IV is what affected the DV.
Let’s start by talking about threats to internal validity. Threats to internal validity, starting
out with maturation. Recognize that participants change over time. We all change over
For example, let’s say that you are doing some study on college students. And I know
this is not what we generally do as behavioral analysts, but let’s say we are doing
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
something in college students who’ve gone away to college. Individuals are going to
adjust, and sometimes relatively quickly.
And the problem is there’s differences in maturation. There’s differences in how quickly
one person adjusts to being away from home for the first time, living in a dorm, having
to deal with the responsibilities on their own, not having a parent telling them they need
to study. That type of thing. Different people become adjusted at a different rate.
So if you’re studying individuals, you have to recognize that maturation may threaten
your conclusions, that this is what you’re doing to help them adjust to college life. But
the fact that just being there is doing it by itself. The key thing to remember– the longer
the study, the more this is an issue. In most cases with the individuals with whom we
tend to work as behavior analysts, maturation is not that big an issue.
When we look at things, it’s generally over a period of time. They’ve been consistent in
the behaviors with which we are intervening for a long enough time to recognize that
they’re not likely to change due to just maturation. But it is something in other studies
that you may have to consider.
History. And I don’t mean history going back years and years. I’m talking about history
meaning that did had some unanticipated event occur while the experiment was in
process, while it was progressing, while you were conducting the study? Did these
events affect the dependent variable? And that may confound your conclusions. Did
something happen in the lives of the individuals you were working with that may affect
If it’s something really big that happens– let’s just say their parents were in a car
accident. And God forbid that will ever happen, but let’s just say that happened. It’s a
traumatic event. It’s probably going to mess up enough that I’m blaming the car
accident. But the car accident is going to be a significant enough event in the child’s life
that both parents were hurt, and both parents had changed how they are interacting
with the child.
The parents may be in the hospital, the child had to go live with a grandparent, or
something like that. That’s a major thing. You pretty much can just throw away your
study because that is such a clear confound that may change the behavior of the
individual. We generally don’t have things happen that big. It’s usually smaller things. It’s
things you may not even be aware of, but something to keep an eye out for.
Attrition. This is generally not going to happen in your job. But if you’re doing studies,
you’re doing research, it could. And that means that individuals drop out of your study.
The key thing about that is that individuals don’t always drop out randomly.
You have different conditions. It may be something to do with the conditions that you’re
doing. Maybe one of them puts more requirements on the participants, and I’ll give you
an example from one of my students’ dissertations. She’s actually doing a study on
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
yoga therapy. And what she was interested in is– and she’s doing a single-subject,
single-case design– were there changes on certain dependent measures as a function
of how often individuals received yoga therapy.
So we had to keep it somewhat simple, but she had individuals who were receiving it
once a week, and individuals who were receiving it three times a week. All of these
individuals had certain physiological and emotional issues. They all pretty much had
some sort of chronic pain, usually lower back pain. And they were looking at depression
and anxiety.
One of the things that we thought from the beginning, it came true. There was greater
attrition in one group over the other. It really did affect how she could interpret her study.
And that is, the attrition happened– yes, yeah– in the group that had to go three times a
Started out with three people in each group. She maintained the people that went once
a week. Two of the three dropped out. So her to study, it’s never going to get published.
It’s fine because she’s gone through the procedure that she needs to go through for a
dissertation. But it completely destroyed her results because there was selective
attrition in one of the groups.
Another thought you may have had in a statistics class, and that’s regression towards
the mean. When participants are at extreme levels of any behavior, the tendency is
naturally to regress or move towards the mean. So if someone is at a very extreme level
of a behavior, over time there is a natural tendency to regress towards the mean.
Most of the studies that we’re doing, there’s been a long enough period of time prior to
our intervening with this individual to think– if there’s stable baseline of, let’s say, the
problem behavior that’s high, that it’s been there a while, and that if there is any
regression to the mean, it’s going to be so small it’s not going to have any discernible
effect on our data. But if you’re doing it on a long-term study, yes, there could be
regression towards the mean.
Any type of pre-test that you’re using. Behavior could change as a result of the pre-test.
It’s not because of your intervention. It’s because of what you did to assess them prior
to the intervention. [INAUDIBLE] that’s not a big issue with us in behavior analysis, but it
is possible. It’s something you had to be aware of.
And the instrumentation, or how we do the observations. The method we use to record
behavior can change over time. I talked about that in observation. It also comes true
with validity. If we change in how we’re measuring the behavior, it’s going to affect not
just the reliability, but the validity.
That was one thing I probably should talk about– the relationship between reliability and
validity. Reliability is necessary but not sufficient for validity. What does that mean? It
means that reliability comes first. If you don’t have reliability, don’t even have to
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
consider validity, because you can’t have a valid study that’s not reliable. Because you
have to have that consistency in measurement to start off with.
If we don’t, we don’t know what those numbers mean. Therefore, you’re not going to
have validity. So you have to have reliability. But just because something is reliable
doesn’t mean it’s valid. So keep those things in mind.
So those are things we’re talking about in terms of internal validity. Next comes external
validity. External validity is simply the extent to which the results of any experiment can
be generalized beyond the specific conditions of the experiment to other populations, to
other experiments, to other conditions. To different points in time. How do the results of
this study generalize past the confines of that experiment?
Let’s start out by talking about threats to external validity. Start out by talking about the
sample characteristics, the individuals who are participating in the study. When you
conduct an experiment, you demonstrate an effect with a particular sample.
In single-case methodology, we are doing it with a specific sample. It could be children
on the autism spectrum. It could be intellectual disabilities if you’re working in a regular
education setting in a school. It could be regular education students in a school.
So to what extent can the results of your study be generalized to other people who may
vary in age, race, ethnic background, gender, education, socioeconomic status, or any
other characteristic you can think of? In single-case design, single-subject design, we
generally expand our generalizability by doing different people doing multiple studies,
using the same or similar types of interventions with children of different ages. Children
at various functioning levels. Different levels on the spectrum.
Boys and girls. Children who live in cities versus children who live in rural areas. And so
on. So the cumulative amount of research gives us this external validity.
But in areas other areas of psychology, especially social psychology. So the main
complaint about social psychology is– I remember hearing this back when I was an
undergraduate– is that social psychology is the psychology of the college sophomore.
Could be college freshman, too, but it’s whosever intro psych at that time. Because they
had these subject pools, and people taking intro psych are required to participate in
these studies. So much social psychological research was done with college students.
And the idea that the results, the findings from these college students– I mean, they are
a relatively select sample in both age– we’re assuming intelligence. They’re in college,
so therefore they’re probably a little bit more intelligent the general population. I don’t
know about that, but they could be. Obviously, they have higher education than people
in the general population.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
SES may be higher, especially if it’s a private university. Their experiences may be
different, et cetera. So it’s possible that these may relate to the independent variable of
interest and affect their performance on the dependent variable. So much research.
Social psychology has been done on college students, which has been the greatest
criticism of social psychological research, is this external validity. They try to
compensate for that by having very good internal validity. Very clear, making sure that
there are not other confounds that are affecting them that are possibly controlled off. To
have good consistency. But this still doesn’t generalize past that group.
Now, in clinical and educational research, we frequently evaluate the treatment of
subjects, sometimes from a particular population. A lot of research, for example, has
been done in university lab schools because they’re there. Don’t know if you went to an
undergraduate, to a university, that had a lab school, or you’re familiar with one. But
these are basically schools– elementary schools, mostly– that are affiliated with the
Department of Education at the university. And there’s designed as a place for people to
do research, partly. Also to get practical experiences and things like that.
But are these children who are attending a university lab school really the same as
children in, say, an inner city public school? There could be a lot of difference, because
a lot of times they’re children of professors. So that’s just one example. So the sample
of who we are doing. Can we generalize past the confines of the individuals with whom
we’re working? So those are sample characteristics.
The second threat to external validity are stimulus characteristics. The extent to which
the findings extend the cross stimulus characteristics of the experiment. Characteristics
such as the setting in which it’s conducted. Who the experimenters are. If you’re doing
interviewing, who does the interviewing. Other features of stimuli that the subjects are
It may restrict the generalizability. So the findings may be restricted to a particular
setting, a particular situation, a particular feature of that experiment. Going back to the
lab school that I talked about, the study done in the lab school with great resources and
low teacher-to-student ratios. Does that generalize to an inner city public school? So not
only is it the sample, but it’s the setting. So I use the same example for both as a threat
to external validity.
Just like in social psychology, something done in a laboratory in a psychology building
and on a university campus, does that generalize to real life? To people who are out in
the street and dealing with all sorts of things in their life, and not just whatever that
independent variable is? The next threat to external validity is contextual characteristics.
Like any study that is conducted, be it in ABA, be it in social psych, be it whatever, it’s
embedded into a specific context, such as the time of year, that may limit the
generalizability to other contexts. I’m using time of year as an example but there could
be other contextual factors.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
I’ve done a lot of research in schools and educational environments. If you’re doing an
intervention, say, for homework completion, you may have an entirely different effect in
September, at the beginning of the school year, versus February, in the middle of the
school year. So the context may not generalize to other contexts.
And next is reactivity. Reactivity of the experimental arrangements. And that is– excuse
me– simply if the participant is aware that they are participating in a study, that may
change their behavior. So if you know you’re being studied, and especially if you know
the purpose of that study, you may act in a certain way.
Would the results be the same if they had no idea that they were being studied? We
can’t do that. Ethically, we can’t let people not know that they’re being studied. It’s very
relevant when you’re doing an observation to record a dependent variable.
Now, it doesn’t affect us as much as behavioral analysts because the population with
whom we’re working may not be at such a functioning level that it matters. And plus they
may be in an environment where there are people around all the time. So it’s not going
to have a tremendous effect. But it is something I want you to be aware of. Reactivity of
experimental arrangements.
Next, and I’m almost done with this but stuff, but is multiple treatment interference. Oh,
multiple treatment? What are you talking about? OK, I’ll tell you. In some experimental
designs, subjects are exposed to more than one experimental condition. Like if you’re
doing a multiple baseline across treatments, or simultaneous treatment design. The
effects of one treatment may influence other treatments. So there are factors that there
may be– when using multiple treatments, one may interfere with the other.
Let’s give you another example. Let’s just say you’re doing a group intervention with
students to increase homework completion. And you’re also looking at a behavioral
contract. Improvement may come only at the time of the contract. You may not know if
it’s the contract or it’s the group contingency that’s causing the effect. It’s always a
problem in single-case design when you’re looking at treatments, especially if
individuals have had multiple unsuccessful treatments before your intervention. It is
possible that treatment history may contribute to the lack of success with your
The last threat here is on novelty effects. I always found his interesting. And this refers
to treatment effects that may be due to the intervention– not the specifics of the
intervention. Just the fact that the intervention is new. It’s different. It’s novel. Something
they haven’t experienced before.
If you’re adding reinforcement into an environment that they have been reinforced
before, it’s not so much that they’re being reinforced. It’s the fact that it’s different than
what’s happened before. That’s possible.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
A really interesting study that I’ve talked about for years, done way back in 1978 on
taxicabs. Now, most of you were probably too young to remember that we used to just
have brake lights on cars, right at the back. Where they have the– the same place-they still have brake lights there, but in the tail. On the very tail. Because you always
see the lights also above the trunk, for example, on a car, or an SUV at the top, above
the door.
So the study was done on taxicabs with those brake lights mounted above the trunks.
They were more visible, so it was believed that people would see these brake lights
more, and it would reduce rear end collisions. Now, lo and behold, they did a study. I
believe it was in Philadelphia.
And the results showed that when they put– no cars had them except for these test
cars. They put those things up above the trunk at a higher, more eye level, for brake
lights. And they saw a 54% reduction in rear end collisions in cabs that were equipped
with these lights. So this led the government to mandate these lights. That’s why you
probably don’t even notice them that much anymore, because every car has them.
Every truck has them. They have brake lights that are not just down low, but also up
Well, guess what. You got this reduction in first. When they put those things in there,
you got the reduction. Rear end collisions decreased. And then, over time, they started
increasing and increasing, and they went back to the same levels they were prior to
doing this.
So what was happening was when they were novel, when those lights, being in that
location– I remember when this happened. I’m going, oh– I kept seeing them. Now I
don’t even know that they’re there. But people paid attention to them because they were
novel. When they got used to them, they didn’t pay attention to them, just like they didn’t
pay attention to the brake lights that were lower.
So some things work real well in the beginning, but they don’t work well afterwards.
They work because it’s novel. I’ve noticed this and in working in the schools quite a bit
with school home notes. When you start a school home note, and once the teachers
sending specific information back to the parents, and then the parents do reinforcement
at home, it works incredibly well to start off with. But over time, I don’t know if it’s
because the integrity slips or whatever, but it stops being as effective.
So we’ve been talking about threats to external validity. Talked about sample
characteristics. I had to look back at my notes. Stimulus characteristics, contextual
characteristics, reactivity to experimental arrangements, multiple treatment interference,
and novelty effects.
Next, I want to talk about internal versus external validity. Which is more important?
Most people will argue logically that internal validity is more important than external
validity because you must first have unambiguous findings before you can talk about the
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
generalizability of those findings. But this doesn’t mean you should slight external
validity. It’s also very important, especially in behavioral research.
And the cumulative nature of single-subject methodology research adds to that, not just
your single subject. Single study, sorry. Just remember, first you must show what can
happen, and then you can worry about other aspects. Next, we’re going to be talking
about construct validity.
We’ve talked about internal validity, external validity, and I know you’ve been listening to
me a long time today, so don’t worry. We won’t be talking about the last two types of
validity for nearly as long. Start with construct validity.
Now, internal validity, we focused on the independent variable, the intervention. Is it
responsible for changes in the dependent variable, the behavior? You then asked a
question as to what is the intervention, and why does it cause the change in the
dependent variable. So construct validity refers to the reason behind the behavior
Is the change in the DV– in our case, behavior– is it due to the construct, the
explanation, given by the investigator? So is your change in behavior due to your
intervention as you describe the intervention? If you are using, say, a positive
reinforcement-based intervention, is it the positive reinforcement that is causing the
change in the behavior?
When we talk about tests, we ask, does the test measure what it purports to measure?
When we talk about construct validity, we’re talking about does the intervention, as we
explain it, is that what the cause of any behavior change was? So let’s talk about threats
to construct validity, starting with attention and contact.
Does the attention or contact accorded to the individuals, especially if you have a
control group, differ? Or does attention given to individuals from baseline to intervention
phases, does it differ? So if it’s a group study, do the control group– do they get less
attention than the experimental group? Maybe it’s attention that’s causing it, and not
whatever the independent variable is.
So it may not be even any feature of the contact. It may just be attention. That’s what
we call a placebo effect. Just taking a pill, even if it’s a sugar pill, gets you better. Or just
the attention that goes along with the intervention gets you better.
So in intervention research, it’s important not just to have a treatment or no-treatment
groups, or a baseline and intervention, that doesn’t involve some similarities in attention.
If you have a control group, the control group should get attention that’s for the same
amount of time that the intervention group gets that just is not specific to the
intervention. And you should have similar levels of intervention in baseline as you have
in the intervention phase. Otherwise, it may just be the fact that they’re getting more
attention. So you provide attention not related to the intervention and the baseline.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
The next construct validity threat is single operations and narrow stimulus sampling.
Now, this is closely related to external validity. And I previously mentioned that by
having a very narrow stimulus situation limited that limits the generalizability of the
results to other stimulus situations.
Now, the way in which the idea is operationalized may also affect the construct validity.
Frequently, the intervention is operationalized in such a way that it is inseparable from
teachers thought to be irrelevant. Let’s say you are doing a study comparing treatment
A treatment B. And let’s say you get experts to implement treatment A and treatment B
because you want them to be effective. So you get the expert in treatment A to do that
intervention, and experts in treatment B to do that one.
You could argue that it’s not treatment A treatment B that differ. It may be that the
person administering treatment A differs from the treatment of the person administering
treatment B, and that’s what you’re measuring. So you’re not really testing the construct
between intervention A and intervention B. You’re testing the construct of interventionist
an interventionist B. That’s why, we’re doing that, it’s always better to have the same
person, but you have to make sure that that person is at the same skill level.
Next are experimenter expectancies. I’ve talked about this before. I talked about this in
reliability. It comes a problem sometimes when the person designing the experiment
and the person running the experiment are the same person. It’s possible that their
expectations, their beliefs, their desires for the results that they’re hypothesizing
influence how the participants perform.
Now, they’re usually unintentional. I’m not saying that people are intentionally trying to
skew the results of this study to fit their hypothesis. So they’re usually unintentional. But
that makes it somewhat worse because you don’t know that it’s your expectations, your
beliefs, that are skewing the results in a certain direction. It’s these subtle changes in
experimenter behavior.
It could be tone of voice. It could be a facial expression. It could be posture. It could be
how they follow standardized instructions. All may influence how the subject responds.
So you’re not really testing the intervention. You’re testing– or there are confounds in
there that may explain any change in behavior.
Next are cues in the experimental situation. Now, this involves what we frequently called
demand characteristics. You may have heard that term. Now, these include sources of
influence that may have been conveyed to the participant prior to their arrival in an
Now, we don’t worry about this as much when working with the clientele we tend to work
with as behavior analysts. But they may have heard rumors about an experiment. There
could have been information in subject recruitment or participant recruitment that did it.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
So this is more in social psych and other types of research. It could be done in
interventions [? that ?] work with, say, adults with anxiety disorders or depression. They
know what you’re looking for. They know that the idea is to get better. They want to
please the person. They report getting better. So that’s what I mean. We generally don’t
have this as much in behavioral analytic research, but it’s something I just want you to
be aware of.
See? I didn’t spend that much time on construct validity, and I’m going to spend even
less time on this last one, which is statistical conclusion validity. Because this is when
you’re doing statistics. And we tend to not do statistics, so it’s not that big a deal. So
anyway, this refers to those facets of quantitative evaluation that influence the
conclusions we reach about the experiments. And it’s primarily in group research, and
it’s when you’re running inferential statistics.
And this is basically– you just keep doing analysis until to find something that supports
your hypothesis. You keep doing it until you get the results you want. I mention it
because I want you to be aware of it, but it’s not that big a deal in ABA because we use
single-case methodology.
I’m going to talk later on in evaluating how we evaluate data using single-case
methodology and slightly on group methodology, but mostly in single-case
methodology. So I’m not going to go into it here. Statistical conclusion validity is not that
big a factor because we are measuring behavior. Observable behavior. So let’s call it
that on statistical [INAUDIBLE] validity, and let’s move on to our review.
So you’ve been here a long time this week, and I appreciate it. So this week we talked
about reliability. We talked about types of reliability, factors affecting reliability, of
observations, how do we increase accuracy of observations, and how do we calculate
agreement. Then we went into validity, focusing more on internal and external validity.
But we also talked about construct validity and very briefly on statistical conclusion
Next week we’re going to get into some specifics of single-subject design, single-case
methodology. We’re going to talk about basics of single subject design and we’re going
to go into the first design that we’re really going to go into in detail, which is an ABAB, or
reversal designs.
So I hope you enjoyed this lecture. I know it was long. I hope you found it interesting. I
mean, I find reliability validity stuff very interesting, but I’ve been doing this for a long
time. And it’s something we should always keep in mind.
So again, until we talk again next week, I hope you had a good week. And I hope you all
have good behavior. Now, you all take care. Bye.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity
Reliability, Validity, Generality, and Integrity
Content Attribution
Studio Cutz
Studio Cutz
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
Program Transcript
ANGELEQUE AKIN-LITTLE: Welcome to the podcast, everybody.
ALL (SINGING): Fall is here, here the yell. Back to school, ring the bell. Brand-new
shoes, walking blues. Climb the fence, books and pens. I can tell that we are gonna be
ANGELEQUE AKIN-LITTLE: Sreeroopa, me and you and Steve are already friends.
Yay! Good job, Steven!
ANGELEQUE AKIN-LITTLE: Welcome to the podcast. Hello, everybody. This is Walden
University’s VCS in ABA– that is Verified Course Sequence in Applied Behavior
Analysis, or as we say master’s in psychology with an emphasis in ABA, Applied
Behavior Analysis. We’re very excited today. We did that little twist on that Jack White,
White Stripes song. We’re very excited to have Dr. Sreeroopa Sarkar here with us
today. Welcome, Sreeroopa.
ANGELEQUE AKIN-LITTLE: I am Dr. Angeleque Akin-Little. Joining me is-STEVEN LITTLE: Dr. Steven Little.
ANGELEQUE AKIN-LITTLE: And Dr. Sreeroopa Sarkar has been on the faculty at
Walden University for almost 15 years. She currently serves as Program Director,
Faculty Chair for MS programs in psychology and the MS PhD program in
developmental psychology. She received her doctoral degree in educational psychology
from the University at Albany, State University of New York. We know that place,
STEVEN LITTLE: Yes, we do.
ANGELEQUE AKIN-LITTLE: Before Walden, Dr. Sarkar was an adjunct faculty at the
University at Albany and held a joint appointment there as the assistant director of
graduate studies. Her areas of interest include mental health promotion within school
settings, cross-cultural and international research studies related to mental health and
gender issues, and human development. She’s presented extensively at national and
international conferences. She’s published in many peer-reviewed journals.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
And just as a sign of her humility, she did not list on here– because she sent me this to
read over– she did not send me, she’s actually presenting in Prague this year. And
everybody out there, this is how humble she is. She was named Walden University’s
Administrator of the Year this year. So a friend of this show, friend of this program. This
program would not be in existence without her. Everybody welcome Dr. Sreeroopa
ANGELEQUE AKIN-LITTLE: Sreeroopa, welcome, welcome. So we are very glad to
have you here. Well, so we generally start by just talking about– letting you talk about
your background. And I guess, really, you’re the reason the program is here. And we’ll
get to that talking about it. But something else that Steve and I have concerns are
issues of diversity. So especially the people we choose to whom we talk on this
podcast– and you may know this, ABA is a field that is not always that diverse in terms
of the trainers, the professorship.
STEVEN LITTLE: [INAUDIBLE] behavior analysts out there, too.
ANGELEQUE AKIN-LITTLE: That’s right, that’s right. So we’re also always excited to
have someone who represents diversity, too, like you do, as a woman, as an IndianAmerican woman. So we’d love for you to talk about your background. I know you grew
up in India. That’s right, isn’t it?
SREEROOPA SARKAR: Yes. Correct. That is correct.
ANGELEQUE AKIN-LITTLE: Yeah. So where are you from, and how did you get to
Albany and into ed psych? Just take it away, Sreeroopa. We’ll probably interrupt you.
SREEROOPA SARKAR: Absolutely. Yes. I will get started. And I am a native– yes, I’m
a native Indian, as you both know. And I came to the United States 26 years ago for
pursuing doctoral program. So I am from the eastern part of India, Calcutta, where
Mother Teresa did all her volunteer work. So I’m from that city. And so from when I was
a child, I knew about diversity. Particularly it came in with Missionaries of Charity and
[INAUDIBLE] that always inspired me. And my family was directly involved with
But anyway, I wanted to pursue my PhD in the United States because still now, India
did not have enough emphasis on the area of psychology. So that’s an area that really
intrigued me, although as a different major in history. But I was more in the– I was more
interested, rather, in doing something that has an application in real life, in everyday life.
So I was thinking about psychology, and I wanted to do something with either school
children or the population that worked with schoolchildren.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
ANGELEQUE AKIN-LITTLE: Sure. But did you say your undergrad degree is in history?
SREEROOPA SARKAR: Undergraduate and graduate degree. I had a master’s in
ANGELEQUE AKIN-LITTLE: Only undergraduate. Then I went into teacher ed at San
Francisco State University, so, whatever they are. I don’t think they have a mascot. I
don’t know. But– and your family worked with Mother Teresa’s organization, is that
SREEROOPA SARKAR: Correct. My brother was actively involved with Missionaries of
Charity and-ANGELEQUE AKIN-LITTLE: Oh, wow!
SREEROOPA SARKAR: –one side note that I’m very proud of, on my wedding day,
Missionaries of Charity held a special prayer for our happiness.
ANGELEQUE AKIN-LITTLE: Oh, wow! That’s amazing.
SREEROOPA SARKAR: Yes. Mother Teresa– yes, Mother Teresa personally held that.
So I believe that’s what contributed to my happiness in my marriage life.
ANGELEQUE AKIN-LITTLE: Of course. Yes. Oh, that’s so interesting. But I interrupted
so– we both share a history undergrad. I didn’t pursue a masters. It was more a
masters in education to teach in the San Francisco Bay Area. But all right, so you
pursue psychology. And how did you choose Albany? So cold, so different from India,
SREEROOPA SARKAR: Well, it is. And that was a personal reason. My husband came
to Rochester for his PhD, and then I got admissions into Albany. So he took a transfer
to Albany. So that was kind of a compromise on his part.
ANGELEQUE AKIN-LITTLE: Yes, she’s married to a lovely, lovely guy. We both have
met– well, I only got to meet him once, but he’s– you can just– he’s a lovely guy, so-no.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
SREEROOPA SARKAR: He is. [INAUDIBLE]. So yeah, that’s how I got to Albany. And I
found Dr. [INAUDIBLE], my advisor, and that changed my life, my research life, my
academic life. She has been very– you know how involved she is with international
research. So I got onto her research team, started working on HIV/AIDS prevention
projects, how important those topics were in the mid ’90s. So I started working working-ANGELEQUE AKIN-LITTLE: Oh, yes. And still important.
SREEROOPA SARKAR: Very much, yeah. But so I was working with her for a couple of
years. And then she started her mental health promotion program in Sri Lanka, which I
was actively a part of that project. So we presented, we published, did the research and
And that gave me an idea. I wanted to extend the same model, pretty much extend the
same model in a different context, for India. We have a very similar culture with Sri
Lanka, but we just– I wanted to kind of test the same model in India on the mental
health promotion. And that was my dissertation topic. And how pretty much gender
influences mental health status of women in India. So I did a qualitative research for my
study. I focused on an ethnographic research paradigm.
ANGELEQUE AKIN-LITTLE: Right. That’s right.
SREEROOPA SARKAR: Yeah. And conducted my research, produced my dissertation.
I had a huge population that I worked with, and I believe that generated a very
informational dissertation, particularly in terms of how really gender can impact those
school children’s mental health, particularly their self-esteem, self-efficacy. It also
brought out a very interesting topic which was how much the little girls were sexually
SREEROOPA SARKAR: You know, that came out during those interviews. That was
typically not a focus of my dissertation or doctoral research, but that helped me come up
and start a new research project when I started at Walden actually, and started that
project in 2014 about 15 years after my dissertation. But I did two back-to-back research
studies in India. One was on how Indian women define sexual aggression-ANGELEQUE AKIN-LITTLE: Oh, right, right.
SREEROOPA SARKAR: And how they cope with different types of sexual aggression,
what kind of resources that are available to them, and what are the factors that kind of
encourage the men to act the way they do and molest the women. And so we–
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
ANGELEQUE AKIN-LITTLE: Right. Because if you did– if you look at– sorry, because if
you look at antecedent behaviors like that, what is it– and the aggressors– what
happens in the aggressor’s life to promote sexual aggression at all, you know, that’s an
easier place to send that sort of behavior if you think about it, right?
STEVEN LITTLE: Oh, absolutely.
ANGELEQUE AKIN-LITTLE: But one thing I wanted– and so you worked with Bonnie.
Why did you choose ed psych and not school psych? And maybe that’s a dumb
question. But I just– you know, I want to hear-[INTERPOSING VOICES]
SREEROOPA SARKAR: That was a very interesting question. I was actually– my, what
do we call it, sub-specialization was kind of school psychology, because I worked
mostly with school psychology. But the major reason I didn’t want to go for school
psychology– I was, but I did not want to become a practitioner at that time.
ANGELEQUE AKIN-LITTLE: Yeah. That’s what I figured. But I think you can get
licensed in California as an ed psych doctor, right? Can’t you?
STEVEN LITTLE: –what the criteria are.
ANGELEQUE AKIN-LITTLE: OK, I’m off in the weeds.
SREEROOPA SARKAR: Probably not. Probably not. And honestly, I never wanted to
practice as a school psychologist or as a counselor. I wanted to do more research that
will lead to developing instruments and things like that that are culturally specific. So
that’s what my thesis was on.
ANGELEQUE AKIN-LITTLE: And I know, telling the students listening, that’s exactly
what I thought. So if you go into something like– and even the developmental
psychology program at Walden, you know, you’re looking at probably more– you’re
more of a research track, educational psychology that’s more a research track, as
opposed to school psychology, which is definitely research-based but more practitionerSTEVEN LITTLE: Or behavior analysis.
ANGELEQUE AKIN-LITTLE: Or behavior analysis. Now I know, Steven, you wanted to
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
STEVEN LITTLE: Well, first I’m curious as to how you ended up in Walden.
SREEROOPA SARKAR: Well, that was an interesting story, was that I was actually very
happy with Albany. And I had a joint appointment with school of education as well as
graduate studies, office of graduate studies. And I was almost up for tenure in a year. I
was going to be permanent, but at the same time I got– I saw this in a job opportunity.
And I was thinking, I wanted a little more flexibility in terms of where I can work from,
because particularly for my husband’s job and his career, I thought, if I went for
something more flexible, that would be very helpful. That’s a personal reason.
But on an academic level, I would say I was very interested in seeing the [INAUDIBLE]
of online education. At Albany I just started developing an online course in educational
psychology. And I was going to prepare that course. So we had to travel up north– I
can’t remember, but it’s about an hour north of Albany– to develop the courses.
And I saw this [INAUDIBLE], and I saw, it’s so different because this school, this
particular school, was very motivated and well-prepared to offer online education. They
had the resources that we needed. And they had the platform, the resources. So they
were, I would say, thousands of miles ahead of any traditional schools in 2003. So that’s
what really fascinated me. That captivated my attention, and I saw, this is the future of
education. You know, in traditional schools, we are very limited students.
ANGELEQUE AKIN-LITTLE: No. But I agree with you, too, Steven.
STEVEN LITTLE: As someone who’s been with Walden almost as long as you, I
remember the first time that Angeleque and I met you was at a retreat-ANGELEQUE AKIN-LITTLE: Wait, wait. Do you remember the first time we met? You
SREEROOPA SARKAR: Yes, it’s Tempe, Arizona. I remember.
ANGELEQUE AKIN-LITTLE: Oh, wow! That’s right! [INAUDIBLE]. We thought you
wouldn’t remember. Yeah, yeah, yeah, OK.
STEVEN LITTLE: Yeah, that was right after I was hired and before I actually started
working full time. We were living in Riverside, California, and we drove over to that
retreat. And that was– how things have changed since then.
STEVEN LITTLE: Could you reflect a little bit on how Walden itself has changed since
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
SREEROOPA SARKAR: Right, right. And I remember both of you guys, Steve and
Angeleque, both of you guys emphasized at the meeting– emphasized the importance
of applied behavior and analysis and how much we should focus on that.
ANGELEQUE AKIN-LITTLE: [INAUDIBLE]. I’ll tell you, a lot of good that did. And we’re
going to talk some more about this. And kudos to you, it took you to get ABA at Walden.
It took you to do it. But anyway-SREEROOPA SARKAR: Thank you. Well, I couldn’t have done it– I couldn’t have done
it without Marilyn Powell, our dean a couple years ago, and Steve and you.
ANGELEQUE AKIN-LITTLE: But so I know you were reflecting– sorry to interrupt, you
were reflecting on changes from that Arizona meeting. So-SREEROOPA SARKAR: I believe Steve was saying something about how things
changed. Did you want me to focus on something specific?
STEVEN LITTLE: Well, [INAUDIBLE]. Just how did– I mean, the main thing is Walden
was so much smaller then and has grown in so many different directions. But what are
your thoughts on the development of Walden over the last 14, 15 years?
SREEROOPA SARKAR: I would say– it is such a dynamic institution. That’s what pretty
much kept me here. We all know our workloads are sometimes kind of really huge,
humongous. But still it keeps me going and keeps me motivated just because we have
so many things to do. And we can grow in so many different directions. And people
listen to us. It’s not that we always get what we want in terms of programs, but people
do listen to us. And then if you can justify why you are going for this and how it’s going
to impact the students, the students, the curriculum and the academics– so we
definitely get the resources of our leadership. So I’m very proud of that. Sometimes it
did take longer than we expected, but we ultimately got things done.
And yes, we grew– I don’t even know how to compare it with anything. We grew at a
fast, fast pace over the last 15 years. We made a lot of changes. I started with the MS in
psychology program that had only two specializations when I started in 2005. From that,
it grew to maybe 12 and 13 specializations. Yes. So right now I believe we have five or
six specializations, and we just rolled out the applied behavior analysis specialization.
And I’m really looking forward to the growth of this specialization.
STEVEN LITTLE: Well, one thing– I know that Dr. Sarkar is being somewhat humble in
this, but back– you know, we talked about it when we first had contact with Walden in
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
2006. I can remember at a meeting, a faculty meeting in Hunt Valley, Maryland, in 2007
where we were discussing the possibility of new programs. And I was pushing very
heavily for ABA. And they started some programs that no longer exist.
ANGELEQUE AKIN-LITTLE: He always– he has to put that in. You always have to put
that in.
STEVEN LITTLE: But– and I pushed for it year after year. But it really wasn’t until Dr.
Sarkar really got on and wouldn’t give up that we finally got the program. And so we had
our first students in 2019.
SREEROOPA SARKAR: That’s correct.
ANGELEQUE AKIN-LITTLE: [INAUDIBLE]. So I wanted to go back to one thing. And
then I want to talk about what you were saying, Steve. I just want to point out,
everybody, your husband moved for you at one point, and you did the favor. And was
that when you guys– when you went to Walden, was that when you guys moved to
Iowa? Because I know you spent some time in Iowa. But I just think that’s a lovely-SREEROOPA SARKAR: That’s correct.
ANGELEQUE AKIN-LITTLE: It’s a really lovely story how he said, OK, I’ll move for you,
further your career. And then you said, all right, I’m going to take this job basically for
you so we can further yours. I just think that’s a lovely example of a wonderful
partnership. So you talked about that. But you can also talk about Iowa, because you
know snow like we know snow.
SREEROOPA SARKAR: Yes. Yes, we do. Iowa was wonderful. 11 years we stayed
SREEROOPA SARKAR: I have to say– yes, yes. The landscape was not the best, but
the people were the best. I really loved them. I really loved the people of Iowas. They’re
very friendly. You know, contrary to many of the things that you might have heard about
Iowa or-ANGELEQUE AKIN-LITTLE: Midwest, yeah.
SREEROOPA SARKAR: –people are very friendly, welcoming. We had very good
neighbors and friends there.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
ANGELEQUE AKIN-LITTLE: Oh, very nice. Very nice. And now you’re in Oregon, I
guess [INAUDIBLE]. So it’s gorgeous out there. So you guys– how long have you been
in Oregon? How long have you guys been there now?
SREEROOPA SARKAR: It’s three and a half years.
ANGELEQUE AKIN-LITTLE: How time really goes. It goes really fast. So three and a
half years. Yeah, yeah, yeah. OK, so I did a little segue there, but we’ll go back to-Sreeroopa, why ABA? I mean, you’re not anti-behavioral obviously. But you didn’t
receive or go for specific training in ABA– of which I know I don’t know about that. But
why ABA for you? You know, because you pushed it. You held onto that.
SREEROOPA SARKAR: I’ll start with the real story. You know, with the master’s in
psychology program director, I was constantly getting those applications from different
students, like when they are applying for their certification in different states. So I had
to– I had to sign off on many of those. So that’s what first kind of reminded me like, oh,
Dr. Little did talk about– or rather Dr. Littles did talk about the ABA program, so applied
behavior analysis. So honestly, that’s what really caught my attention initially.
SREEROOPA SARKAR: I saw a lot of our students are going in that direction. Our
program had nothing to do with a specific behavioral analysis curriculum. And then I
heard about the applicability of the ABA and how you can practice it with special
education children as well as the autistic– what is it?
SREEROOPA SARKAR: Autism spectrum.
ANGELEQUE AKIN-LITTLE: That’s it. No, child with autism. That’s exactly right. That’s
exactly right. Yeah, yeah.
SREEROOPA SARKAR: So that really caught my eyes because I can’t remember
[INAUDIBLE] it was one of those CNN or another media story. And I was looking at
those children with autism. And then I was thinking about what we can do to help these
children who need special help. And they were actually talking about behavior analysis
and applied behavior analysis of how the folks with these degrees and [INAUDIBLE]
helping these children.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
So that got me thinking what we can do with that. And I started talking with Dr. Marilyn
Powell. And so we had several discussions, and then we– she was like, yes, we have a
huge [INAUDIBLE] for that, ABA specialist. But we don’t have enough folks really for it.
SREEROOPA SARKAR: So we kind of decided to go in that direction.
ANGELEQUE AKIN-LITTLE: Now, Sreeroopa, in here, I always thought it was because
you like Steve so much that you just wanted– you wanted him to work for you so bad.
SREEROOPA SARKAR: That’s another reason. That’s another reason. Well, honestly,
if Dr. Steve Little weren’t here, I wouldn’t have volunteered to take this job up.
ANGELEQUE AKIN-LITTLE: And remember– and then students should hear,
Sreeroopa still called him Dr. Little [INAUDIBLE].
STEVEN LITTLE: Yeah, please call me Steve. We’re just having a conversation here.
SREEROOPA SARKAR: Well, without Steve and your support, Angeleque-STEVEN LITTLE: Please call him-SREEROOPA SARKAR: [INAUDIBLE].
ANGELEQUE AKIN-LITTLE: Oh, sorry. Please call him Steve.
SREEROOPA SARKAR: Sure thing. Well, so without Steve and your support,
SREEROOPA SARKAR: –I wouldn’t have probably volunteered to take on this job.
ANGELEQUE AKIN-LITTLE: Oh, no. And I like to crack a joke here and there. But so
that was a really– that was so interesting to hear that you’re watching things. And I want
to add to that, because in a lot of our conversations, of course, ABA or– you know,
BCBAs, they’re known for working with the ID population or cognitive impairment,
whatever, and children with autism.
And a lot of things we’ve talked about is expanding that role. Steve and I talk about that
in other conversations we’ve had with people and then in other podcasts. What we
really push is we want to see BCBAs in that regular ed classroom. We think BCBAs-they can change the world. We talk about that a lot, because it seems to fit Walden’s
culture so well, you know, with that social change emphasis or– I’m saying it incorrectly.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
STEVEN LITTLE: Positive social change.
ANGELEQUE AKIN-LITTLE: Positive social change. And that’s what we talk about a lot,
too. So seeing that role even expanded more regular education [INAUDIBLE] decision,
database– it’s kind of like a database for every decision you make. There’s a big
cultural behavioral conference coming up that’s part of ABAI. It’s brand new.
And it starts with the line– you know, Skinner said, behavior analysis can change the
world. And we really focus on that. We believe that so strongly. So it’s interesting. You
were watching kids with autism and saying, how can I change that world? What can I do
at Walden to change the world? And I just think that fits-SREEROOPA SARKAR: You are right.
ANGELEQUE AKIN-LITTLE: And it fits so nicely with everything we’ve really been
talking about.
ANGELEQUE AKIN-LITTLE: We can’t let her go. I want her to– qualitative research.
STEVEN LITTLE: Yeah, you are an expert in qualitative research. And we are going to
be using this podcast in the research design class. Now, there is no lecture in that
research design class on qualitative research. And what we were hoping that you could
do is talk a little bit about how qualitative research could be of benefit to behavior
ANGELEQUE AKIN-LITTLE: And even single case design.
STEVEN LITTLE: Yeah. And can it be incorporated with single case design research.
SREEROOPA SARKAR: Right. Absolutely, absolutely. I mean, as you can know, no
situation, no child is similar to another. There are always differences between children,
between any other individuals. And if we tried to get a cookie cutter model and applied
that to every child, it might not even help in terms of analyzing their behavior and what
we are seeing.
So we are seeing a problem in a child, I am perceiving it from a different perspective.
[INAUDIBLE] child might be having some issues the parents are perceiving in a different
way. So we can use the qualitative design to, you know, even ask the parents, ask the
teachers, ask anybody else, like the caregiver who is particularly with the children
spending a lot of time, they might watch something that I might not be seeing within an
hour or within two hours.
[INAUDIBLE] qualitative research, we can do some formative studies in terms of
understanding their behavior. Like, what’s exactly happening? What is triggering this
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
reaction? How he or she is acting, how he or she is coping, what kind of resources are
available when the child is acting that way. So these are the things that we can pretty
much try to understand, observe, interview different stakeholders to understand exactly- or rather to analyze the behavior that we are trying to rectify. So I say qualitative
research– the single design case might be a wonderful, wonderful approach to
understand problem behavior.
ANGELEQUE AKIN-LITTLE: Oh, yes. And also [INAUDIBLE] where you talk about the
parents’ and teachers’ perception– you know, ideas of social validity come up for us,
that acceptability of any intervention. But perception too, when you talk about
perception, that’s placing that behavior in a sort of context– I like talking about context. I
think it’s important to– we’ve got new students who may have no context to put in some
of these new terms and things we learn. So we try to provide a lot of context too.
But back to parents and teachers, a lot of times when we’re working with parents and
teachers, our main thing sometimes is to decrease the complaining of parents and
teachers. So you and I might say, that child’s behavior is kind of normal for that child.
Maybe our goal is to decrease complaining of parents and teach– we talk about that.
And that has to do– and then on the flip side of that, anything you implement, any
intervention you’re going to implement, it’s not the intervention that doesn’t work
sometimes. It’s the implementation. And that has to do everything that [INAUDIBLE]
ability– basically, we have maybe a different term than what Sreeroopa is calling this-you know, interviewing data collection. Steven, you know, acceptability has to do with
that, how interventions are going to be.
STEVEN LITTLE: And basically how parents and teachers may subjectively view the
ANGELEQUE AKIN-LITTLE: Yes. That’s exactly-STEVEN LITTLE: I’m actually on a dissertation committee. I’ve been on it for a couple of
years. I’m not the chair, so I don’t know exactly-ANGELEQUE AKIN-LITTLE: You better not be the chair.
STEVEN LITTLE: I don’t know the progress, but it’s a qualitative study. I am the content
expert on it because this is somebody who is a behavior analyst who is doing qualitative
research with parents to try to identify obstacles to the implementation of behavior
analytic interventions in the home. So that’s one example that I-ANGELEQUE AKIN-LITTLE: [INAUDIBLE] exactly what Sreeroopa is talking about.
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
STEVEN LITTLE: And also, when behavior analysts say they take a new case, they’re
working, say, in a practice where they’re taking on new cases. And one of the first things
you’re going to do is you’re going to talk to the parents, try to learn about that child. I
think what you’re saying is that in essence you will be doing a small qualitative study
with the parents to try to get as much information about the context of the behavior.
Yes, we may ask them specific questions about how often things happen. We’ll try to
get information on antecedents and other environmental factors.
STEVEN LITTLE: But it is in essence a small qualitative study to go along with the
single subject design qualitative study.
SREEROOPA SARKAR: Exactly. That’s why I refer to it as a formative study so that you
gather all the information about the context before you design the intervention.
ANGELEQUE AKIN-LITTLE: Yeah. I never thought of it like that, Sree, but that’s exactly
what we’re doing. We’re doing a formative– and forgive me if I use words wrong,
incorrectly, a formative qualitative study to then inform our database intervention.
STEVEN LITTLE: Exactly, yeah.
ANGELEQUE AKIN-LITTLE: You’re perfect for the podcast.
SREEROOPA SARKAR: Exactly. Right, and that’s– right now you all the major grants,
they are always asking for the mixed method. So they want [INAUDIBLE] than
quantitative research. So it’s crucial that we understand the ecological concept that
[INAUDIBLE] talked about. So we need to understand, data is important, but data is
important in a certain context. If we miss out on the context, I don’t think we’ll get to the
real root of the problem.
ANGELEQUE AKIN-LITTLE: And that’s something we– oh, yeah.
STEVEN LITTLE: No, I just want to say, I think one of the criticisms that some people
make with single subject design research is also one that people make with qualitative
research, and that’s generalization. And so in that sense, we’re both in the same boat.
And at least in single subject design research, it’s the full body, a number of people
doing similar type of single subject design studies that contribute to the external validity
of the interventions. And I would think something similar would be going on in qualitative
SREEROOPA SARKAR: Correct. One thing we have to remember, when we do a
qualitative study, we will not be looking for generalization. And that’s a very quantitative
concept that we should be able to use to resolve to a larger population. But qualitative
© 2020 Walden University
Reliability, Validity, Generality, and Integrity with Steven Little, PhD,
Angeleque Akin-Little, PhD, and Sreeroopa Sarkar, PhD
looks are a specific situation where an individual is, and we study that individual in that
context. And we are– as I was saying, every organism is different. Every child is
So if we try to use the same method to another child who might be having a whole
different context, it’s not going to work. So what we say in quantitative study is, if we
get– we don’t call it a single design in a qualitative study. But specifically we use case
studies, or we can call it single subject case studies. So it’s pretty similar what we try to
say in terms of external validity. If we took the same individual or same situation and try
to replicate the study, we …
Purchase answer to see full

error: Content is protected !!