10.12 - Intro Review & Good Measurement Pt 2

Can you hear us, Rick?

Yeah, I can now. I forgot

I didn't have on my headphones.

I thought it might
be helpful to go

over some of the assignments
that are due next week.

Can everyone see
the Canvas page?

Yes. Great.

I opened up Module 4,

which will include
the introduction,

and I just wanted to give you

a brief overview of that today.

I have a video

I recorded for another
research methods class,

and I can put that up as well,

but we'll be talking
about it today.

That peer review is not
happening. Here we go.

There's an introduction
template that I wrote that

could help you write the paper.

I also want you to check out
the actual submission link.

Within the submission link,

you'll get more information

about how to write
the introduction.

But I think the most
important component

of this will be to take
a look at the rubric.

Because when you take
a look at the rubric,

you'll be able to tell what

I'm placing most of
the attention on.

This whole thing is out of 50,

but in reality,

the introduction will
only be worth 25 points,

so you have to divide it by two.

As you can see,

just by looking at
the point values,

what really matters is

the main body of the intro,
so the literature review.

Most of you are
already working on

that with your Assignment 1B,

looking for different
articles that might be

relevant to your
topic of interest.

But let's go through
the opening paragraph.

First, you have to put a title

before your first paragraph.

That's half a point.
Should I make it

bigger or is it too small?

A little bigger would be better.

I'll do that. Give me a second.

How about there?

Is that good? Title before

the first paragraph
is half a point,

a catchy opening hook to
grab the reader's attention.

Hailey, what's your topic?

It's about sexism in

the workplace for women in
higher positions of power.

It could be like a catchy hook.

It could be cheesy right
now. It doesn't matter.

[LAUGHTER] This is
going to be really bad,

but I just thought of it

like imagine you're
a woman at work

and one of your male colleagues

smacks your butt on your way in.

Though this is less likely to
be a reality for women now.

This is really common, like
in the '50s and the '60s.

Although overt sexual harassment

is less likely to happen now
to women in the workplace,

sexual harassment is
still an issue for women,

and then you can keep talking.

If I read that, I'd
be like, my God, no.

I want to know what happens
to this guy if he smacked

my butt. Something like that.

State the big picture or
problem and its importance.

Although sexual harassment might

not be as overt
as it was before,

women in the workplace

still experience
sexual harassment.

That is the importance
in the big picture.

Frames the context of the
story to be discussed.

It depends on what exactly
you'll be looking at.

Hailey, what exactly
will you be looking at?

Like perceptions of sexual
harassment or the behaviors?

I really want to look at how it

affects the women and

their ability to continue
on in their career choice.

Got it. Maybe you'll

recruit women and you'll
have them recall a time

when they've experienced
sexual harassment

or imagine a scenario
where they experienced

sexual harassment and then

assess the likelihood
that they'd

be able to continue
in their job,

that they'd want to persist,

that they feel
like they're safe,

that they could seek mentors
and so on and so forth.

When you're framing the context

of the story to be discussed,

you're going to talk about
what your interest is.

Then you'll provide a brief
outline of the paper.

In this research paper,

I will be discussing
historical articles

on women's experiences of

sexual harassment
in the workplace,

the types of sexual harassment

that are typically
likely to occur,

other people's perceptions of

sexual harassment
in the workplace,

and finally, women's
likelihood of

persisting in the
workplace after

experiencing a trauma like

sexual harassment
in the workplace.

That summary is
basically going to

be the four or five articles
you're going to talk about.

You're just talking
about the main points,

and then boom, you're done
with the opening paragraph.

The problems are things that you

might get dock points on.

I would try to just get
as many points here.

I won't really dock points here,

but I will note these as

problems so that you can
work on it in the future.

If your introduction appears

instead of the paper's title,

no, I want your title.

If you don't introduce properly

before getting into details,

so don't have a
catchy opening hook,

that is not great.

The statement of the problem
is too brief or lacking.

Does not provide a
framework for the story,

so you're not telling
me what you will

tell me in the future
or how you're going to

outline or make an argument for

your research paper
and other things.

It could be like citing
issues and things like that.

Then for the literature review,

this is going to be the
bulk of your paper.

These will be like the
four or five studies that

you'll be talking
about that you may

have already done for
your Assignment 1B.

I want you to describe the
relevant parts of each study.

Hailey, have you read any
papers for your study yet?

I haven't completed
the assignment yet,

but I have five articles
picked out for it.

Cool.

One is how male allies can
help women in the workplace.

That one just gives an
account from male allies

in the workplace and how they

have helped in the past and how

the women want them to
help in the future.

I think that would be great.

It might not be one of
the first paragraphs.

It might be one of the
ending paragraphs,

but I would want you to evaluate

the importance of each study

and describe the relevant parts.

How did they collect
data from these males?

How did they identify allies?

What behaviors did they
describe as allied behavior?

How did women perceive
these behaviors?

That would be good.
I also want you to

tie findings together and
exploit common themes.

Across all of the five papers

that you're reading, of course,

they will be distinct,
but they will share

a similar theme of women

experiencing sexual
harassment in the workplace.

I want you to find similarities
and differences between

them and also to cite

your paper or your
articles in APA format.

This is why the citing
paper comes into play.

I'll be grading that today.

Great. The problems would

be an insufficient
literature review.

Maybe you just talk
about two studies.

This happened before you give
me the title of the paper,

but you don't really tell me
what the paper was about.

That really doesn't
give me an idea

of what research you've read,

that leads you to
ask this question.

I don't want all the details
about the study methods.

I don't want you to tell me they

recruited males from Portland,

Oregon on Tuesday and

Thursday because those are
the days that males go out.

I don't want to know
all that information.

I just want to know general.

You don't discuss the
importance of each study,

so, of course,
describe the study.

Main claims and supporting

evidence not clear
from citations.

Again, don't just put
the title of the study.

Give me some summary.

Summarizes article after article

instead of synthesizing
research findings.

I do want you to
include a summary,

but you should also have

a common thread
throughout and talk

about how the
articles inform one

another and how they

help you form your
research question.

Improper APA
citations and other.

Of course, APA is going to
be important throughout.

After you write the bulk of

your research or your
literature review,

you'll have a
paragraph identifying

gaps to motivate your study.

For instance, for Hailey,
it would be something like,

although research has examined
how male allies can help

women experiencing
sexual harassment

in the workplace,
limited research.

This might not be true,
but I'm just spitballing.

Limited research
has investigated

how sexual harassment affects
women's persistence in

the workplace and what
strategies women can

engage in for themselves

to deal with sexual
harassment in the workplace.

You'd compare and contrast
findings across studies.

This is where you can
include a sentence or

two for each of

the studies that you've
already discussed.

Talk about the limitations of

previous work and gaps
in the literature.

Sometimes allies are people

who exhibit sexual harassment
towards other people,

so maybe talk about how
allies are important,

but they can also be

perpetrators of sexual
harassment, etc.

Reiterate your main claims and

outline evidence to set

the context for
your current study,

and have APA
citations throughout.

If you don't compare and
contrast, that's not great,

if you don't critically evaluate

the limitations of the
work, that's not great,

if you don't summarize
everything that

you've talked about so far

before jumping into
the current study,

that's also not great.

Then your final paragraph will

be talking about your
study more specifically.

Hailey, the purpose
of my study is to

see how women experience,

sexual harassment
in the workplace,

and how that might
affect their persistence

in mostly male dominated
job or something.

Describe how your
experimental variables

will be manipulated
and measured.

Participants will be assigned
to one of two conditions.

In condition 1, participants
will be asked to either

recall or imagine a scenario

where they experience
sexual harassment,

or in the second scenario,

participants will be asked to

recall one of their job
experiences just generally.

Then all participants
will complete

measures of persistence
in the workplace,

and other things, whatever
your dependent variables are.

Then your testable hypotheses.

I predict that people in

the sexual harassment condition

will exhibit lower levels of

persistence in the
workplace compared to women

who describe your
typical job scenario.

The problems might be
lacking theoretical support,

but that's where
your intro comes in.

That's where you're
talking about

the articles that actually

helped you come up
with this theory.

I do want you to talk about
your IVs and your DVs,

so maybe you'll talk
about persistence.

Persistence will be measured by

the everyday persistence
scale created

by Chatterley and
colleagues in 2020.

If you don't tell me
what your hypothesis is,

then that's not great.

If I don't know what
your hypothesis is,

or it seems unclear because

of the lack of information
from prior studies,

it's also not great, and if it's

unclear how your study will test

hypotheses, it's also not great.

Let's say Hailey
said something like,

"I'm interested in
seeing persistent rates

among women who experienced
sexual harassment.

I'm going to recruit
10 male participants,

and I'm going to ask
them about how many

times they've sexually
harassed a woman."

That's not answering the
question that she posed.

That's completely different.

Now your references.

Again, references are
pretty important APA style.

Having a title that
says references,

using APA style,

everything you cited in

your paper appears in
the references and

having at least four
papers that you can cite.

Of course, you should
already have five with

assignment 1B, so
that should be great.

The problems could be it's
not formatted correctly,

APA style isn't followed
or used inconsistently.

That's why I'm going
to try to get you

feedback by the end of today,

if not the end of
tomorrow on your site me,

so that you know if

you did anything wrong,
what you can do to improve.

They have to be in
alphabetical order.

If you use less
than four sources,

you will be docked points,

does not match the text.

Hailey, if you cited Chatterley
and colleagues in 2020,

but then you use Tingle
and colleagues 2020,

then I would be
like, "Well, what

happened to the other citation?"

Too many secondary
sources cited.

So this would be using
data from CNN to see

how many women experience

sexual harassment and not
enough empirical articles.

You can use secondary sources,

but try not to use
more than two.

Then just overall format,

your organization, your
grammar, your spelling,

the length is sufficient for

the full picture of the problem,

your margins are good,

your double spacing is good,

your indentation's good,

you're using 12-point font,

and you have page numbers.

You don't start out
broadly and narrowly.

Of course, Hailey
with your paper,

you're going to be like, "Women

used to experience
sexual harassment."

That'll be really broad,

and then you'll go very
detailed because you'll talk

about your experiment
specifically.

It is a lot easier
to paraphrase.

If you're having trouble
paraphrasing something,

let me know and I can help

using active rather
than passive voice.

Don't give personal opinions.

If your margins
are inconsistent,

your double spacing
is inconsistent,

consistent font is not used,

page number is inconsistent
in the wrong place,

you spell check isn't used,

they're choppy transitions,

they're fragmented sentences,

length is insufficient to

capture the breath of a problem.

After all that, you
can get 50 points,

you divide it by 20, and
that'll be worth 25 points.

Just to let you know,

I do have a way of grading
all of these papers.

The first draft is worth less,

and then the second
draft is worth more than

the final draft is
worth the most.

I do that because I know you're

still learning how
to write a paper,

so if you do get docked
points on the intro,

I want you to be able to gain

those points by doing
better on the second draft.

The points that you
might lose here aren't

as meaningful as they would be

for the second or
the third draft.

Once you write the second draft,

it will include a revised intro,

and then your first draft
of your method and results.

Then your final draft will
include your revised intro,

revised method, and results
and a discussion section.

You're always going
to get feedback,

and my goal is

not for you to get it
perfect the first time,

because I remember this
was really hard for

me when I learned how
to write a paper,

so I want to see improvement.

I don't expect everyone
to get a perfect score,

but I would like to
see improvement.

Are there any questions here?

I will be uploading a
sample paper as well,

but I did want to just give
you a brief introduction.

But, how do I write
an introduction?

I feel the rubric is
good for telling you,

like, "These are
the things that you

should and shouldn't
do to get the points.

But I wrote this template

that you can use when you're
writing your introduction."

Let me open it up
in Word document

because this looks weird.

I'm going to stop the share,

and let me get my
Word document open.

That's a lot of information
I've thrown at you-all.

Any questions so far?

I really want to read
Hailey's paper now.

Let me share the screen again.

Great, and I'm going
to make it bigger.

I created this document to

help you on the
upcoming assignment.

You can read all of this.

But the author of your textbook

recommends you write
the main points of

an argument and individual
research findings

on index card,

so then you can
rearrange them to

figure out what makes
the most sense.

Let me get the sun out of

my face so to figure out
what makes the most sense.

You can also do this by
writing the topic sentence

of each paragraph in this
Word document and then

finding support for
your arguments from

the five or more articles you've

reviewed for Assignment 1B.

Things to keep in mind.
Your intro should be

between five to maybe
seven paragraphs.

That might be anywhere
between 3-5 pages,

so it shouldn't be that intense.

The most difficult part
is finding how to weave

the entire narrative through

the article summaries
that you have.

The first sentence
of each paragraph

should be a topic sentence.

For instance, if Hailey is

going to be talking
about male allies,

so she could say
something like, men,

not only women can provide
support for those who are

experiencing sexual
harassment and

also to prevent sexual
harassment from re-occurring.

Then you'll go into that study.

The topic sentence makes a claim

that the following sentences

support using research
findings as evidence.

Chatterly, is it Chatterly?

Chatterley and
colleagues conducted

a study where men were recruited
and so on and so forth,

and then you go on to the
summary of the study.

Respected and
experienced researchers,

researchers do not
use many, if any,

quotes when writing,
so try to paraphrase.

You can always paraphrase.

When discussing the
findings of past research,

only provide as
much detail about

the researcher's method as

is necessary for your reader
to understand the results.

I don't want how many
people they collected,

I don't want when
they collected it.

I just want maybe to know that
all the participants were

self-identified as male and

what questions they were
asked about allied behaviors.

If I'm curious about the
methodological details,

I think it's really interesting,

I can always look up
the information myself

by using the references
that you've cited.

Sure. Go ahead.

Well, this is all for
the intro, so okay.

Yeah. Do you have a question
about it, or is it?

It just caught me right there.

I just go, okay,

that's a lot, but I'm
following it. Sure.

It is a lot. I feel

like I would rather give you

more information
than not enough.

This is helpful. A
guideline is what I need.

Yeah. This is two pages long,

so I'll show you what I created.

There's just like things
to keep in mind generally,

and then I have paragraph
outlines that you can use.

They can end each paragraph
with a sentence communicating

the implications of a finding

you discussed in the paragraph.

I want you to answer
the question,

so what about the
research you summarized?

This tells the reader what

they should have taken
away from that paragraph.

The research from
Chatterley and colleagues

suggests that males are

important sources of preventing

future instances of
sexual harassment.

These are outlines
for your paper.

You could use this. Begin your
introduction with a hook,

something that is general.

What did I say? Imagine
you walk into work and

your male colleague
slaps your behind.

Though this might not represent

the sexual harassment that
women experience in the work.

Women do still experience
sexual harassment.

The present study will

explore the ways in
which women experience

sexual harassment
and the consequences

of sexual harassment for
persistence in the workplace.

First, I will outline.

Next, we will discuss

and then following,
blah. Then finally,

you talk about your
actual experiment.

First, I said,

then your first
paragraph follows.

That could be the
first paragraph in

your intro. Does
that make sense?

It's something really broad.

You talk about your
present study,

and then you give an outline

of what you'll be talking about.

You're telling me what
main points I should get.

The next sentence
should introduce how

your hook relates to
your research question.

Though this might not represent

the sexual harassment that women

experience in the workplace.

Today, women do still
experience sexual harassment.

The next few sentences should
discuss existing research

broadly and also outline

the limits of the
current research.

How your research question
will address the existing gap.

These sentences should
foreshadow what you will be

discussing in the
following paragraphs.

The present study will
explore the ways in

which women experience
sexual harassment and

the consequences of
sexual harassment

for persistence
in the workplace.

First, you're going
to outline law.

Next, we'll discuss
that following,

blah and then finally,

I will explore how these
studies tie together.

But I'd be more specific to
inform the current study.

The final sentence should
be about your study

that outline your IV
and your levels of it,

your dependent variable and

your predictions, and
so on and so forth.

I have a question.

Yes, go for it.

About the existing gap.

Could you give an example
or explain a little bit?

I missed really.

Sure.

How your research question
will address that?

Limited research to date
has directly examined

the consequences of
sexual harassment

for persistence
in the workplace.

You want to show work that
needs to be done, but hasn't.

You can say that about
mostly any study

because there are specific
studies that are out there,

but there are always ways
to make studies unique.

Right there at that point,

you want to state how years
is going to be unique?

How this study is
going to fill that.

Yeah. Rick, your study is
about COVID-19, right?

Well, yeah, depression due to
isolation from COVID, yeah.

Right. Your intro could
be something like,

in 2019 and before,

most people would say that

they'd love to stay home
and watch Netflix all day.

In 2020, that became a reality.

But what people didn't
anticipate is that they would

experience increasing
levels of loneliness.

The COVID-19 pandemic
was unprecedented,

and people experience
increased people

may have experienced
increased sense

of loneliness and depression.

Almost no research to date

exists to examine
how pandemics and

quarantining has effects on

people's emotional and
psychological state.

The current research
will explore law.

Because that's, like,
brand new research.

Well, I did found
some good ones.

Yeah, so there have been papers

that have been published
really quickly.

But prior to those papers,

there's been really
nothing because we weren't

doing this research
during the Spanish flu.

But those statements can be
made almost about any paper.

Thus, the present
study will explore how

these studies tie together
to inform the current study.

This would be a lot better.

This is just like
me spit-balling.

Also, you should know,
writing is hard.

This is not something
I would submit,

but this would be
something that I'm like,

Okay, that's a good start,

and then I would
go back and edit.

I wouldn't anticipate writing

this paper the day before
and submitting it.

I would maybe set a timer
for 30 minutes and say,

All right, I'm just going
to outline this paper.

No matter how crap it is,

it's still better than nothing,

and you can always work
with it to make it better.

Okay, so that's the
intro paragraph.

Then I want you to
support your argument.

Write the support for
your argument statement

as the beginning
of the paragraph.

Based on the literature
review you did,

what articles are finding,
support your argument?

Write a few sentences based
on these articles here,

reiterate your point, and create

a bridge between this
paragraph and the next.

Repeat until your argument
is clearly fleshed out.

Not to put you on
the spot, Hailey,

but I am putting you on spot.

Give me some titles of the
papers that you've read.

Okay. One is a
confirmatory study

of the relations between
workplace sexism,

sense of belonging,
mental health,

and job satisfaction among

women in male-dominated
industries.

Give me a sec.

Workplace harassment,
belonging, and what else?

Mental health and
job satisfaction.

Okay, and what's another one?

Hold on, women and male.

Another one is harmful
workplace experiences

and women's occupational
well-being, a meta-analysis.

You know what a
meta-analysis is?

Well, from what I've read,

isn't it just like the author

of that article is

taking findings from other
studies and kind of,

like, compiling it into one?

It's not his personal research.

It's him comparing a
bunch of other research.

Yeah, which might be helpful

because you'll be able to read

like 20 research articles
in that one article.

Yes, that's good. Any other?

Yeah, women and women of
color and leadership,

complexity, identity,
and intersectionality.

Okay, so let's just
go with these three.

You could say something one of

your paragraphs following
the intro could be

sexual harassment may lead to

decreased levels of belonging,

job satisfaction, and
occupational well-being.

Then I talk about
these two articles.

This could be two paragraphs.

These two articles I
would talk about here.

Maybe what I would
do is I would make

that a subheading and
make that a subheading.

So that will bolster
your argument there.

This may be particularly true

for women in
male-dominated fields.

Maybe that could be
the second paragraph.

Wow.

Does that make sense? Yeah.

Yeah.

Okay. The first sentence

of the second paragraph

might be something more
specific than that,

but that could be

your first sentence for
the first paragraph,

something like this for
the second paragraph.

These are the articles
that you will

use to support those statements

, and so on and so forth.

Then the third paragraph
could be something like women

with multiple stigmatized
identities may experience

the negative effects of
workplace harassment.

Seeing multiple may compound

the negative effects of
workplace harassment

for women or something.

Then you'll talk about this:

women and women of
color in leadership.

Then your fourth paragraph might

be something about allies

and preventing future
sexual harassment.

Then your final
paragraph, there you go.

I outlined your paper, Hailey.
It doesn't have to be that.

You can rearrange it,

but does that make sense Alicia?

I can do this for
anyone else's paper.

I'm just picking on Hailey.

If you want me to help, I can
do this for your paper too.

Then your final paragraph,

you'll summarize the arguments,

points listed in
Paragraphs 2 and 5.

As stated earlier,
sexual harassment

has negative
implications for women.

Women who experience sexual
harassment experience

lower levels of belonging
and job satisfaction;

citation, lower levels of

occupational
well-being; citation.

Further, those who possess

multiple stigmatized identities,

so on and so forth.

Then allies may help

prevent future sexual harassment
and so on and so forth.

However, limited work has
examined how sexual harassment

can affect not only
job satisfaction,

but also persistence.

In the current study,

I will examine how
sexual harassment

affects persistence among
college-aged women,

women of different ethnicities,
whatever you want.

You can talk about your sample.

I will manipulate sexual
harassment by blah, blah, blah.

I will measure persistence

by using the everyday workplace.

This doesn't exist by
the way. I don't know.

Persistent scale;
Chatterley and Tingle,

2020 which includes items like,

"I want to burn my
office down," on

a scale from one
strongly disagree

to seven, strongly agree.

Just to give me a sense of
what the scale measures.

"I often daydream about
burning my office down."

Then that's the end.
That's your intro.

There it is.

Well, does that make sense?

Alicia? Does that make sense?

I'm looking at it, but I just
want to say it out loud.

On the final sentence,

you're going to bring in

your references
and say what they

offered like a summary thing.

Mm-hmm. Then finally,
you're going to say,

"I predict that women who are

in Condition 1 will want to burn

their office down more
than those who are in

Condition 2," even though that's

not what you were
originally measuring.

I messed up. That
should be persistence.

I'm not burning
your office down.

But you get what I'm saying.

The last part, this is
the reaction I am going.

Here's the questions
that haven't

been asked or answered
or something.

Yeah. Hailey, I
can send you this.

I'm not sure how
helpful it'll be.

But anyone else who
comes to these meetings,

I'm more than happy
to use your study.

Perfect. Does that make it
less intimidating I guess?

I created this
just for your sake

and also for my sake because
I want better papers.

Then your support
for your argument.

This will be Paragraphs 2-5

and then your final paragraph;

Paragraph 6, the end. That's it.

Also, remember I do offer
extra credit for visiting

the writing center
and you can get

extra credit for every time
that you submit a paper.

For the intro; for
the method and

results and for the discussion,

so schedule an appointment
today if you want to get

that extra credit and

they'll also help you
write your paper.

On that should we state,

is it the timing of
when our thing is

before the paper
or should we state

in our thing with

the writing center what we
want to go over them about?

Do you mean, should you schedule

your appointment before
the assignments due?

No, just to get credit for
what we're going over.

It's just the timing of
the the tutor assigned.

Yeah, you can go over anything.

You can go over APA formatting,

you can go over making
your logical argument.

You can go over
anything you want.

I just want you to visit
the writing center

and make use of the
resources we have available.

I wasn't able to get in touch
with Emily for some reason.

Was that who you were
about to say we also have?

Yes, Emily. I'll reach out
to her. Did you email her?

Do you have that information?

I will get that information

and send it out to
everyone because that's

a tutor that we have
that's separate from

the writing center and
she's experienced in 301,

so she can help with that.

I need to talk with her;

it's getting deep
on the stuff here,

so just to go over stuff

and I feel more
comfortable in it.

A few things. Appreciate
that. Thank you.

Of course, yeah. I'll
send it out to the group.

I wrote it down. Alicia,
how are you feeling?

I'm feeling okay. I've been
having a little bit of

trouble with finding studies.

I think it's because I'm not

sure specific keywords to

use or what to
actually look for.

Has anyone else been
having trouble with that?

Hailey, it seems like you've
got it all down. Rick?

My keywords, I think I had
them right off the bat.

Well, depression due
to COVID isolation.

It got me some real
pinpoint stuff.

They're using different
methods so I can cross.

It's nice. There's
one self survey and

then there's statistical
and physiological also.

It's all great.

Alicia, you scheduled an
appointment with me. Right?

I have not.

Okay. Do you have
time after class?

We might not go the full
hour so I can help you.

Yeah.

Okay. Anyone else?
If you want to help,

schedule an appointment
or stay after class,

and I will go over those things.

I do want to finish the lecture

today. Everyone
take a deep breath.

The intro will be fine.

I need to take a deep breath
too because I feel like

I talk so much that
I forget to breathe.

The intro will be fine.

I'm more than happy to meet with

you during my office hours.

I still have office
hours this week.

I have advising hours
that no one is coming to.

You can come to those
advising hours too.

I can make time
to meet with you.

I'm more than happy to help you.

I love research especially
when it's not mine.

When it's my research,
I'm just like,

"I don't want to do it,' but
I love helping other people.

Let me know. Alright, so
let me get my PowerPoint.

We're almost done
with the PowerPoint.

Where were we?

Share screen.

Yes. I believe yes,
that is what I want.

Because I just got
my external monitor

back. Thank goodness.

We're going to continue
with Chapter five.

If you have a question,

just holla because I can't see

your faces now for some reason.

Identifying good measurement.

I do want to see
your faces, though.

What do I do? I'm
going to put here.

I can't see. Oh, man. I
can't see your faces.

Just speak out if you know

the answers to the
questions I'm going to ask.

Can you see the screen

without the notes or
can you see the notes?

I just see you.

Yeah, right now, it's basics.

I see the school.

Oh, my bad. Oh, man. Let's see.

There they are.

Now.

Perfect.

Now I can see you.

We talked about three
types of reliability.

What are they?

I didn't ask
questions last time,

but I'll do it this time.

The three types.

Alicia. Anyone else?

I don't know where
I'm at. Oh, my God.

Yes, Hailey.

Test reliability, inter-rater

reliability, and
internal reliability.

Perfect. I won't make you
answer what they are.

But Rick or Alicia
or anyone else,

do you remember what
those mean? Test, retest.

When you test or do, I
want to call it a study.

I don't know if that's the
right word, but basically,

do it at one point.

We're looking at test-retest
reliability for IQ.

For IQ? Say, today
you test someone's IQ

and they get a specific answer,

and then say next
week you test them

again and compare the results.

Good test for test
reliability would tell us

what about the IQ scores?

That it's an accurate
representation of

their IQ and it's
reliable data to use.

Close. Rick or Hailey?

I think Meghan
raised their hand.

Oh, Meghan, go for it.
I didn't see your hand.

It would be consistent,

so it would be around
the same quantitatively?

Yes, perfect. Let's
say at Time 1,

you get an IQ score
of 125 and Time 2,

you get an IQ score of 128,
that's relatively consistent.

I think that's what you meant

Alicia when you said accurate,

but it's about consistency,

so having almost the same score.

I would have low
test-retest reliability if

one day I score a
40 on an IQ test,

and then the next
time I score 140,

so that doesn't have good
test-retest reliability.

Perfect. We talked about
inter-rater reliability.

Does anyone want to
take a stab at that?

Maybe Meghan? Or Rick.

Well, do the same test,

but it's not you doing it again.

It's two different people

doing the same test on the same.

That's what it is. You're
looking for consistent scores,

but using two different
people doing this film.

Yes, can you give us an example?

I came up with really
silly examples,

so just go for it.

I test to see if,

I run around the block
and I get sweaty,

and then someone else

test to see if running
around the block.

No, that's wrong. I don't
know. I can't think.

No, that's okay. I think

the key here is that
there are two observers.

There's two observers.

Yes.

But for the same testing.

Yes. In your example,

Alicia would be my
research assistant,

and she would be coding for how

sweaty you get
versus someone else.

I would also have Hailey.

Inter-rater reliability
means that Alicia

and Hailey score you similarly
on how sweaty you get,

and Hailey and Alicia score

the other person similarly
in how sweaty they get.

It's two observers looking at

behavior in someone else.
Does that make sense?

Yeah, thank you.

Yeah, I think you
were almost there.

You just had the
two people down,

but not sure where it
went, but that's good.

That's still good. What
about internal reliability?

Isn't this where if R
is positive and strong,

you have a good reliability?

Yes, but what does that mean?

I'm not quite positive.

Who was that?

Grace.

Grace. Yeah. Good
job, Grace. Hailey.

Isn't it where, when you ask
questions to the people,

so it's like two
different questions,

but they both are trying
to get at the same answer.

They're just worded differently,

and the people answer
them consistently.

Yes. Rick.

Same question, word
it differently.

Yes. In Hailey's
questionnaire that I created,

it would be if you ask people,

how likely are you or how

often do you daydream about
burning your office down?

Then also, if your
office was burning down,

would you call the
fire department?

Those are questions that
are framed differently,

but also measuring the same,

like, how much do you

appreciate or want
to work at your job?

Oh, okay.

Yeah. They would
answer consistently.

Good. I'm going to keep

asking these questions
just to make sure.

It's okay if you don't
get the entire measure

or the type of
reliability, 100% there.

As you can tell, most of you
were 80% there. It's okay.

And during the exam,
it will be open book,

open note at the
end of the term,

which is also, I guess,
the end of the year.

Still good job. We
talked about this.

We looked at the measurement
of head circumference,

at Time 1 and Time 2.

Which reliability would this be?

You measure the head at Time 1,

you measure the head at Time 2,

and the measures are pretty
related to one another.

Which of the reliabilities
would this be?

Test-retest.

Yeah. Perfect. This is
inter-rater reliability.

Does anyone want to
explain the difference

between the left and
the right to me?

The left one is showing
that both observers,

their ratings are very
close to one another,

whereas the right
side is showing

they're all over the place and

not quite matching
up with one another.

Perfect. You would
say that this one has

high interrater reliability and

this one has low
interrater reliability

because we have two observers,

so Mark's ratings
and Matt's ratings,

and they both seem to
track on the same line.

But if we have Mark's
ratings and Peter's ratings,

they don't seem to track on

the same line;
they're scattered.

This would mean that this has
low interrater reliability,

so I might not trust
Peter's ratings.

Maybe he's using a
different assessment

for sweatiness if we're testing

Rick and other people's
sweatiness when they run,

but Mark and Matt seem to

be on point with what
they're measuring.

I believe this is where we were,

and this is what Grace
was talking about.

Using the correlation
coefficient

r to evaluate reliability.

The correlation
coefficient finds

the relationship between
two different variables.

Someone give me two
variables that could be

related or maybe not related.

Foot size and height.

Foot size and height.
We have foot size

being X and height being Y.

We could use all
of these different

scatter plots to represent that.

Another thing to note for
the correlation coefficient,

the values go from
negative one to one,

so it includes zero.

The sign in front of the number,

whether it's negative
or positive,

tells you the slope direction.

If the slope seems to be
increasing like this one,

all the data seem
to be increasing,

as X increases, Y increases;

you would say that this
is a positive slope.

As X increases as shoe size
increases, height increases.

This would also be
a positive slope.

This one, on the other hand,

would be a negative
slope because as

shoe size increases,
height decreases.

If you're a size, let's
just say this is 11,

then you are five foot or
four foot two or something.

as shoe size increases,
height decreases.

But this one doesn't really
have a slope because there's

no way to predict as X
increases what Y would be,

since it's all scattered around.

You can see what the
slope direction is or

whether there's a positive
value or negative value

by looking at the slope of
the pattern of the dots.

Does that make
sense? Then you can

tell the strength by how
closely together the dots are.

I would imagine a line going
straight diagonally here,

and how well do the dots
match up to that line?

They don't match up
super well here.

This is a 0.56 relationship.

Again, from negative
one to one, this one,

if you had a line,

they all seem to
track really well.

This is an r of 0.93.

You can tell that it's
really strong by how well

the dots adhere to one another.

This one is -0.59,

so it's a little better
than this one or a

little stronger
than this example.

This r of 0.01,

there's no strength, it's
not really predictive.

Perfect. Let me see

my notes if I need to
tell you anything else.

Any questions here?

Which is the strongest r
in these scatter plots?

R 0.93.

Perfect. If it was a -0.93,

would it still be the strongest?

Or actually, negative 0.94.

I think so because

negative correlations
can also be strong.

It just depends on
what you're measuring.

Perfect. It doesn't really

matter whether it has a
positive or negative.

It's more about the absolute
value of the r. Of these,

the r 0.93 is the strongest,

and which one is the weakest,

Rick or Elysia, or Grace?

I can't see the numbers,

but the scattering ones.

This one.

That one. 0.01, yeah.

Perfect. How does
the correlation

coefficient evaluate reliability

for the test-retest reliability?

You would correlate
their first IQ score to

their second IQ score and
see how strong it is.

If your first IQ score was

125 and your second
IQ score was 126,

they would correlate
pretty highly.

Your r would be very positive.

If r is strong and positive,

at least 0.05 or above,
you have good test.

I think it should
be 0.50 or above.

Then you have good
test-retest reliability.

If r is positive at weak,

this is a sign of low
test-retest reliability.

This reliability is
only relevant for

variables that don't
change over time.

It wouldn't be relevant
for something like weight.

Weight tends to fluctuate
more than shoe size, maybe.

What about for
interrater reliability?

Two observers rate the same

participants at the same time,

and then r is computed.

You want to see how Hailey and

Elysia's evaluation of
sweating after running,

how related they
are to one another.

Here, if r is
positive and strong,

the value has to
be 0.7 or higher;

you have good
interrater reliability.

Here, you want it to be
positive, not negative.

If r is positive but weak,

you don't have good
interrater reliability.

Negative r in this situation
would be meaningful,

and it would mean terrible
interrater reliability.

If you had a -0.7,

that would mean that when
Hailey rates the person

as 5 on sweatiness,

Elysia would rate them
as -5 on sweatiness.

As Hailey's scores go up,

Elysia's scores go down,

or the other way around.

Internal reliability, this is

when you have more than one item

to tap into the construct.

This is like the example
that I gave with

Hailey's burning
the building down.

I would correlate the two values

to see how related they
are to one another.

Another important
thing to note is,

I asked one question
where positive means more

likely to burn the
employment place down.

The other question, I said,

how likely are you to

prevent the burning
of the place down?

I need to reverse
code that item,

and we'll talk about
what that means,

and then correlate the items to

see how well that taps
into the construct

of workplace satisfaction.
Does that make sense?

I must hear more about
reverse code. I like that.

We'll talk about that, and
I'll show you how to do that,

but maybe not in this lecture.

Essentially, what
you want to do with

reverse coding is you want to

make sure that all of your
items mean the same thing.

I want to tap into
the construct of

workplace satisfaction
by assessing

desire to burn your
workplace down.

The question, how
likely are you to

call the fire department

if your workplace
was burning down,

is not necessarily measuring
your workplace satisfaction,

but it might be
measuring how likely you

are to not want it to burn down.

But if I want to measure

how much I want the
building to burn down,

then I need to
reverse code that.

If you said, strongly disagree,

I would not want to
call a fire department;

that would mean now
with reverse coding,

it would be strongly agree.

I would word the
question instead

as I would not be likely to
call the fire department,

and then strongly agree.

But it'll make more sense once

we get to that part
of the chapter.

Does it make sense now?

Yeah.

That is almost word for word
what I said. Yes, Hailey.

Just like what you were talking
about the reverse coding,

I've taken personality
tests for previous classes.

Is that why some questions
are just the same words,

but worded backwards on the
big five personality tests?

Exactly. That's why they're

worded backwards because
we want to make sure

that you answer consistently
irrespective of

how we word the questions
because if I asked you,

on a scale of 1-7, how
happy are you from one,

no emotions, to seven,
extremely ecstatic.

I might also ask you,

how unhappy are you from one,

to not at all,

to seven, no emotions.

Then I'd reverse code one of

the items to actually
assess happiness.

I've always seen those
questions and I'm like,

didn't I just answer
questions exactly like

this, but that makes sense.

Rick. Sorry.

Hey, [inaudible] exactly
what you're saying.

That's exactly what I'm
thinking when I'm reading this.

It just clarified something.

It was really good
because all the

time in all these surveys,

I'm going, I've just read
that question, so this helps.

That's the reason why we do it.

We want to make sure that
there's reliability in

the scale because if you

answer the questions
differently,

then that means that
we're not appropriately

measuring the construct or we're

not measuring it as well
as we thought we would.

I said those in words so
you can have that later.

What about in journal articles?

Rick, you'll probably
read about this.

All of you will probably
read about this, but Rick,

you were talking
about physiology,

and self-report, and
things like that.

You'll see the reliability
presented like this.

I'm trying to get
my mouse over here.

Test-retest reliability,
for instance, 0.83,

0.84, 0.6, 0.82, 0.5, 0.54.

Hailey, are you looking
at a meta analysis?

You might find this too.

I have, I think,

two articles that
are meta analysis.

You might find some values
like this for test-retest.

This is telling you how
long between the tests.

This is a satisfaction with

life scale, but
they're different.

There's Alfonso and Allison,

Pavot et al, Blais et al,

Diener et al, and
so on and so forth.

If you take the tests
that Alfonso and Allison

gave at Time 1 and
two weeks later,

the test-retest reliability
is 0.83 with one month.

With a Pavot et al, 0.84.

Two months, you see
it going down, 0.64.

The Diener et al
test-retest reliability

is actually higher
at two months.

At 10 weeks, of course,

it goes down to 0.5,

and in four years,

it goes down to 0.54.

That means that the
scores that you have at

Time 1 aren't necessarily as

related at Time 2.
Does that make sense?

The coefficient Alpha is for
the internal reliability.

How good do all of the questions

that are in that scale measure

the construct that
you're interested in?

Here you want a good value.

How well does, I'm very happy

and I feel really sad,
reverse-coded measure happiness?

Here, the coefficient
Alpha is 0.89.

That's good because
it's close to one.

0.85, still good,
0.79-0.84 still good.

All of these are
greater than 0.70,

which means that all the items

tend to hang together nicely.

Does that make sense?

That's what you'll see in

the text or any
articles that you read.

Now we're going to talk about
validity of measurement.

We're going to talk about
measurement validity

of abstract constructs.

We've talked about reliability.

Now we're moving on to validity.

What is the difference between

face validity and
content validity?

Does it look like
a good measure?

Criterion validity.

Does it correlate
with key behaviors.

Convergent validity and
discriminate validity.

Does a pattern make sense?

Relationship between
reliability and validity.

What are the three
reliabilities again?

Rick. Alesia or whoever
was saying that.

Test-retest. I keep
wanting to say relator.

Interrater.

Interrater. My gosh. I'm
blanking on the third one.

Internal. How do we
measure the strength of

those reliabilities?
It's a letter.

The coefficient R.

Coefficient R. The
coefficient R,

what are the endpoints?
Or what's the range?

Negative one to one
including [inaudible]

Perfect. What do
we want the value

to be to be good reliability?

We want it to be positive and
closer to one as possible.

That's what I've been lecturing
about for the last hour.

[LAUGHTER] Also last
lecture and this lecture.

I just want to make sure we
are all on the same page.

Now we're moving to validity.

Let's move on to validity.

We talked about
reliability here,

so test-retest, internal,
and interrater reliability.

Now, we're going to move on to

here with construct validity,

and we'll talk about all of
these different validities,

but we're mostly
going to be talking

about construct validity here.

We're going to jump into

two subjective ways
to assess validity,

so face validity and
content validity.

Face validity is exactly
what it sounds like.

Does your measure look like
what you want to measure?

Then content validity
is the measure

contains all the parts that

your theory says
it should contain.

We have different standards for

measurements as
physical scientists,

so we can use rulers, scales,

thermometers to ensure that

we're measuring
things like length,

weight, and
temperature reliably.

However, psychological
scientists

are interested in measuring

abstract concepts
like self-esteem,

or how much you hate
your job, or depression.

That's more abstract.

Because of that, our
construct validity is a lot

more important to
psychological scientists.

For people who measure
physical things like

weight, well,
construct validity.

How would you measure weight?

The scale.

That's pretty easy, but for
more abstract concepts,

construct validity
is super important.

Let's jump into face validity
and content validity.

Like I said, face
validity is it looks

like what you want to measure
and content validity,

the measure contains all of

the parts your theory
says it should contain.

Alesia, what is
your study about?

How introversion and

extroversion affect
anxiety management.

Having good validity would

be asking questions
related to introversion.

Do you enjoy spending
time with others?

Do you get energized by spending

time with others?
Things like that.

It seems to have good face
validity and content validity.

Contains all the parts that

your theory says
it should contain.

With introversion, we're talking

about how much time you
like spending with others,

so we're asking those questions,

but if I ask
questions related to,

how much time you spend online,

that might not be good content
validity because it's not

assessing how much time you
enjoy spending with others.

[inaudible]

It could if you did how
much time you spent on

Facebook talking to people
or whatever the case may be.

Does that make sense, face

validity and content validity?

If I was measuring happiness,

it might be asking questions
about how happy you are,

how sad you are.

It might not have as

good content validity if
I ask how angry you are,

how violent you are.

That might not be a great
measure of happiness.

It might be a great
measure of likelihood to

engage in violence,
but not happiness.

Then we have criterion validity,

and this is when you
want to see whether

your variable of interest
correlates with key behaviors.

Most psychological
scientists prefer to rely on

empirical assessments of

validity over
subjective judgments.

One way to empirically
assess validity is

by examining criterion
validity, which is here.

Whether the measure
is related to

a concrete outcome that
it should be related to.

Let's say you work
for a company that

wants to predict how well

job applicants do a salespeople.

Initially, the company used

IQ scores to predict sales
aptitude. Here we go.

The criterion
validity. Right here,

we're checking how
well aptitude test

A relates to sale figures
in thousands of dollars.

You wanted your IQ test to

predict or relate to a
behavior, sales figures.

As you can tell,

as aptitude test scores go up,

so do sales figures in
dollars. They're, great.

We can use IQ to predict how
well our employees will do.

Let's say that then they
also have aptitude test B,

which is a different IQ test.

They're looking at
how that relates to

sales figures and what they

find because these dots are a

lot more scattered
from that line,

that aptitude test B
doesn't have as great of

a criterion validity
as aptitude test A.

That means that aptitude
test B is less likely to

predict sales figures
relative to aptitude test A.

Is that clear? Just
criterion validity is you're

using one measure to

predict behaviors or
something another scale.

You're comparing the measures
that you're going to

use to see which one
is best serious.

No, that's good.

For the burning of the building,

you might want to use burning
of the building scale

to what people actually do
when the building is on fire,

how likely you are
to put the fire

out if there is an actual fire.

What you find is maybe

people burning saying
that they would burn

a building is not the same
as what they actually do in

a fire so might not have
great criterion validity.

Another way to gather evidence
for criterion validity

is to use a known
groups paradigm.

What is a knowns group paradigm?

This is when you have

a group that you know
that scores high

on this measure or on
a related measure,

and you create a new measure.

For instance, the Beck
Depression Inventory

is one of the most
popularly used inventory

to measure depression.

But when it first came out,

there were already existing
measures of depression.

If you want to create
a new measure,

you have to have a good reason
to include that measure.

The BEC depression inventory is

a 21 item, self report scales.

In order to test the
criterion validity

of this depression scale,

you could administer the BDI

to a group of people
with depression,

so are already in treatment for

depression and a group of
people who aren't depressed.

If you find here,

this is the known group.

You have a group of
people who are depressed.

You have an inventory that
should measure depression.

If you find that depressed
people score higher on

your depression
inventory relative

to those that are not depressed,

then that tells us that there's
good criterion validity.

The depression
inventory actually

predicts who is depressed
versus who is not.

This is the known groups method.

Another way is to separate

the depression that people
are experiencing from mild,

moderate, and severe.

This could be the
psychiatrists rating of how

much their patients are
experiencing depression,

and the BDI score,

the BEC depression
inventory should be able

to distinguish between
those who have mild,

moderate and severe depression,
which happens here.

Severe people are experiencing
the most amount of

BDI relative to those who are

moderate and mild depression.

Is that clear? Do you guys

like me to use the
weird examples.

Example. We're good.

I was fine. The burning

the house down or burning
your place of employment,

you might have pyromaniacs and

measure whether
pyromaniacs would

score higher on the
likelihood to burn

the employment down relative to

those who are not pyromaniacs,

and you would want
people who are

pyromaniacs to score higher on

likelihood of burning place
of employment down relative

to those that
aren't pyromaniacs.

That would be a good
measure of fire.

I actually like my job.

I don't know why I came
up with that. [LAUGHTER]

It's very dark.

It is very dark. I don't

know I came up
with that example.

Perfect. Does that make sense?

Your measure should be able to

predict likelihood of starting

a fire and the degree

to which you would be
likely to start a fire.

Pyromaniac should
be high here and

then mild pyromaniacs versus

moderate pyromaniacs
versus severe pyromaniacs,

it should also be
able to predict

the extent to which
people are pyromaniacs.

That's one way to establish

whether your scale
is good or not.

We can also use known
groups evidence

for criterion validity.

This is Diener's subjective
well being scale.

Remember, Diener created
this to measure happiness.

We're using it for different
groups of individuals.

What we are using is
American college students,

French Canadian
college students,

Korean university students,
printing trade workers,

veterans Affairs
hospital patients,

abused women, and
male prison innates.

These are the references where
they got the scores from.

What the Diener's subjective
well being scale would want

is scores that map on to

these different
measures of happiness.

Does that make sense? When
you measure happiness

with Pavot and Diener measure,

you have a 6.4 of

happiness for American
college students.

With Blais et al, you have 6.1,

for Suh study, it's 5.8,

for George, it's 6.0,

for veterans Affairs
hospital inpatients

with Frisch, it is 5.6,

for Fisher at 7.4,

and that's the
standard deviation.

Abused women is 20.7,

and male prison inmates
with Joy is 12.3.

Just by looking at this,

do you think that

these measures are okay
measures of happiness?

For instance, who is the
happiest based on these values?

Printing trade workers.

Printing trade workers,

their jobs are
probably pretty chill.

American college students or

these college students might
still be super stressed out,

but they're still
relatively happy.

What about male prison inmates

and Veterans Affair
hospital inpatients?

They're pretty low on happiness.

It tracks so it does seem

to accurately measure happiness.

Because the actual values

of happiness are tracking
what we would expect,

then we can say that the
subjective well-being scale

would have high
criterion validity.

It actually measures
what it's intended to

measure. Does that make sense?

Yeah, I can read that one.

This one's very clear. Now we

have convergent validity
and discriminant validity.

This is concerned
with whether there is

a meaningful pattern of
similarities and differences.

Convergent, as you might guess,

is how well does the BDI,

Beck Depression
Inventory relate to

an already existing
inventory for depression,

which is the Center
for Epidemiologic

Studies Depression Scale.

It's not hard work.

The Center for Epidemiologic,

no I can't say it.

You know what I'm trying to say.

Studies Depression Scale,

and these have a correlation
coefficient of 0.68.

I'm just going to
call it the CES-D.

When the Beck Depression
Inventory came on the scene,

this was the most popularly
used measure of depression.

But the BDI needs to

be similar or converge

on an already existing
measure of depression,

but it also needs to be slightly

unique because if it's
exactly the same,

then why are we
creating a new scale?

Right here, let's say

we have people that we
know are depressed,

and they take both
CES-D and the BDI.

What we see is that the
scores do tend to converge.

As people increase in BDI,

they also tend to
increase in CES-D.

But it's not a perfect
relationship, and that's fine.

We don't actually want
it to be perfect.

Now, discriminant
validity is how

well and how different is this
scale from other measures.

A measure should
correlate less strongly

with measures of
different constructs.

In other words, there
must be differences.

The Beck Depression Inventory
is supposed to measure

psychological well being or

non-well being and not

necessarily physical
health problems.

If this has good
discriminate validity,

which it does, that
means that your BDI,

Beck Depression
Inventory should be not

related to physical
health problems

and that's what we find.

As BDI increases, it doesn't

necessarily predict how many
health problems you'll have.

Convergent validity
is how well does

your measure map onto
already existing measures,

and discriminant
validity is about how

much your measure correlates

with measures of
different constructs.

Think of convergent as same and

discriminant as different
with different scales.

Any questions here?
Then the relationship

between reliability
and validity.

We talked a lot about
reliability and validity.

Remember that just
because a measure is

reliable doesn't mean
it's a valid measure.

For instance, I could measure
my cat's head every day,

and I'll have a really reliable
measure of my cat's head,

but that might not be
a valid measure of IQ.

Head size might not be
a valid measure of IQ,

meaning that it doesn't
actually measure IQ.

It's reliable, but
it's not valid.

Reliability is more of how well

a measure correlates
with itself.

Invalidity is how
well a measure is

associated with something else.

A measure can be less
valid than it is reliable,

but it cannot be more
valid than it is reliable.

For instance, a measure can
be less valid than it is

reliable so measuring
my cat's head

might not be related to its IQ,

so it's less valid
than it's reliable,

but it can't be more
valid than it's reliable.

IQ scores cannot be a
more valid measure of

head circumference relative to

how reliable the head
circumference and IQ is.

Reliability is necessary but
not sufficient for validity.

These are just things
to consider as you're

thinking about your projects.

There will be some checks and

balances that you'll have
to go through when you're

thinking about validity
and reliability.

You'll want your scales to be

reliable and you'll also
want them to be valid.

Reliable means that the scales

actually measure what you
want them to measure,

and validity is how well

that measure is associated
with something else.

For instance, Hailey, if you're

interested in job persistence,

you want the measure of job
persistence to actually

predict the people being

in their jobs maybe
five years later.

That would be the validity.

That is it for today.

Any questions?

We're going through
a lot of chapters,

and I think it's good
that we're actually

keeping on track.

But if you don't have any more
questions, then that's it.

Enjoy your weekend.
Alasia, if you

want to stay. Thank you.

Thank you.

Thanks.

Thank you.

Thank you.

Let's see. I stopped
recording. I think I did.