Marty: Hello, this is Marty Blair, I’m the policy director for the National Center for Disability to Access to Education, NCDAE. It is a federally funded project, funded by the U.S. Department of Education, which is dedicated to improving access to electronically mediated educational services, including distance education for all students by monitoring and promoting education policies and practices that enhance the lives of students and people with disabilities. Today's web cast is focused on issues related to computer-based assessment and testing. This topic has surfaced in several national meetings hosted by NCDAE, as one of the most important issues for students with disabilities. We are delighted today to have three panelists who will be discussing with us various issues related to computer-based assessment and testing. We have Carol price with us, she is from CTB McGraw Hill. She is a development manager for CTB McGraw Hill, and has more than 20 years of education experience. Prior to her position she worked at Louisiana State Department of Education, where she was responsible for the development and administration of the alternative assessment. She has two PH.D. degrees from University of New Orleans, one in educational administration, and another in special education, with concentrations in deaf education, mild moderate disabilities. Carol, we welcome you, thank you for being here.
Carol: Hello.
Marty: Our next panelist is Dr. Ruth loew. Dr. Loew is the assistant director of the office of disability policy at Educational Testing Services, or ETS, in Princeton, New Jersey. She holds degrees in linguistics in education from Brown University, Northwestern University, and the University of Minnesota. Before coming to ETS Dr. Loew served on the faculty of the National Technical Institute for the Deaf in Rochester, New York and held research positions with Rutgers University. Her current responsibilities include consultations on adapting tests for individuals with disabilities, coordinating activities of ETS's disability policy team, representing ETS on the national task force on equity and testing deaf and hard of hearing individuals, participating in test accessibility research, and training staff involved in testing individuals with disabilities. Dr. Loew, we welcome you.
Ruth: Thank you.
Marty: Our third panelist is Mr. Preston Lewis. Mr. Lewis is the manager for the program services branch, the division of exceptional children's services with the Kentucky Department of Education. In this position, he has served as the state lead on the Kentucky universal design for learning initiative. This includes the development and implementation of Kentucky's accessible statewide assessment for students with disabilities, known as cats on line. Mr. Lewis has been involved with special education for 30 years and has served as the principal investigators of two stepping stones projects, project care, and project status. And he is the University of Kentucky liaison for the universal design of assessment applications of technology stepping stones project. Mr. Lewis, we welcome you.
Preston: Good afternoon.
Marty: We are delighted to have our three panelists with us. They will be discussing from their various perspectives issues related to computer-based assessment and testing, and we have asked each of them to share with us for five to seven minutes their perspective on this topic, in the order that they were introduced. So we will start with Carol price.
Carol: Thank you, Marty. I’m going to discuss the current state of technology to improve accessibility in testing situations and the use of testing tools. And I wanted to look at this briefly from a three-prong approach--the classroom, districts, and states. First, in the classroom, we really need to encourage professional development that explores best practice for accessibility, so that students can implement a wide range of accommodations from low-tech to high-tech solutions. And then when we look to the district we need to ensure that districts have in place their infrastructure, and they, so that they use the appropriate hardware and software that's required to make assessments accessible for individuals with disabilities for all schools. And I use the word "individuals" here, not just for students, but often in low-incidence populations there are teachers of the blind or of the hearing impaired and deaf who are also blind or deaf themselves. So it has to be for the whole school. And we need to look at that for large city school districts, campuses, all the way to the remote rural school houses. At the state level, states need to embrace 508 standards, and consider them a road map for their comprehensive plan to ensure accessibility for all individuals with disabilities. Faced with funding challenges, states can seek direction from their on-line assessment vendor to create a long-term approach to accessibility by phasing in critical accommodations, like those for students with IEPs and 504 plans. And then do the critical need for the largest population first, and then add enhancements later. Another area - there are two other areas that need consideration. One is test security. When a student has an accommodation on a test, tests read aloud, it is very important that the text reader that is used by the school district does not allow importing or exporting, that it sits on top of a browser, so that the test security is not breached. And I know Preston will talk about this further, so I won't go into detail. But that is a major concern for high-stakes assessments from a vendor's perspective. And the other area of concern is computerized adaptive assessment. This is an effective means of formative assessment, and formative assessment empowers teachers and enhances student performance by identifying areas of strength, as well as areas that require improvement. Computerized adaptive assessment can be easily designed to provide speedy results and links to instructional resources for both students and teachers. And that's my part for this presentation.
Marty: Well thank you very much, Carol. You've certainly addressed issues from the K-12 perspective. We're going to shift gears, now, and talk about computer-based assessment issues, the broad issues from the professional and post-secondary perspective. Before we ask Dr. Loew to speak to us for a moment, I would just like to remind you that at the bottom of the form on the web page that you are viewing, there is an area for you to submit questions. And we invite you as you are listening to our panelists, if you have questions, please submit them, and we will address them as time permits. Dr. Loew.
Ruth: Thank you. The issue I wanted to address, for starters, was considerations of test construct validity, and accessibility issues. As we move into a more technological environment, there are many, many accommodations that are being used in educational settings, both the K through 12 setting and the post-secondary setting, that may or may not be appropriate in a high-stakes standardized test. Some of the most obvious examples are things like audio, whether it's a live reader, or in the case of computer-based testing, the recorded audio, or text-to-speech rendering. What is thought of as accessibility for classroom purposes, just making information available in whatever modality works for the student, may, in fact, run counter to the purpose of a test. If you really want to know whether a student can read, in the sense of decoding, then producing a test, or allowing a test to be delivered in audio doesn't enable you to find out whether the student really can read. The same thing is true of a sign language delivery for a deaf or hard of hearing student. If a test is rendered in sign language you don't necessarily find out whether the student is able to read. Whereas in a classroom setting, things like signing avatars or signing animations may be excellent aids. So people have to make a mental transition from the world of pedagogy to the world of testing. And it's not always an easy transition to make. I find myself doing a great deal of consciousness raising in that regard. I do see the state of the art improving. I think that in both the K through 12 setting and the post-secondary setting, which is what I work with primarily, testing organizations and schools, I think, are working together more to get an understanding of what the key test constructs are. But it's clearly an area where more work is needed, and sometimes there's sort of a truism that you want the student to be tested the same way they are taught, with the same accommodations with which they are taught. And I'd like to propose the heresy that that's not necessarily the case. Because sometimes in a testing situation you have to withdraw some support that might be available in the classroom in order to find out whether the student really has mastered the skills. We can continue with this later, but I think it's time to move on.
Marty: Thank you. And I think the issue that you're talking about applies not only to the population that you work with, those in post-secondary situations, but it applies to all students who are required to take tests, particularly high-stakes testing. Which moves us to talk with Preston Lewis from the Department of Education in Kentucky. Mr. Lewis.
Preston: Thank you, Marty. First I want to say how important this issue is. In Kentucky we're moving into our third year of online accessible assessment for students with disabilities and accommodation, and it's extremely important from the student standpoint because of newfound freedom in order for the student to have a way to independently show what they've learned. Kentucky's not unlike a lot of other states, and students have to have support to participate, especially in statewide assessments. About 40 percent of students with disabilities need accommodation to participate in our state assessment. We have students who have gone from, as we say, stigma to status, the previous stigma of having to have an adult there to read a test or support with a test accommodation, today's technology they can take a test independently. And it's important not only in terms of the ability of the student to use that technology, but what it does for their ability to do that, again, independently, as opposed to dependently, and how they begin to feel about the test itself and their role in it. As the students told us over and over, now can I do it myself. A lot of our kids who used to be the first ones through with the test because they didn't want to deal with the stigma of adult support using technology are the last ones using the test now they can control the pace.
This is all extremely important for states and this realm of NCLB and AYP where populations are going to have to improve in terms of performance, and we have to use all the tools that we can to ensure we are better able to measure that performance. In looking at their experience with assessment, there are a lot of issues that have come up. One major issue is that students have to be able to bring to the table the technology using accessible tools of assessment site. And this can't be the typical issue of where you have kids using a single station text reader. We've had to move into a site license mode so that schools, if they have 100-something kids testing at one time you can't have kids lining up at the computer of a test. You have to have a side license of a particular reader so it can be loaded on any computer in the school so students in math can take the assessment at one time.
And so the importance of that, too, is not just for assessment, but for kids to have the technology of where they're going to learn, which is in the regular classroom. If you can't use your accessibility software in a setting where it's needed, like general education, of social studies, of science, language arts, it doesn't get much good to in testing because you didn't get access to the general curriculum. But the student has to be fluent with use of the software so they can't come to the test day and that be the day that they're introduced to some kind of a software to use for technology support, whether its text reader or whatever it is they're using. It has to be well tested in terms of technology. We've had to test our test. What students with visual disabilities, win as a reader or e-reader or read and write go, whatever it is, we have to be sure it functions, so we have to acquire all those softwares and test them with our site. And then we find problems with Mac systems and PC systems, whether it's 8, 9, 10, or 11. You have to do a lot of test site evaluation on its accessibility. No matter what the tool that the students are going to be using to bring to that site. The other thing, we're going to live in a world for a long time of accessible assessment that is going to have to be somewhat merged with paper assessment. States aren't going to shift into online assessment overnight, so we'll be in a paper world and digital world. Part of this is because the technology is not there yet.
We have questions in math and science that can't be rendered for a student, so they have to revert back to their paper versions. That becomes complicated because we have students responding online where they can for the language arts and social studies and areas that are more text based, but when you get to formulas and things that are more complicated then they have to go back to paper. You end up with student responses that cannot be submitted online. Like to draw a graph, you can't always do that with current technology so you may have to have part of this on line and off-line so venders have to figure out, how do you reconcile these responses? What comes in online and what's submitted offline? So that's a complication we have to deal with for a while. Then if you're creating a web site you can't have a student come to day one and know how to use your web site, so we've had to create a practice site that mimics in terms of the design and operation and navigation the real web site they're using for online assessment, so the student can practice that web site in terms of sample items.
So when if comes to live assessment they will know how to use your web site and use those tools, whether they're students bringing unique tools like jaws to it, and they have to have a different kind of accessibility, so your web site design has to be prepared for those two types of issues and disability. Some students have keyboarding issues. They can tell us better in the use of technology, a fourth grader may not have the same keyboarding skills so they may need additional help, open response questions as opposed to multiple choice. A huge amount of work has had to go here into the fact that we're still converting from a paper world to an online world. So what we're doing on line has to, for psychometric purposes, exactly mimic and be verbatim of what you have in the paper world so students taking the paper test and students taking the online test are taking the same test. But we're not starting from an online creation. We're starting from a paper creation. We're having to retrofit it into a digital version, so therefore a tremendous amount of time gets consumed by quality assurance of being assured of comparability. Until we can get into all online assessment we've that problem, because with the universally designed test we're retrofitting from a paper world.
Test security is a huge issue, as was mentioned, as well. We had to do a lot with a our online assessment. There can't be hackers or save information. We have to shut out some of the tools that were mentioned, like if you don't need them to be using spell check or word prediction, or dictionary, those kind of things, the use of those tools while they're on line and give them to them and they are appropriate. So we've had to set our test up with an individual accommodation profile so the accommodation profile that is unique to what the students should use is set up for that student. You don't only need to have a test that's designed for a wide range of users, but then as individualized for that user when necessary, come into it so it matches their unique needs. And it's a very hard thing to do, but you know, it requires a lot of preparation. We have to deliver all of our assessments through sales, so that we know it's not going to be able to accessed for any public means. We have to be able to be sure the message is encrypted over the internet so that it can't be grabbed by somebody, and then our test content is no longer secure. We have to be sure that students are situated when they're taking a test, if you've got two kids taking fourth grade reading they can't be sitting next to each other and see each other's screens, that I’m marking A and you're marking C on question number 4.
So you have to arrange all the issues there of test security between students that may be taking the same test. The issue was mentioned, too, in terms of the tools that are used. Carol mentioned the need for the avoiding of importing of test content, so we've had to go, what we can find is the only reader we've been able to use is in almost all of our schools is write and write go from Textel, which has an ability to read in the window you're in without importing to any other window. So it reaches directly in the web-based window. Whereas with others you have to grab the text and import it into the program to read it. And our test folks from our office of assessment are certainly not at all very understanding about their test content being imported to any other window and possibly saved on a hard drive.
When you're through with the test we have to go through the procedure of everybody cleaning up their hard drive. Everything should be set up to be sure you don't end up unintentionally caching something on your system, but we have procedures for everybody where they go in and do a hard drive cleanup every day, to be sure that test items don't inadvertently remain somewhere that could be retrieved later and result in a security breach. So these are just some of the issues that we're faced in terms of live delivery of assessment. And it's very complicated and a lot of time and money goes into it, but again, in terms of what we've seen of student benefits it's well worth the effort.
Marty: Thank you very much, Preston. We appreciate that, that broad overview of what's happening in Kentucky, and a very well-thought-out listing of all of the complications. We remind those of you that are listening that if you would like to submit questions for our panelists during the next 35 minutes please do so at the bottom of the web page that you are currently at right now, there is a place for you to submit your questions. I have a question that you may have addressed, but I'd like to explore a little further. And this was brought up, Dr. Loew, when you were talking and Mr. Lewis, when you were following up. How do we ensure that students are not tested on their ability to interact with the computer, but rather on content? Other than a practice site, are there other strategies we can implement?
Preston: One of the things we've been doing is trying to task analyze all the skills needed for online assessment, and then to incorporate these into instructions, so that we try to remove all of those things, whether it's keyboarding or use of key functions, from the test environment itself. That's a long-range professional development issue, but if you don't analyze those skills to know what it is the student's having to bring into that test environment in terms of keyboarding or other computer skills, then you're right, you may be testing something differently. But I think that's still not too different than what we do now with paper testing. Where if we have a child who's taking a science test we're more likely to be testing their reading skills and some of their other comprehension skills than testing science. I think at least the technology, even though there are skills you have to be able to bring to the table to do that, they're not as complicated or as time consuming to teach them some of those skills. They're compensation skills as opposed to remediation skills, for how students can better and more effectively tell us what it is they've learned.
Ruth: I think the primary strategy that I’ve seen in the graduate and professional levels is likely to be test familiarization and practice materials. And when I say test familiarization, I mean that to be somewhat broader than strictly practice materials, because at the graduate and professional level there are information manuals, there's often correspondent between our offices and the test takers, so the more information we can exchange with the test taker about just what the test taking environment will be like, the more likely they are to know what to expect, to be able to prepare, and to be assessed based on, again, what they really can do as opposed to their familiarity with the technology.
Marty: Thank you. Carol, given the current state of technology from a vendor's point of view, how do you see accessibility playing out in the near future? The next 5 years, what challenges are you facing as a vendor?
Carol: The challenges we're facing, I think Preston Lewis stated them with quite, he was quite articulate in his statement, because those are the same challenges as we partner with states to address their online assessment needs of comparability of paper and pencil tests, that's truly a struggle for states, districts, schools, and our research scientists. We have to ensure that the, like Preston said, we are retrofitting our tests into online assessments. I see the future informative assessments as we grow the population of students to meet the higher standards for no child left behind, we need formative assessments so that that, what we assess in the classroom, and we have instructional links, immediate feedback, so that teachers can prescribe for individual students, whether they're typical students or special ed students, or across their entire school district, and throughout the state, and formative assessments could answer, could be the, you know, that answer. And having it convenient in your classroom daily would really be positive. In some states they're moving toward that right now.
Marty: Thank you. We have a question from one of our listeners; this comes from one of our listeners in Iowa. And the question is, specifically, what qualifies students in the K-12 system, specifically to Kentucky, to use online testing accommodation? I'd like to broaden that question, how do you determine an accommodation for a student who's taking a test online? Are there any differences between the online version of the test and the paper-pencil version of the test?
Preston: We have three eligibility requirements. One, they have a need for that accommodation in their IEP, 504 plan. Second, they have the need for the technology in terms of verifying the IEP, they have a print disability that they need this type of technology and they use this technology on a routine basis in the classroom. Again, it does no good to use this on the first time on the assessment. And the third eligibility criteria is they have access to practice area. If the student has not access to online assessment practice area then it won't let them into the live assessment because of the tracking system. We know it's in their IEP, we're supposed to have documentation they use this accommodation routinely in the classroom, and third, we have a way to know they have experienced our web site in the navigation tools, and they should be familiar with that. And that kind of gets back to an issue you just touched about, the issue of classroom assessment, because our eligibility for state assessment is once a year. But students need to practice accessible assessment year round. That's why we've got to look at how to create accessible assessment tools that are available for the teacher when she's creating a spelling test, in the fourth or fifth grade. Right now if you try to do it in a word document you put that up for a child to take it in the digital environment, it doesn't work very well. There's software that is created to help that be more acceptable and make it work appropriately, whether it's test taker or other software. But teachers have to know how to use that and create that practice all year round so students are familiar with that when they get to a point in time assessment, such as like accountability testing.
Marty: Thank you. And Dr. Loew, in the post-secondary setting, are there differences between accommodations for online versus paper-pencil versions of tests?
Ruth: Well, most of our tests that are computer based are not, in fact, on line as far as the test taker is concerned. So I’m not necessarily talking about online testing when I’m talking about computer-based testing. But again, the accommodations are typically decided on an individual basis, and in the post secondary level we're not looking at IEPs. So we are dealing with applications and student documentation. But typically we do have to look at some different considerations. For example, someone who uses font enlargement. Well, there are differences between just the manipulation of hard copy large print versus looking at the font enlargement on a computer screen, and we have to ascertain whether someone is, in fact, able to use both, relatively interchangeably, whether there are aspects of their vision that make one different from the other. So that's just one example. But we also have to look at the specifics of the test, and see exactly how materials are delivered, how much scrolling is involved on the computer screen, just what the physical layout is. Because that will affect how a student is able to navigate, and again, may relate to whether they can physically and cognitively navigate the computer layout versus the paper layout.
Marty: Thank you. A reminder for those of you who may have just joined us, that if you would like to submit a question for our panelists, please go to the bottom of the web page that you are currently on. There's a form there for you to submit a question. We received another one, this one we don't have a name or location. The question is this, “I was under the impression that for testing under No Child Left Behind it is required that testing follow instruction. If we are to test what is being taught, and we teach using assistive technology, why would we want to, or should we ever test differently? I think the question is, if you are giving instruction through assistive technology, could you test without the use of assistive technology, or should you?”
Ruth: I think this may be addressing a comment I made. And overall I think that one would generally want to test using the technology. There are situations in a teaching setting where you may want additional information provided for support that you might have to withdraw in a testing setting. So in order to ascertain whether a particular skill has, in fact, been learned. And the observation I was making involved things such as providing a student with support for reading by providing audio, or providing an image for a deaf student of someone using sign language. Which may be fine to support the learning of reading in an instructional situation. But if those are present during a reading assessment, you certainly would change what it is you're actually measuring. So that's up to a particular state, a particular test developer, a particular testing organization to decide just exactly what is being tested, and how the use of given accommodations impacts that. I didn't mean to suggest that one wouldn't want to, in general, use the technologies to which a student is accustomed, but there can be, there are particular cases where technology adaptations in the classroom may affect the assessment when you're actually trying to find out whether something has been mastered.
Preston: I think there's a much bigger issue that comes up, too, there, that you pointed out, which is if students are using this technology for routine instruction, and now they can more accurately, through use of these technology supports, whatever it is, read aloud, et cetera, more accurately, tell us what they've learned on a day-to-day basis, how can we ethically and conscionably come with another assessment environment, especially for state accountability, and say no, now you have to have adult support, you have to have, you have to be dependent upon someone else to help you? If we know this technology more accurately, or at least more non-discriminately allows a student to indicate to us what they've learned, and they become accustomed to that on a daily basis, then it just doesn't seem like it's an equitable situation to come to state assessment and deny them the accommodation and - and you understand what we're measuring versus use of technology revealing something different, like a reading test. But if a student is taking a science test we're not measuring reading, and they've learned more in science because they have an accessible curriculum. We have to have that available to them. On accountability, I can perceive this as being the expectation because of the fact that otherwise we're not being fair to the students.
Marty: Excellent, thank you. Another question, this comes from one of our listeners in Pennsylvania. The first is a comment, “Great information for everyone, thank you. This is a question specific to you, Preston. When you surveyed and tested the various software and tools, did you do it in a formal testing lab, or was there input from various stakeholders involved in the project?”
Preston: Yeah, we had input from a variety of vendors, whether it was folks from Mac or from PC environments, Microsoft, or from other vendors such as Textel, we, of course, had to have the people who designed the web site itself in their lab environment to do all the QA test and make sure things are working properly. In our case we happened to contract with Denver for this purpose. They deliver our Kentucky virtual high school, and they understood accessible web site design. So we took our paper test and worked with them in terms of the conversion from our test vendor to folks in Denver who did the conversion delivery, and then we identified all the possible range of tools that would be used by students, and for them to then bring those into the lab and test them well before the test time, so we could be sure that things were compatible with the web site.
Marty: Thank you. Another question from a listener in north Carolina. This is asking a question about a specific piece, or excuse me, I can't speak, a specific application. The question is this, and it's to any of you. “Have you been able to effectively use talking web browsers or tool bar applications, such as the read please reading bar?” I don't know if anyone one -
Preston: Read please has been used. There are some issues with read please, it's great in terms of free reader but it doesn't bring all the tools that you sometimes need. Stay student issues or other accommodations needed on a tool bar. The other issue with read please, and I’m not familiar with it, it's a digital electronic voice, whereas with a more current, and the more commercial type readers, it provides you a real speak quality, a much more intelligible voice, and it's very important that when a student is reading a question they get the most intelligible reading as possible so they can understand what it is that they're reading, as opposed to that kind of electronic digital voice. So that's been one issue that we've had in relation of the use of some of the lesser expensive products such as read please. But for classroom teachers who can't afford anything else and just need to provide reading support, that's a great option.
Marty: Okay. One of the questions that we've had that sparked this whole web cast was what should be a balance, if any, between developing computer-based test models, developing them so they're natively accessible, or should we be developing tests that can be easily accommodated? The question, then, specifically being, what should be the balance, if any, between native accessibility or universal design, universally-designed materials, and test accommodations?
Carol: Well this is Carol, I can start that one off. Because having been in education for over 20 years, we went from seeing tests that were, you know, black and white, quite simple, no pictures, and boring, to the let's have our tests look like--and I’m talking like a teacher's perspective--like our readers and the work books the students use every day. Let's put pictures and graphics, and more, make them more user friendly. Let's add color to the test, whether it's a shelf product or a custom test. Now with universal design, the tests are looking like those original tests that we took back when we were in school. And I imagine for online delivery, the more universally designed an assessment is, the ease of reading, you have trouble with text readers if you have lots of graphics or designs that are complicated, or like the cross-section of a hurricane in a science test, it's hard to put that type of item that requires a constructive response onto a web site, or a computer-based test delivery system, and have a text reader read it. So I guess, from our perspective, that's difficult.
Preston: I think that's a big question that goes to the issue of universal design too. If you look at the classic example that we all benefit from curb cuts, whether we're trying to roll our suitcase down into the street or riding a bike, you can use a curb cut. That's an accommodation. And there are accommodations as well in the assessment domain that all kids can benefit from. So I think we're going to have to, down the road, begin to look at, you know, why are we limiting these accommodations to kids with disabilities? We have kids who use a text reader in a classroom to construct a writing portfolio, and then have that text reader, these are kids who can read, high achievers, who can then have that text in their writing portfolio read back to them and listen to what they've read. Then they can critically evaluate what they've written by listening to it, and use the technology to make efficient changes. They're using that accommodation. They don't need it for the same reason, but they benefit from it. So I think we're going to have to, down the road, look at the larger issue of these accommodations and how the universal design issue is not a disability issue, it's an issue for all persons who can benefit from technology, regardless of what they may or may not need on the basis of a particular disability problem.
Marty: Dr. Loew, from your perspective, with adult, or post-secondary individuals, is there a balance between, or should there be a balance between a native accessibility and accommodation?
Ruth: Well, certainly there should be, and I think Preston's comments were very well put. I'd like to play devil's advocate here a little bit and once again bring up the issue of validity and construct. Where absolutely things are currently viewed as typically as accommodations can be used by the rest of the world. Just because someone might need, say, to be able to use voice-to-text software because of a disability, someone who, say, because of a motor disability can't type but can use a software to recognize their speech and render that as text, I know people who just use it as a matter of preference. But on the other hand we have to think through exactly what it is that we're trying to teach, and decide globally whether the use of, say, speech to text, or text to speech, or any other of the adaptations we've been discussing, how those really impact what we're trying to accomplish. There are many, many cases, I would agree with Preston, where these adaptations can be used successfully by the rest of the world, by all of us, and there are also cases where they could undercut what's being taught. And part of that relates to how we redefine our educational goals in a technological world. Just what does literacy mean? What do we want literacy to mean?
Marty: Thank you. Again, a reminder for those of you that would like to submit questions, please go to the bottom of the web site, web page that you're on and submit your questions. Another question, we do not know where this person is from but it's a good question. One of the presenters mentioned the difficulty of retrofitting print materials to an electronic format. The question is this. How might the National Instructional Materials Accessibility Standards, or NIMAS, which is part of the Endividuals with Disability Education Act of 2004, how might that help us retrofit print materials to an electronic format with regard to testing? They've asked a specific question, also, of CTB McGraw Hill, what steps has CTB McGraw Hill taken to provide accessible alternative formats?
Carol: CTB McGraw Hill is very well aware of the NIMAS, and that's in our publishing department, and I’m a development person, but we're aware of it. I know corporate, for McGraw Hill, they are working very closely with the--I don't know--the housing vendor for NIMAS with the GL Nextco products. And that's basically books to print. And I think NIMAS came out of, if I’m not mistaken, so students with disabilities receive their textbooks and print materials in large print at the same time that the typical child receives theirs. And this warehouse, or this clearing center would produce the books so that the students, the end result is that on the first day of school all the students have the same. It hasn't gotten to the assessment part yet, because a custom assessment is a little bit different than publishing a book. But we deliver our custom assessments, the students get their custom assessments at the same time. So whether it would go through this clearing house or not is not - I don't know that, and that wouldn't be my decision, that would happen in our publishing offices.
Marty: Good, thank you. Another question, this comes from a listener in New Jersey. “What are the challenges that the panel members believe that test takers with learning disabilities may face in taking computer-based tests? Besides more testing time, what accommodations are afforded these individuals in a computer-based testing world?” Anyone want to attack that one?
Preston: Well, I think students with learning disabilities bring a lot of the same print disability issues to the test environment that other kids do, and there may be some unique things in terms of learning disability and how a child is seeing something on the screen versus how another child is seeing it. I saw something recently about text creation in a way, for instance, that is more conducive to students who are reading because of dyslexia, how to construct a text environment that is less likely to pose a problem for a child with dyslexia. Maybe that's something Ruth can tell us down the road. But I guess the ability of the child to selectively use tools they need because it may - for instance, they don't need to read a whole page, but maybe they do get to certain words or certain phrases which give them a problem, so then they can selectively go in and pick that word and go to more information, whether it's dictionary or whether it's a thesaurus or word prediction, for them to uniquely and personally select what they need to get better information to understand what it is they're trying to read and more accurately portray or convey what it is they need to say about that particular item.
Marty: Good, thank you. Another question. Well, does anybody else want to - any of our other panelists want to address that question?
Carol: No.
Marty: That's fine. Next question. This comes from a listener, I’m not sure where, but the question is this. “Besides students interacting with graphs and drawing images, what are the other barriers that need to be tackled by online or computer-based test designers, and assistive technology designers, to get closer to universal design?”
Preston: One of the things mentioned earlier is we bring some of our tools to the assessment site itself. For instance, you might have read aloud or some other tools built in where your site is already speech enabled so you don't need to worry about the individual's tools and what they bring, but you bring accessibility supports into your design. So that that's a part of it. It may operate a little bit differently than the tool that they're used to, but if they go into a practice area and get access beforehand they can learn how your accommodation tools inherently built into your site are part of that presentation and know how to use them before the day of test time so they can accurately and appropriately use them when it comes time for the live assessment.
Carol: I have two barriers. One barrier is there's no common set of standards, or a universal vocabulary across all states in terms of technology, even assessments, when we talk about benchmarks and standards and competencies, different skills, different states use different things. We deal with that easily as a vendor, and you adapt it for each individual client that you have. But in technology, a universal set of standards and definitions would really help communication. Because sometimes when you're in a conference with a state, and they're saying one thing and using the same words that you may use for something else, the communication becomes a barrier there to deliver what they really want. That's one issue. And another one is if, when you look at computer-based assessments in terms of formative assessments, not linear assessments but formative assessments, we need larger and larger item pools to sufficiently deliver formative assessments on line. So no online assessment would require a larger bank of items than would a paper, a typical paper and pencil assessment.
Ruth: I'd like to address another class of test questions that can create issues. People tend to be very familiar with the issues raised by graphs, and other graphics, both understanding them and being able to generate them. But another category that can create problems is anything that's heavily mouse-dependent. Because not everyone is able to use a mouse. Certainly there are students with motoric disabilities who cannot, and of course blind students would have difficulty mousing, because you have to be able to see what you're clicking on. There are some items created for computer-based assessments that involve such tasks as clicking and dragging, so that you move a sentence to the right place in a paragraph, or move a word to the right place in a sentence, or take a sequence of historical events and arrange them in chronological order, or take some sequence of ideas and arrange them in order of importance. Try to envision any of these things being done solely by audio. They're miserably difficult to adapt to audio. A straightforward screen reader simply can't read them intelligently. But just trying to come up with an alternate version that a screen reader could then read is extremely challenging. So I think, on the one hand, if we just take paper-based tests and, you know, stick them up on a computer screen, we're not really taking advantage of the capabilities of the computer, or working with the technological world that students now live in. But on the other hand, if we develop test questions that are, that only make sense in a computer environment, we do run the risk of creating questions that aren't amenable to use for some students. And I think these interactive mouse-based items are kind of a classic case.
Preston: I think an issue related to that is also the size of graphics. Where you've got to be sure you don't have a graphic coming in that's going to delay that question coming in to the student, or delay that response. Our system is set up when a student is taking question number 8, and question 9 is preloading on the server, so that we have that coming in. And the student hopefully won't have to wait for that next question to have to be able to come up and read. But you have to look at the size of a graphic that might be attached to that, like a science or math question, figure out how to give the accuracy of the graphic but not create a graphic so loaded they're going to be sitting and watching and watching and watching that to come up on the screen, and get frustrated and then as a result that affects their response.
Marty: This is an excellent discussion, and I apologize, we still have several questions here that we're not going to be able to address. We would like to give each of our presenters a minute or so to make any closing remarks based on what we've talked about. We would ask Preston, then Ruth, and then Carol to make any concluding statements that you'd like to make.
Preston: I just want to thank you again for the opportunity. I think this is a very timely discussion and important discussion. It's a very big issue, again, in relation to not just meeting requirements of NCLB, but one of overall accessibility for individuals with disabilities in terms of learning and measuring what they've learned. In our post-test evaluation last year students taking the testing on line, almost 90 percent said they thought that they performed better by taking their tests on a computer. We don't know if that's true. Taking the test on the computer itself doesn't mean you know more of the right answers. But if it does mean that because of what you're doing in the classroom with accessible instruction that's where you're getting successful assessment in is more likely to mean we're allowing kids to get access to learning so they can better demonstrate what they learned when it gets to assessment.
Marty: Thank you, Ruth?
Ruth: I agree that this was a very timely and very interesting discussion, and one point on which I'd like to close comes from my background as a test developer, test question writer. Which is that the consideration of accommodations and of universal design really pushes us to think, think more clearly about what we're really trying to test. There are lots of assumptions of standard paper-based testing that I think weren't really questioned. You know, this is how you present the material because this is how you've always presented the material. And when we start trying to present material in different ways for different populations, and try to figure out what adaptations are appropriate for a particular population, we really have to think about what it is that we want students to learn, and what it is that we want students to - what it is that we want a test to measure. So the whole process of thinking through what's appropriate for students with disabilities, I think, leads to better thought out and better designed assessments for all. I certainly hope that's the case.
Carol: And I agree with Dr. Loew, because if our mission is to help the teacher help the child, we embrace the goals of the state, the districts, and schools, and work closely with them to achieve accessibility for all. And Marty, I would like to personally thank you and NCDAE for hosting these presentations, and I appreciate the opportunity to participate.
Marty: Well thank you very much. We appreciate Dr. Ruth Loew, Mr. Preston Lewis, and Ms. Carol Price for your participation in today's audio web cast. In mid-June we will hold another in our series entitled "Accessible Distance Education Technologies and Techniques--Can It Be Done?" We will be reviewing the accessibility of distance education systems used in university and secondary education systems and will include panelists from industry and education. So we will have a developer and a user perspective. In late summer or early fall we will host a web cast, dates yet to be announced, on libraries and accessibility of library resources. Visit our web site for more information and daily updated news regarding accessibility, technology media, education, and distance education. We appreciate your participation today. Thank you very much.