- Alright, seems like a good hour. Let's get this started. So, welcome, everyone. Welcome to today's NROC webinar. My name is Ahrash Bissell, I am the president at NROC, and I'll be facilitating this conversation with my guest, Peg Russell, who's the Regional Director of Adult Education and Student Testing at Gateway Community and Technical College in Kentucky. And, as you can see, the topic today is gonna be "Low Stakes Placement" where we want to assess and address student readiness. This webinar is being recorded, so everybody who registered, including all those who are here live, will get a copy and a follow on email with a copy of the recording. I think I have some info on that. And, so you'll be able to access that at any time. For those of you who are here, we certainly encourage as much interactivity as you care to provide. So, if you have any questions, comments, use the Q&A area, pop things into the chat. There are a number of us on the call behind the scenes, and we'll try to respond either in the chat or I will grab questions, and then try to answer them live, or Peg will, depending on how they're directed. Feel free to also tweet about all of this at NROCpd. And, if you need any help as this unfolds with audio issues or whatever else, feel free to email and we will try to help you. So, with that, let's dive in. So, I am going to provide a bit of background... Excuse me, in both, both in terms of NROC as an organization, but then also the specific body of work that led us to this national effort that we're engaged in to really change the way we do placement. And, then you'll get to hear about a particular case here from Peg. So, the NROC Project, we are a nonprofit organization, we are celebrating our 20th year this year, and it's been a fascinating journey of which I've only been a part of it for a little over half. So, it gives you an idea of just the depth of our work and the numbers of different endeavors that we've engaged in over the years. But, it really started off with a focus on OER, Open Educational Resources, and the new affordances that technology, in particular, the internet, give us to distribute stuff easily and virtually cost free, and also to manipulate things. And, that leads into questions about, well, can you make things that are more personalized, that are more adaptive, more modular, et cetera? And, so that work morphed into not only distributing material, which we do through a site that we still host called Hippocampus, but creating material. So, starting with NROC algebra 1, moving into developmental math, developmental English. These are course materials that are high end, multimedia, very engaging, designed for customization and localization. And, then to facilitate the adoption and utilization of these materials in interesting and effective ways, we built a platform called EdReady for personalizing and adapt... Personalizing the learning experience using an adaptive learning model. Since the launch of EdReady, most of our emphasis has been on scaling. We have tried to engage more and more partners all across secondary, post-secondary, adult ed, what have you, to think differently about how we support student readiness, how we support their success and their persistence, what are the key variables here, and how do we interface with the programs on the ground, the populations we're trying to serve, et cetera. So, just to reiterate, EdReady is an adaptive learning platform that's been adopted by district systems and states to personalize math and English instruction and to empower data-driven student success decisions. The technology was purpose built to not require a one size fits all. The whole point is that institutions vary, populations vary, programs vary, and so you need tools that allow you to accommodate that variation, but, nonetheless, push everyone forward, right? At some scale. And, that's exactly what EdReady does. That means that EdReady's used in a lot of really interesting ways. We see EdReady deployed in all these different ways you can see on the slide and to support different programmatic needs for different kinds of institutions. And, it's a significant part of our work as an organization to partner with our institutions to help them make sense of the possibilities, find an effective deployment model, and get it out, and get it used, and start helping students. With that as a background, I want to dive a little bit into the logic behind why there's this focus on placement. Placement practices have kind of followed a very similar model for a long time. And, more recently, there was quite a bit of research that demonstrated why a lot of those practices are problematic. The evidence is that many, many students were being misplaced, and that could happen in both directions, where some students were being pushed forward that really weren't ready, but also a lot of students were being held back and that wasn't to their benefit. There's also just the challenge of the cost and the way that it not only costs additional money often, and resources, but it also costs time. And, these are things that all accumulate to limit the efficacy of traditional placement practices in ways that really didn't help students and frankly didn't help institutions either. So, to kind of zero in on where is this problem really reside? Like, why does this problem exist and why couldn't we figure it out? I'm gonna walk you through, quickly, this logic model that we developed that I think leads very clearly into what you'll then hear is put in place in Gateway. So, placement... Ooh, I did not want that to launch anything. The way that placement is traditionally done is we give students a placement test. A placement test is what we call a high stakes measure. The whole point of the test is to ascertain if you have a student who's trying to enroll in your institution or in your program, and you're not sure if they're gonna be ready, meaning you're not sure where they stand in terms of their, in particular, core math and English skills. You just wanna find out, you treat them as sort of an object, where you're wondering are they a strong student? Are they a weaker student? Can we kind of figure that out with a test? And, a test is a traditional instrument for doing that that made some sense. These days, we see a lot of growth of multiple measures. So, more and more, instead of testing, people are using grades. They may be using other metrics that are available. At the end of the day, it really doesn't matter how you obtain a number, if that number is definitive, if it just simply puts a label on the student, the same problems ensue. The way this works is, whether it's testing, or grades, or whatever, we decide that a student is something on some scale, and let's just pretend it's zero to 100. So, a student that gets a very high score would be deemed a strong student, and a student who gets a low score would be deemed a weak student. And, then there are students in the middle. If you plot a sample of students, all of whom you've measured in some way, you will get a bell curve. That's just inherent in the way the statistics work when you do a sample like this. And, with this bell curve, the way you interpret it is that there's some number of students, a relatively small fraction, that are gonna end up being very low in the score range. And, there's some small fraction of students that'll be very high. The vast majority of the students, assuming the test is remotely accurate or the grading is remotely accurate to assessing the full range of ability, they'll be somewhere in the middle. And, the problem now is that if you're choosing this sort of an approach to decide who's ready and who's not ready, that's a binary. That means you need to decide, well, where is that place? Where is a student too... Scoring too low, so that they're not ready, versus where are they scoring high enough where you would say they are ready? So, here's not ready, and then they are ready on the other side. Where you place this line is hugely important. Hopefully, that's kind of obvious, right? If you push it too far to the right, then it means the number of students you're saying are ready, it's gonna be pretty small, because relatively few students are gonna score high enough to get past that line. And, that means most students are gonna be deemed not ready. And, now there's something you have to do to them or with them to help them succeed. Similarly, if you go the other direction, and you make that line too low, now a lot of students are gonna be pushed forward, but, in truth, quite a number of them probably aren't truly ready. And, if you're just ignoring them or pretending that they're ready, then they're gonna struggle, they're probably gonna fail, they will probably drop out. And, so is there some answer? Is there some place that you can draw that line? And, the answer is no, right? It actually is not possible to determine a cut score that makes any sense, because the exercise doesn't fit what we're trying to achieve. The way we represent that in our modeling on this is we recast this... This sort of spread in this manner, where we again use score on the X-axis, so we're still plotting score, which is the test score, or grades, or what have you for each student. But, on the left, we're looking at our confidence in the result. So, how confident are we that the number we have is an accurate reflection of that student's abilities, if you will? And, what we can say is, if a student does really poorly, it's unlikely that a really good student is gonna completely bomb the test. And, so we can be pretty confident that a poor score is representative of somebody who probably doesn't have strong mastery of the material. Similarly, if a student aces the test, that's hard to do if you don't actually know the material. Even a barely adequate test is probably gonna find the students who are really strong. And, so we, again, we're very confident in that result. Anybody in the middle, we don't really know what to say. And, that's the problem. That whole middle zone, what we like to call the murky middle, is really difficult to decide. Is a students strong? Is the student weak? We don't know. The student's some mixture. That's also, as you can see with the gray background, where most of the students reside. So, the majority of what we're trying to do resists the reality on the ground. Like you can't really use a test or a single metric to, in fact, figure out who's ready and who's not ready in any kind of way that you would have confidence in. So, we wanna shift this narrative. Let's stop trying to simply bin and sort students. Instead, let's take the framework that our job, as educators, as administrators, who are focused on and desirous of student success, is to make everybody successful. Is there a method here that not only helps us ascertain relative readiness, but pushes every student forward, right? Gets every student to the place of readiness without sending 'em off to developmental education courses and that kind of thing. And, so that's the nature that's... That reflects our talk title, right? We want to assess and address readiness. We wanna actually move the student success curve. And, fortunately, a technology like EdReady let's us do that and let's us do that at scale. A quick review of some of the things people may know or not know about EdReady is that EdReady has a kind of, you can think of it as almost two parts, it's actually got multiple parts, but it's easier to just break it into two, there's what we call the initial diagnostic, which you can think of as being akin to a placement kind of a test. It is intended to provide at least a basic overview of whether a student is relatively strong or relatively weak across the learning expectations that are defined by the institution. And, based on that initial diagnostic, you're gonna get a first breakdown of where students are strong, where they're relatively weak, where they're somewhere in the middle. That, once that's been given though, we don't generally stop there. So, there are cases, and I believe that in Kentucky they do do this, where the diagnostic alone can tell you enough information that there's something, some action you can take, and, specifically, if students do really well, there are places that will accept that much like you would in any placement test, as evidence the student is strong enough that we're not necessarily that worried about them. For the vast majority of students, what you're then gonna wanna do is move them into the study path. So, yes, you have an initial sort of overview, but the job now is to get the student fully ready. And, EdReady does that by building them a personalized study path. It presents all the different material that's been tested and kind of where they stand, and then they work their way through the material over time and they move that score up. It's a personalized approach. So, EdReady zeroes in on the stuff they really need to be studying, because we can see that they don't have mastery. And, then there's this recursion period where they learn material using all the onboard multimedia, stuff built stuff, that's provided by partners, and then they get themself to the benchmark that qualifies them as, yes, you are now ready for the program of study you're trying to take. These materials are high end, they're multimedia, it's presentations, there's worked examples, there's an embedded textbook, et cetera, et cetera. Something I was asked to point out, which just dovetails from what I said, is that in Kentucky, there is a... There's a set of criteria that are used to determine whether a student can just go straight into credit bearing courses or not. And, so you can see that EdReady is included on this band, this is a reflection of that either they hit the benchmark right away, meaning they did really well on that diagnostic, or they worked their way through the study path to get to that benchmark, and that's what qualifies them to move through. So, with that as a backdrop, we get to hear about, from Peg Russell, about what they're doing at Gateway, and what specifically she's doing with her program, and so I'm gonna hand it off to Peg. - Great, thank you, Ahrash. Hello, everyone. Just a little bit of history, and I know we have some Kentucky folks on the line, and one who was really instrumental, I think I saw Shauna King Sims signed in, and she's with our system office, and has done a lot of work over the years with assessments. So, she'll have some great information to drop in the chat, I'm sure. We've been an EdReady College, fully EdReady, since 2018. Many of you probably go back to the days, depending on where you are in the country, where Compass or Accuplacer were mostly the options for placement at two-year institutions, which is where I am. In Kentucky, we are a united system with 16 colleges spread across the state. I'm at the northernmost campus, which is directly across from Cincinnati on the Ohio River. We were a Compass College for a while, and when Compass decided... ACT decided to pull that product, we knew we were gonna be looking for something else. So, as a fill in, we use the TABE, or the Test of Adult Basic Ed. And, if you're familiar with adult ed, you know that that's pretty much the one and only test that's accepted in many of the federal programs. We also had a homegrown version of a test called KYOTE, which stood for the Kentucky Online Test. And, we used a version of those two until we came together as a system, and this is something I can't stress enough, the use of EdReady and the development of this instrument was driven from the faculty up. So, you had subject matter experts coming together in all of our different areas, so the very first version we developed of this placement test using EdReady was we field test it, we did a pilot, we came back together probably six or eight months later, and decided we might need to scale it back a little bit, but it was a really good experience, and I believe it's as accepted as it is, because faculty were involved from the very beginning, and they've been very accepting of that. Little bit of data about the number of students we test each year. Obviously, during the pandemic, we began relying a bit more heavily on GPA for placement. And, that's been probably more of a sell, at least at my institution than the use of EdReady, since there's not a lot of standardization with GPA, but we do use it, and we will continue to use that. So, next slide, please. I won't read the slides to you, because you can see that. I put these here so that you can... It makes a little bit of sense now that you've seen the reporting structure and you know the difference now between not ready, meaning they scored below 50% in that subject area, or needs review, they were 50 to 99%. You can see our data, and this might be a month or two old, but you can see that of the 892 students, 58% used their study path. So, that's that diagnostic tool that is given to every student once they complete the initial placement. And, for us, we only require one test. They don't come back, they don't retest, unlike with other tests, where we would allow them two or three times to retest, there is no reason to retest the way we have set up EdReady, and I wanted you to see the mean diag... Or, the median diagnostic score, the study time. And, those areas shouldn't surprise any of you as the areas where folks needed to review. Those are things, whether you're a GED student or a student attempting to enroll in college, those are pretty common areas where students need to do some review. Okay, next slide. Everybody's dreaded area of math, whether you're taking a college class or you're working on your GED. The nice thing about the math test is we do have a basic math test, which will place you into several courses in our course catalog, but we also have what we like to call a challenge or a calculus test. And, probably, as many of you at your institution, you're doing dual credit, you're working with a lot of high school students, or we are certainly in the state of Kentucky, that continues to increase every year. And, many of those students are ready, have the skills to go into the higher level credit bearing mathematics that require a calculus score. So, we don't have a lot of requests to administer it. But, this time of year, in the spring, when folks are signing up for fall courses, we're often asked to administer that calculus test. And, again, it was developed between the good folks at NROC and between our math... Our college math faculty. you can see the number of folks using their study path. Again, the personalized program. They're spending more time studying in math than they are in reading and writing. And, again, probably those areas that need work are not surprising. The one thing, and Ahrash went through, and you saw the bell curve, and you see it moving left and right, I will say that by using EdReady, it is very rare that we have a student test so low that they can't get into some type of course in the KCTCS catalog. It may often be with support or what we used to think of as the old developmental ed classes, but all of our institutions offer gen eds paired with support courses. And, for the most part, the majority of our students are able to place into credit bearing work, which is a great thing, because we know that that often helps with retention and leads to completion. So, if we go to the next slide. I ask my good folks in institutional research, in the next two slides you'll see our data specific to Gateway, and the first question I asked them was, is there a difference in GPA based on the scores that are used for enrollment? So, they were so kind and ran my data. You can see the largest in of that population was ACT. And, Kentucky is very much an ACT placement state. In fact, every junior and high school, it's mandatory for them to take the ACT, but you can see that our EdReady students actually did as well or better than students who were placed with one of the other tests. So, we were really encouraged to see that. Next slide talks a bit about... And, these are brand new stats, so I know Angie's on the line, and she didn't get to see this slide when we made this presentation the last time, but you can see that retention actually looks really, really good based on the numbers, and the number crunching done by our institutional research. Whether you're looking fall to spring, fall to fall, students are persisting, that are enrolling using EdReady as their placement. Again, good news for us. So, if we could move forward please. I wanted to talk to you a little bit about how we implemented the process, not so much about data, but really about how we took this process, and made it our own. And, that's the nice thing, this... I would imagine if you step foot on any of the other KCTCS schools, it may not look exactly the same, but we were all in from the very beginning in using EdReady. So, we put a lot of time, and thought, and then the pandemic came along, and we were so thrilled that we could continue, we made a few tweaks, but that we could continue to use this product, so that students could get enrolled, whether they were high school or whether from the community. And, we didn't miss a beat during the pandemic, it was actually great. So, our process is we allow students to really handle this registration and testing pretty much on their own. We use something called Register Blast, which is a national service that you can buy. And, we have a direct link to that on the Gateway website, where after they've applied to the college, it says, alright, if you need a placement test, click this giant blue button that says "Register Blast", and we'll walk you through the process. We also were fortunate enough, several years ago, at Gateway, to receive a multimillion dollar First in the World Grant. And, it allowed us to do many things on our campus. But, one of the very best things we did and is so helpful to our students, is we establish what we call an Information Commons at all of our locations. So, if someone's uncomfortable completing this process on their own, they can stop by any desk on any one of our three campuses, and the folks there will walk them through the registration process to sign up on their own. I put in capitals that next point, because we were one of those colleges that believed, man, you had to come to campus, and I have to watch you take that test, and you can't do it any other way. And, you know what? During the pandemic, we all realized that a lot of things were gonna change, and we have to... Had to shift our thought process. We implement this test fully online for college placement, fully online. We test four days a week, including Saturdays. Three of those four sessions are non proctored, and I'll explain how that works. And, one is done through Blackboard, where we send the students a link to Blackboard. They don't have to be a student, they don't have to have a Blackboard account. We've created a permanent link that we can then send to the student. And, we've found that that one is very popular in the evenings and catches a lot of our students who may be non-native speakers, who are first time students, and just need a little more support. We have a really great process in place. I run the whole testing process as a part-time duty that I have with Gateway, because, as you saw on the very first slide, I am... I'm the Director of Adult Education, which takes a tremendous amount of my time, and I love, but I also love keeping my hand in into assessment. So, I have two very part-time proctors and myself run the entire testing process for Gateway. So, my proctors will go into Register Blast, they'll pull their roster for the day, they'll say... They'll see who's registered, and they reach out to those students, and send them a greeting, hey, we see you're signed up to take the EdReady. Welcome to Gateway, we're gonna do everything we can to make this a painless process. And, in that email, we send them the instructions, so that they can go in and create their EdReady student account, because they're gonna need that, and we have a lovely flyer that we send to them, and it's really pretty foolproof, and we found that folks don't have any problems, so that, next day, when the test rolls around, they are ready to go. So, if we could advance the next slide, please. So, there's a little bit of background work required to make EdReady work at your place, but nothing like it was with Compass, with TABE. It is really wonderful, and once you've got it set up, you simply go in and create test sessions, and the goal key or the code that you see referenced there is sent to the student, and you can sort of think of that as their test ticket, that they cannot get to the test or open the test without having that code. So, the morning of the test at 8:00 AM, my proctors will send that email step by step with their codes. If they're taking the reading, writing, and math, they'll receive two codes to get them into the test, and a direct line, email, or phone number to contact the proctor if they run into problems. And, I have to tell you, they don't. I was really doubtful. I worried that there would be way too many technological issues, that students wouldn't get it, but it is rare that we have an issue where a student can't create their account, can't log on. The biggest problem we have is that they often forget their password to the account. But, otherwise, it works beautifully. I have given access to the EdReady database to everyone at Gateway who could possibly use it. The more people that have access to the information, the better it is for the student. So, my proctors recover scores the same day. We're a PeopleSoft College, so we've set up a Dropbox in the registrar's office where those scores go, and, literally, they're normally within PeopleSoft in just a few hours, so that your advisor can see them. If you're an apprenticeship student, your counselor or advisor can see those. We want everyone that needs that information to have access to it. They can't launch tests, they can't create tests, but they can all go in and see how a student did, and what their next steps are. We also, that same day, when they complete the test, we send them a next step letter that says, based on that chart that you saw for KCTCS placement, we have one letter that says, "You did an amazing job on this test. You are ready to contact an advisor." And, we have a link in the letter and in the email, so that they can click on it and immediately make an advising appointment. But, we still say to them, "You still have a diagnostic plan out there. If you wanna do some review just to help get yourself a little bit more ready before classes start, go in and work that study plan, but you don't have to." By far, the largest majority of our students fall into the middle category, that we kind of called murky on that bell curve slide. And, those are the students are able to enroll, but they would need supplemental instruction. So, we say to them, "You have two choices here, and you are in the driver's seat, you can enroll with your current scores, and just know that the advisor will also put you into a paired course, or you can work that study path, bring it up to the level where you won't need paired courses, save yourself maybe some time and money, and enroll directly into credit bearing work." Or, very few students fall into that third category where we say, "You're gonna need to do some review and bring your score up X number of points before you have an option to enroll." And, I... One of the nice things is that also at the Information Commons, we have dedicated admissions folks who are listed in the information letter as someone they can call or email if they need help getting into their study path. But, along with that letter, they receive another really nice handout that says, "Here's what you need to do in order to work your study path. Here's how you log in, here's what the screen's gonna look like, and here's your process." So, we really try to make it as painless and as simple as we possibly can. So, next slide, please. - [Ahrash] Before we move on, Peg. - [Peg] Yeah. - [Ahrash] We do have a question. - Okay. - [Ahrash] Someone's asking, "How many of the students testing speak English as an additional language, and how successful are ESL students in navigating the online testing system as you've set it up?" - We actually, the majority of our students are native speakers. We do have ESL students, and that's something that's near and dear to my heart. because that's part of my programming in adult ed. We find that our non-native speakers, and we use this in adult ed as a knowledge recovery tool, they don't... They don't have... Let me step back. We use CASAS as our assessment for non-native speakers for adult education, not for credit bearing courses, but we do expose them to EdReady as a study tool. And, I can speak from an adult ed perspective that they don't struggle with it. I would have to ask institutional research to filter that data. I will tell you, we do have non-native speakers take the exam, and, by far, they choose the proctored Blackboard session,, and my proctors report very few problems with those folks. - [Ahrash] That's great. Thank you. Move on. - Alright, I think the benefits have been pretty clear. It has been a really wonderful tool to work with and to administer. I cannot tell you how many parents and students say to us, "Thank you for not making us come to campus." We have really tried to make our process online from beginning to end. A student really doesn't have to set foot on our campus at all, depending on what credential and what program they're looking for. Everyone can see those results, so that you don't have any bottleneck waiting on scores. Advisors aren't sitting there waiting on things. But, the key piece here is you put the student in the driver's seat. They decide, how am I gonna do my enrollment? If I don't have GPA and I don't have ACT, I'm using EdReady. I'm either gonna work that study path and I'm gonna try to get as high as I can, or I'm okay taking those paired courses. But, the decision is always left up to the student, which is a really great thing. Okay, the next slide. We, as I mentioned, my other role, my primary role is adult ed, and we are very fortunate in Kentucky that we... Kentucky Adult Ed also has a relationship with NROC, in addition to the partnership with KCTCS, and have developed specific study paths for students who are seeking their GED. It's a beautiful program. It works incredibly well. Many times we find that some online learning platforms designed for adults are a bit childish, and folks are not real thrilled in working in those platforms. We... That is not the case in EdReady. So, it's a popular option with my teachers. And, if you work in programs that are grant funded, you know you have to document every single thing. And, EdReady reports easily allow us to look at the amount of time a student has spent on a task, what they've mastered, how many hours they've put in, so that we can enter that into our adult ed reporting system. Next slide. Just a quote from one of my teachers that loves the use of this for math, they love Hippocampus, Khan Academy, and they think it's a really great supplemental tool, and they love the reporting feature. They're always, always looking for ways to make their life easier as they administer instruction. And, the reporting feature for EdReady is really exceptional and very easy to use. Okay, next slide. Next slide where we might be at the end. - [Ahrash] This is another testimonial. - Oh, sorry. Yep, just another, and you can certainly see that. And, for some folks, and I think that last sentence is important about the gaps in knowledge. We often hear in adult ed, students will say, "I feel like a piece of Swiss Cheese. I know I should have known this, but I just can't recall it or when I learned it. And, we found that this tool is not intimidating, it is user friendly, and it really does help to plug in those gaps." And, a really nice thing is that one of the measures by which we are judged in Kentucky Adult Education is how many of our students go on to transition to college or to the workplace. And, once a student has begun working in EdReady with us in adult ed, it looks very familiar to them when they sit down to take the placement test and to do the diagnostic study path. It's not frightening for them. And, it's... That's been somewhat of a game changer for us. So, I think we're probably at the question stage maybe, or I guess we're gonna-- - We pretty much are, yeah. - Turn it over to Jason, yeah. - No, not quite yet. And, then I want... I certainly have some questions on my own. So, I wanna encourage anybody here, if you have questions, feel free to, again, drop them in the chat or use the Q&A. I just wanna circle back briefly to some of what Peg shared, in particular, the data on the outcomes that are seen, and really what are we trying to do? Like what is our... What is our goal? What is this sort of expectation we have when we deploy a program like EdReady, in particular, to replace an existing process? So, very simplistically, kind of, when you have a population of people, and you understand that they vary, they vary in some metric, and, again, this would be academic performance of some sort. the reason that we're interested in that is because we have a lot of reasons to believe that students who are better prepared academically are ultimately going to do a better job in terms of not only taking and passing courses, but then persisting to getting a degree, right? That's sort of the coin of the realm, if you will, in what we do in education. And, then, similarly, if you're really under prepared, then you are going to struggle, right? There is gonna be this problem that you're gonna have when you try to move on to further studies. And, so this just illustrates the, very simplistically, why do we care, right? Why do we wanna know who's more prepared and who's less prepared? And, it's because of this prediction that you can make. If you don't do anything, you would expect the better prepared students to have higher grades and higher retention and the less prepared students to have lower grades and lower retention. So, by measuring preparedness in some way ahead of time, what you're hoping to do is to intervene. You'll be able to identify the ones who are less well prepared. And, if your intervention does something useful, you will close that outcome, right? You will shrink the gap between those who are better prepared and those who are less well prepared. And, you should see that improvement. Note that it's difficult to come up with a scenario where you would think the intervention would catapult students who were at risk and are worse prepared to be... Have better outcomes than the ones who are never at risk in the first place. As a rule, what we're trying to achieve is some level of parody, right? We wanna take students that we believe were not going to persist, were not gonna do well, and we wanna get them to at least perform at a comparable level, if not a completely equivalent level, to the students who were never at risk in the first place. When you think back to Peg's slides, what we see is that, for that group that did not meet the preparedness threshold, and therefore were in that less prepared, but used EdReady, and then ultimately moved on, and took credit bearing courses, right? College courses, and also we now have those persistent statistics, they were in fact equilibrated. We close that gap completely. And, that's, that's kind of in a nutshell the evidence we present, not just here, but at many of our other partners for how EdReady does it differently, right? Yes, there's this assessment component, but, most importantly, there's this piece of it where students can use it to in fact close that gap and take themselves out of the at-risk category. And, now they are, for all intents and purposes, at least as prepared as the students who weren't at risk in the first place. So, hopefully that makes sense as far as outcomes go. So, I wanna pause and let's have a little discussion. So, we are getting a question, has EdReady totally replaced TABE or do you all still use TABE to some extent as well? - We do not use TABE for college placement. We do use tape in adult education, because it is a requirement, a federal requirement that we use a test with the properties of the TABE that gives us the measure we're looking for. But, at the college, we do not. The only test we administer, if folks come to us for testing, is EdReady. You saw on the placement chart though, that we accept ACT, SAT, many other things, but the only test a student can take if they need placement at Gateway is EdReady. And, I will tell you that we often have students from other KCTCS colleges contact us to make an EdReady appointment, because while colleges don't have to give it, they do have to accept it for placement. - Excellent. Well, another question just came in. They're asking, "Are you using the study path diagnostic or the test to determine whether they're ready or not ready? Or, did you create a different assessment?" I think the answer is the diagnostic is your-- - Right. Right. - Yeah. Yeah, so it is... It is comprehensive in-- - Right, and those scores appear on that placement chart on the assessment and placement chart. - Correct. So, stepping back a little bit, we at NROC, we're a mission-based organization, so clearly the... The method by which we try to achieve some of our desired outcomes in terms of improving student readiness, success, right? Putting them on the paths to their sort of future opportunity in college and career, it utilizes our tools by and large, right? I mean, that's kind of obvious, that's what we know best, and we built them explicitly to support that endeavor. But, we often say EdReady is, we would say, necessary, to doing placement differently, to sort of making this all work, but it is not sufficient, right? You can't just sort of plunk EdReady down and have everything just work beautifully. You spoke to quite a bit of that in terms of how does this interdigitate with the support staff on the ground, right? How do the... When do you sort of make it available? When do you push the scores through to your registrar? Like, there's a lot of other moving parts here, but I guess I'm especially interested in the socialization piece. So, you spoke to this, putting the... The sort of the decision, right? The autonomy back in the hands of the student. To what extent do you feel like students are really getting that and is that a differentiator for students? I mean, is that something where they're saying, oh, wow, this is... This is different, perhaps, in contrast to other places where they've been? - Well, many of our students haven't been anywhere else, haven't been part of a post-secondary path. I will tell you that the high school, the dual credit students love it. And, many of our... Many of our high school partners have specific labs set up where students can work their study path, so that they can demonstrate, perhaps, whether they're college ready or not. I... We hear, and this is only anecdotally, but what we hear is that students, it's overwhelmingly positive. If they call admissions or they talk to the advisors, what we hear is they love the fact that they could do it, they could do it on their own time, that there was assistance available. So, we really... We really don't hear anything negative. The only time there's an issue is typically when a student has problems with equipment. That really has been probably the only thing we've had to deal with. And, we offer those students the ability to come to campus, we'll find a quiet place for them, and they can take the test there on campus if they don't have great broadband, and many parts of Kentucky do not. That's an issue we're facing as a state. But, we try to give them everything they need to really make this a simple and a pleasant process. And, we just say, you can think of this as a test, but really it's the best way for us to put a roadmap together to help you be more successful, whether it's in college or in adult ed. So, overwhelmingly, it's a positive experience is what we hear from our students. - Yeah, that's... I mean, and certainly among our other partners, that's been pretty key. We often have to remind people that, as a rule, we're dealing with adults, at least in the eyes of the law, and they are.... They are supposed to understand that, in order to get to where they're trying to go, there is some responsibility, and to the extent that you can be a partner in that. Rather than saying, oh, we're gonna put up hoops and hurdles, you're saying, no, no, no, there's a journey you need to go on, and we will help you along the way, so that... So that you're properly prepared. One of the questions that came in, I think the way I would look to understand the question, it's really about, why? Like, why don't you provide goal keys, or access to the test, except out of these specific periods of time? And, let's see if I understood it. I think the reason is because many of the students will write on that day be able to either demonstrate through the diagnostic or through the diagnostic and immediate study that they get themselves to the point where they are now eligible to enroll where they might not have been. And, so by kind of having this all teed up in a specific span of time, at least for the students that move that quickly, they get all the way to the finish line, you get 'em properly marked up, and off they go. Of course, if students wanna take more time, they can do that, but is that right? It just kinda-- - That is correct. And, that was our security measure that we put in place that we wanted students... We wanted proctors to be watching to see if they complete it, because we also like to let admissions know if someone doesn't show up for a test, we can say to them, "Hey, you may wanna reach out to the student, see what happened. Let's see if we can't get them back in, get them another test." But, we felt like, because we were going to enforce the no retest, we don't wanna just leave that test out there without any kind of safety or security protocol. So, that code is only good that day the student takes the test and it's never needed again. They don't need it, because once they have their diagnostic study path, that's accessed through their email, and through through the database itself. - Right. Excellent. Well, I don't see any more questions coming in. I think, let's turn it over to Jason who will just quickly describe a little bit about kind of how people can engage with us, like what we do with them to set up something comparable to what you have at Gateway, and then we will come back, and see if there's any more to talk about. So, I will share my screen again. - Thanks, Ahrash. And, thanks, Peg. So, my name is Jason Gipson-Nahman, and I am part of our customer success team here at NROC. So, all of our NROC partner institutions have a dedicated customer success specialist from our team that can work with you to set up and implement EdReady as an alternative placement tool at your institution. And, I wanna emphasize that the... One of the main features of EdReady is its ability to be customized to fit the specific needs of your institution. So, the folks at Gateway have done a lot of work over the years to set up EdReady to meet their needs, but the needs at your institution are unique, right? So, we can help you set up EdReady specifically for your... The needs of your particular institution. So, someone from our team will work with you every step of the way through that process, which I'll try to briefly summarize and highlight some main points. So, must be... I'll say next slide. Sure, this will work. So, the first step is to identify your main objectives, what your main objectives are in terms of content. So, math and or English committees at your institution can review our content and choose the concepts that students should know to place into specific courses or whatever courses you're trying to place students into. And, this can be done as one what we call scope or study path and diagnostic, or perhaps there'll be different study paths for different courses, right? Second, you need to determine which of those concepts that you chose, how much of that should be assessed upfront on the initial diagnostic. So, the goal is to make the diagnostic test comprehensive enough, so that the results are meaningful. But, we're not trying to make students test for hours, hours upon hours. So, next, after that, we'll help you set up first drafts of these scopes in EdReady and your staff will actually go through these assessments and study paths as if they were students, right? So, through this process, you'll test and refine the scopes, making adjustments and changes to both the overall scope of content and the diagnostic as needed. Yeah, 'cause I'm not even paying attention to the slides, but that... It's fine. So, then you and your staff will discuss scoring and determine which score thresholds should place students into the courses that you're using EdReady for. And, depending on your scope and diagnostic configuration, you might need our team to work with you to determine those score thresholds, 'cause scoring can can get a little bit tricky depending on how you set things up. And, after addressing scoring, we can work with you to customize all of the student facing messaging and make sure you're providing students with the necessary information and instructions throughout their experience in EdReady. And, you and your team will work to determine the processes you need to put in place before and after students interact with EdReady. So, like what kind of messaging or communication do you need to give them to direct them to EdReady? And, then, once they're done, like how are the scores reported in your system, and who handles that, and what are the next steps the students need to take? So, you can add that messaging, some of that in EdReady, but some of that will have to occur outside, and you and your team will determine that. So, hopefully, you already know who your customer success specialist is, but, if not, let us know, and we'll get you connected to the right person, and we look forward to working with you. So, at this point, I'm gonna pass it back to Ahrash to wrap up and take any additional questions you may have. - Thanks, Jason. Peg, I don't know if you're... You wanna take your finger away? You're on mute. So, I didn't-- - I see, I'm sorry. I see Angie's question and we would not make a high school student, if it was given in the school, we would not make them retake it at Gateway. - Oh, yeah, that's pretty interesting. And, that... And, that's speaking to... The people on the call may not be aware that a common... An increasingly common implementation for us at NROC with EdReady is to situate EdReady as a, let's call it a bridge, a bridge program of sorts, that allows high school students to, in effect, check and then address again, right? Like, put themselves into the readiness state that they need in order to matriculate to a post-secondary institution of interest. The kind of ideal implementation for that is one where there is something more systemic, so that students have choices, right? They would be able to do that readiness check and then there would be any number of institutions that would accept that as evidence of their readiness, so that they can get right in and get going with no further delay. And, what we find is that the particular rules and politics around that are enormously variable depending on where you happen to be in the country. But, it's certainly a model that we like, because it solves so many problems in terms of keeping students college oriented, giving them assurances that they're not gonna get derailed or delayed when they actually put all the pieces together to go. And, also gives the colleges, right? The receiving institutions a lot of information, right? They'll have the data they need to validate, yes, these students appear to know this material. We should be confident in their ability to succeed in these courses, irrespective of whatever other data you might have. So, I think we can wrap it up there. We're very close to being out of time. So, I don't know, Peg, do you have any final words maybe from people who are kind of on the fence or not entirely sure like how this might be something that their institution could look at and potentially try? - After 28 years, most of which... All of which has been in adult ed, but leading assessment at my college for 23 years, I can say this is the easiest to implement. It delivers what we ask it to do. And, it's just a really versatile tool. I mean, we use it in adult ed for knowledge recovery. We use it for college placement. If you're on the fence, give these folks a call. I think you'd be very pleased. And, it's very easy to personalize, so... - Just kind words. Thank you, Peg, we really appreciate that, and, of course, always appreciate our partnership with you and with all of our partners. For us, this... This is really a manifestation of why we exist. I mean, we are founded with a mission, which was to really take a hard look at the places where we're kind of falling down in terms of our commitment as a society, as a country, to making post-secondary and workforce development pathways true engines to opportunity. And, it just, for so much of our history in so many places, we end up setting up all of these additional barriers that actually make it quite a bit harder for the people who would be best served by those opportunities to even get to them. And, so we certainly end up being heartened by not only the fact that I think we've built something that helps to solve that problem, but we have partners who are willing to give it a shot and then are accumulating the evidence that we might be onto something. So... - [Peg] Maybe. - I will.. I think we'll leave it there. Thank you again, everyone. Happy end of April. And, there will be a recording and a follow on message sent later for those of you who registered. And, I hope everyone has a wonderful day.