Measuring Impact and Looking Ahead

Welcome to our fifth and final installment in the ADDIE series. This month, we are shining the spotlight on the E in ADDIE, Evaluation.

For some instructional designers, Evaluation is an unfamiliar concept. For others, it is the foundation of everything they do. That is because Evaluation can be broad, powerful, and deeply connected with every part of the learning process.

Let’s take a closer look at what it is all about.

Levels of Evaluation

If you studied Instructional Systems Design, you might remember Kirkpatrick’s Levels of Evaluation, a framework
​that remains a cornerstone of the learning and development world today.

At Level 1, we gather learner feedback through end-of-course evaluations to learn how we can improve the experience.

At Level 2, we measure learning outcomes through knowledge checks and assessments to confirm two things:

1. The content aligns with the objectives.
   
   
2. Learners are truly gaining the intended skills or knowledge.
   
   
   

Level 3 looks beyond the classroom to see if learners’ behavior has changed as a result of the training.

Finally, Level 4 measures return on investment by determining if the learning experience delivered meaningful value
to the organization.

That is Evaluation in a nutshell, but it is also much more than that.

Evaluation vs. Analysis

In instructional design, Evaluation is all about looking back so you can move forward.
If that sounds familiar, it is because Evaluation shares similarities with Analysis, the first phase of ADDIE.
Both examine what was to improve what will be.

The difference lies in purpose. Analysis identifies gaps to guide new learning solutions, while Evaluation measures
the effectiveness of existing ones.

Evaluation also extends well beyond courses. It can assess the success of entire programs or even organizations.
For example, a nonprofit might use Evaluation to confirm that grant-funded initiatives are meeting their goals,
with findings that directly impact future funding.

At D3, we have supported clients across this full spectrum. We integrate Kirkpatrick’s model, especially Level 2, into
every course we create, and we have also managed program and curriculum evaluations designed to
measure organizational success and impact.

Looking Ahead

As we wrap up our ADDIE series, we hope these posts have given you a deeper understanding of the instructional
design process and a fresh perspective on how each phase connects to create meaningful learning experiences.

At D3, we do not just follow ADDIE. We live it. From analysis and design to development, implementation, and
evaluation, our focus is always on creating effective, efficient, and beautiful learning solutions that make a real difference.

Thank you for joining us on this journey through ADDIE. Here’s to what comes next: more learning, more growth,
and more opportunities to make learning come alive.

(And do not forget to check out the infographic below for a quick refresher on the full ADDIE process!)

​
​Infographic Source: https://elearninginfographics.com/the-addie-model-infographic/

We’re now in our fourth installment of the ADDIE series! We’ve been discussing how ADDIE has stood the test of time and still has a power-packed punch when it comes to creating engaging learning journeys. At this point in our journey, we have Analyzed the learning needs, Designed for identified needs, and Developed final content. This month we have arrived at Implementation: how to get the learning to the learners.

With the emergence of eLearning and Learning Management Systems (LMSs), Implementation has become quite a large task, often with its own team. The tasks involved are typically performed by an LMS Administrator. In its simplest form, here’s what happens:
​

• Content Accessibility: Final content is made accessible to learners.
  
  
  
• Notification: The target audience is made aware of the new course.
  
  
  
• Launch: Learners enroll in and launch the course.
  
  
  

Sounds quick and easy, right? And sometimes it is. But much more is going on within each of these steps. Let’s break each step out a bit more.

Content Accessibility

When working with traditional LMSs (Cornerstone, Canvas, Litmos), the content creator must “publish” the files so they are in the desired format, such as SCORM, then share those files with the LMS Administrator. The LMS houses and serves all content directly, tracking completion through built-in gradebooks. But today, we have more than LMSs to think about. Your organization may use one of these.

LXPs (Learning Experience Platforms like Degreed, EdCast) aggregate content from multiple sources—internal SCORM files, external links, videos, articles, even social learning. They focus on personalized learning paths and often use AI to recommend content based on learner behavior and skills gaps.

LRSs (Learning Record Stores) primarily capture xAPI statements from various learning experiences, whether formal courses or informal learning events. Content can live anywhere, but learning data flows back to the LRS for comprehensive tracking.

Microlearning Platforms (Axonify, Grovo) are for bite-sized content delivery, often mobile-optimized. Content is typically native to the platform with built-in authoring tools, so there really isn’t any “handoff” since the tool is within the platform.

Notification – How the LMS Knows Who Needs What

The LMS typically determines learner assignments through:

• Role-based rules: All managers get leadership training.
  
  
  
• Compliance matrices: Job function + regulatory requirements.
  
  
  
• Skills gap analysis: Assessment results trigger specific course assignments.
  
  
  
• Manual assignment by administrators or managers.
  
  
  
• Self-enrollment based on available catalogs.
  
  
  
• API integrations with HRIS systems that automatically assign based on job changes or hire dates.
  
  
  

LMSs must gather information about each learner to see which of these categories apply. Short sentence. Big job.

Launch

Launch is all about a successful learner experience. Many checks take place:

• Technical validation: The course functions correctly.
  
  
  
• User verification: The correct learners can access the course.
  
  
  
• Course tracking: Course progress is maintained, including grades and completion status.
  
  
  

Another layer of complexity emerges when Instructor-Led Training (ILT) is the modality. Trainers and locations, participant travel, meals, and lodging—all must be secured prior to course launch.

Of all the phases of ADDIE, Implementation is the most complex, and the one phase NOT performed by the content creation team. Essentially, it's the moment you can confidently say:

"Learners can successfully start, navigate, complete, and get credit for this learning experience"—the full technical and pedagogical handshake is complete.

Wrapping Up Implementation

Implementation is where all the planning, designing, and developing comes together. It’s the phase that ensures learners not only have access to content but also a smooth, meaningful learning experience. While it’s often complex and requires coordination across teams, it’s also incredibly rewarding—this is where your learners truly start engaging with the knowledge and skills you’ve worked so hard to create.


​

Last month, we focused on the “A” in ADDIE—Analysis, and how it sets the foundation for everything we do at D3. This month, we’re spotlighting the next step: Design.
​
Design is so important to our process that it’s part of our name—Davenport Design & Development! Why do we love this phase so much? Because it lets us lean into our creativity. At D3, Design includes three core elements:

• Objectives
• Strategy
• ​Structure

Let’s break it down using a real client example: we’re creating a global sales training program for a pharmaceutical company. With the analysis already complete, here’s how the Design phase plays out:

1. Objectives
We start by identifying Performance Objectives (what the learner needs to do) and Learning Objectives (what they need to know). We also determine how those objectives connect and whether they should vary by role.

2. Strategy

Next, we choose the best instructional methods to help learners succeed. For this project, a blended curriculum might work best. That could include:

• Virtual instructor-led training (vILT)
• ​Podcasts
• eLearning modules
• ​Social media for change management

​
We might also organize learners into cohorts and design the program like a college course. We include discussion boards, group projects, and digital badges tied to progress. We often use acrostics to give the program a memorable structure that sticks with learners long after training ends.

 3. Structure
Finally, we design the framework that brings everything to life. That includes:

• A timeline with key deliverables
• ​Roles and responsibilities across the team
• ​An implementation plan

​
We consider how long it will take to build different types of content, how interactive the learning will be (since higher interactivity usually means longer development time), and how much existing content can be reused.
We also think through practical details like:

• How involved the subject matter experts (SMEs) need to be
• How content will be distributed
• ​How long it takes to upload materials to platforms like an LMS or LXP

​
Where Does the Design Blueprint Fit In?
Traditionally, the Design phase includes a Design Blueprint—a storyboard, content outline, or sample slides with draft scripts and graphics. But interestingly, we’ve found ways to eliminate this step! Why? That’s a topic for another blog post...

With Design complete, we’re ready to move into the next phase in ADDIE: Development. That’s where everything we’ve planned starts to come to life. Stay tuned for next month to find out why our clients rave about our Development practices!

You’ve heard us say that ADDIE is D3’s macro model, and even though it’s been around for a while, it continues to prove its value. In fact, the A in ADDIE (A for Analysis) is alive and well, and accounts for about 25% of our total business!

​Here’s how D3 serves clients via Analysis:
​

• ACADEMIA: Our June featured team member, Zachary Smith, turns analysis into real dollars for universities.          He analyzes social work curricula to determine whether they meet the criteria for Title IV‑E reimbursement eligibility. Using Kay Casey's proprietary set of tools and methods, Zachary ensures that universities are not only compliant but also receiving the funding they’re entitled to. This means assessing course syllabi, learning objectives, and occasionally course content to make sure reimbursement amounts are accurate. You can learn more about Kay and her work here.
  
  
  
• GOVERNMENT: We also provide assessment, monitoring, and evaluation services to a Department of Defense organization to ensure it is meeting its responsibilities to the U.S. government. D3 analyzes monthly reports from up to 30 staff members, tracking training activities, attendee demographics, event locations, and subject matter.  We also analyze Level 1, 2, and 3 data to measure learning impacts, incorporating both qualitative and quantitative findings. These reports are delivered upstream to the highest levels of DoD leadership.
  
  
  
• NON-PROFIT ORGANIZATIONS: NPOs frequently engage D3 to analyze and evaluate their programs, and to report findings back to their grantors, ensuring that funds are being used responsibly and effectively.
  
  
  

So if this kind of work excites you, maybe it’s time to give the “A” in ADDIE a closer look! At D3, there’s always more analysis to be done, and more impact to be made.

In Part 1 of this series we met each member of the team responsible for creating and implementing a large-scale learning solution. In Part 2 we looked at the Learning Strategy Implementation Plan. While it’s critical to understand the roles, tasks, and timelines of the large-scale learning solution, the picture would not be complete without a look at the Learner Journey. The infographic below depicts the learner experience that results from the large-scale learning solution.

​​Inform & Engage. Remember that we are talking about large-scale learning solutions here, so learners will be making a significant time investment. This means they need to plan and set aside the time they will invest in learning. For employees to invest their time in your learning solution, it’s not enough to communicate in one email or corporate announcement that this training is coming and announce time and day (or how to access if it’s on-demand). You must share the information multiple times, in multiple delivery platforms (e.g., email, intranet, posters, corporate social media, etc.). And you must share the information over time.  
In this model, the Learner Journey begins with introducing the learning solution to the learners via engaging media such as high-energy microlearning videos. The Development Team is tasked with developing content that will introduce the learning solution and get learners engaged, excited, and motivated to participate. This optional deliverable is most useful for on-demand learning (where learners choose to participate), and where participant and leadership buy-in is critical. Both organizational leaders and target audience peers are leveraged in these videos to encourage early adopters. The goal of this “inform and engage” communication strategy is to get the learners invested and excited about the learning opportunity you are creating.

Complete eLearning. As the learners move into actually engaging with the learning solution, note that eLearning is leveraged to minimize time spent in synchronous learning events such as classroom facilitated learning, which is reserved for facilitator demos, participant interactions and practice, and peer/facilitator feedback. 

​Hands-on Practice. If participants do not have to perform behavioral skills that need to be observed and critiqued (e.g., soft skills), then facilitated learning may be omitted from the learning solution. However, if learners need an opportunity to interact to cement their learning through the use of small group projects, hands-on practice, or even highly complex scenario walk-throughs, then the classroom is the best next step in the learner journey. These classroom experiences give learners an opportunity to try out their new skills in a controlled environment without serious consequences, and provide the opportunity for self-correction as well as integrating valuable feedback from peers and facilitator. With the emergence of COVID-19 driving many corporate policies around classroom training, VILT may be substituted for classroom experiences.

Retention Support. Finally, you will see that the learning process does not end with the conclusion of formal learning. Rather, in large-scale learning solutions, Retention Support is an essential tool to reinforce learning on the job. Retention support may be in the form of job aids, online help, knowledge bases, online communities, performance guides, mentor observations and feedback, scheduled re-certifications, or even full-blown practicums. Also, during Retention Support, data analytics may combine with input from Leadership and Stakeholders to measure and analyze the effectiveness of the learning solution, which provides the justification for the initiative and allows each team member to celebrate a successful journey!
​
In this series we have explored all aspects of large-scale learning solutions strategy, from the team members and their contributing roles, to the learning plan, and finally looked through the lens of the learner journey. Be sure to take these strategies and tips into your next large-scale learning solution build and reap the benefits of your learning journey through these articles!

In Part 1 of this series, you met the team you’ll need to bring your project to life. Now that you’ve met the team, let’s look at them through the lens of the project life cycle. At the top of this infographic, you can see the four phases of any large-scale learning solution: Plan, Build, Test, and Execute. Let’s break down those four phases to see what the team members will be doing in each phase:

• Plan: The critical milestones that must happen during the planning stage include:
  • Kickoff: The project kickoff typically includes re-introductions of team members if needed, sharing the vision and goals of the project, success factors and accompanying metrics if known, a group SWOT, and organizational benchmark data to support the reason for the project.
  • Alignment: A critical component of a successful plan includes key alignment of stakeholders as well as conflicting goals and agendas. What does each stakeholder need to achieve as a result of this project?
  • Assess Needs: Where are we now, and where are we going? How are we going to get there? What human resources to we have? What must we outsource? What technology do we want to use? Do we have it now? Do we need to buy it to support this project? These questions illustrate a small sample of the Q & A that takes place during the Plan phase. The goal is to be sure all stakeholders have shared all information they consider helpful in ensuring the success of the project.
  • Learning Strategy Design: Perhaps most critical to the success of a large-scale learning solution is a thorough and sound Design. Most large-scale learning solutions include a High Level Design (HLD) that illustrates the skeleton of the project in terms of target audience, deliverables, and timeline. Once stakeholders agree on the HLD, then a Detailed Design Document (DDD) is produced, which includes a blueprint from which the Delivery Team can work. DDDs typically include Learning Objectives by topic, instructional strategies, delivery type (eLearning, facilitated learning, self-paced workbooks, etc.), source content identification and access, assessment strategies (how will learning be measured), and estimated time to complete.
• Build: The Build phase is the heart of the project timeline. This is where the majority of the work is done. Keep in mind that most large-scale learning solutions include multiple deliverables, such as eLearning to equip participants with basic terms, concepts, and examples of best practices. Then there may be a facilitated segment if participant skills must be observed and practiced with guided feedback. And finally, on-the-job performance may require deliverables in the form of performance guides, observation checklists, etc. Each of these types of deliverables may have a dedicated team and its own timeline. In each phase (Alpha, Beta, Gold), the instructional designers render the deliverable. The SMEs review the content and provide feedback. The PM or ID Lead typically does a QA review sometimes before as well as after the SME content review.  Lastly (and sometimes only in the final version), the stakeholders review the deliverable, and once accepted, the cycle repeats in the next phase until the final product is approved by stakeholders. A typical deliverable build includes the following phases:
  • Alpha: Depending upon deliverable, what’s completed in this phase will differ. For eLearning, an alpha is usually preceded by a storyboard. The storyboard, which is a mock-up of the eLearning course, may be a part of the Design phase or the first component of the build. In the alpha phase, all content might be rendered accurate and functional EXCEPT recording voiceovers (computer-generated voice can be used in alpha). For facilitated and other non-programmed learning, ALL content is often presented as final in the Alpha.
  • Beta: The goal is to provide 100% finished product. Upon review, if tweaks are needed, they will be rendered in the Gold phase.
  • Gold: The Gold version is 100% final product ready for Pilot. The Gold phase is omitted if 100% stakeholder approval was granted in the Beta phase.
• Test: Testing includes running the learning solution in the environment in which it will perform, and also allows a segment of the target audience to complete the learning solution to determine if any further changes are needed. ELearning content must go through testing in all environments in which it will expected to run when implemented, while other deliverables may only require target audience trial. During this phase, feedback is used to make required final adjustments. Another task that occurs during testing is ensuring the target audience is ready to participant in the learning solution. This may include course registration, travel arrangements, etc. The final step of Testing is ensuring all content is accessible to the learners prior to Implementation.
• Execute: The Delivery Team is in the spotlight during Execute.
  • The Learning Technologist is focused on ensuring all learners can access the learning
  • The Facilitator ensures s/he is ready to make learning transfer happen
  • Scheduling Support is attuned to ensuring travelers arrive for site-based facilitated learning
  • Leaders ensure calendars are cleared and their direct reports participate in the learning solution; leaders also play a large role in ensuring participants transfer learning on the job

Every project is different, so the team members, deliverables, and timelines will vary. However, you can adapt these steps to your project to be sure your team gets to the finish line with awesome deliverables! Tune in for Part 3 of this series, where we will focus on the journey for perhaps the most important group of people in a large-scale learning solution – the learners! 

​Have you ever been asked to:

• Create a global onboarding curriculum?
• Implement a B2B sales course for thousands of employees?
• Skill up all employees to use a new software system?
• Create a customer education solution for a new product launch?

These are just a few examples of large-scale learning solution launches that may feel overwhelming, but have no fear, the path is about to become clear! In the first of our three-part series, we’ll meet the team that you’ll need to assemble to build a large-scale learning solution. Read on to learn how to make your next large-scale learning solution implementation a launch you can shout about (in a good  way)! 

First Things First: Assemble Your Team

Large-scale learning solutions share certain characteristics, such as the need for a solid team. Let’s meet them:

​

• Stakeholders: They typically include at least one executive leadership sponsor, business unit representatives, Leadership & Development lead, and others depending on project dynamics. Your job is to find out what’s uniquely at stake for each stakeholder. Ask each of them: what does success of this learning solution look like to you? Write down their answers and as the project progresses, and check back with them periodically to be sure their success factors have not changed.
• Development Team: A typical Development Team includes:
  • Program Manager: For very large projects, a Program Manager represents the entire team. This person’s role is to be the face of the team to stakeholders as well as the development team. Program Managers have their eyes on multiple workstreams, and keep the Project Managers and stakeholders aligned. Program Managers have ultimate responsibility for the project budget and timeline for all workstreams.
  • Project Manager and/or ID Lead: These roles are sometimes subdivided where the PM takes ownership of budget and timeline while the ID Lead focuses on equipping and communicating with the content creators and Delivery Team as well as QA of deliverables. Depending on project dynamics and direction of the stakeholder, these two roles may be combined.
  • Instructional Designers: The learning solution would not exist without the Instructional Designers! Instructional Designers (IDs) are equipped with skills and tools to take the learning blueprint – the Design Document (and create the Design Document if none exists), and bring the Design to life in the form of deliverables.
  • Subject Matter Experts (SMEs): SMEs are the knowledge experts. While IDs apply methodology and use tools to create content, they rely upon SMEs for the content. IDs work closely with SMEs to gather content during development. SMEs are also critical during deliverable review cycles because they are the most knowledgeable in looking at the content through the learners’ eyes. Note that sometimes SMEs are also stakeholders as well as facilitators.
• Delivery Team: A typical Delivery Team includes:
  • Learning Technologist: This is the person who receives final deliverables (or accesses them via corporate intranet/network), and places the deliverables in a location that’s accessible to learners, such as a Learning Management System (LMS), a corporate YouTube channel, links accessible via corporate intranet, eblasts (emails sent to a targeted group of learners with links to learning content), social media (e.g., Yammer), or other conduits for learner access.
  • Facilitator(s): The facilitator is the person responsible for hosting synchronous learning, which may be in the form of classroom-based live learning, or a virtual learning session where learners access learning from their unique geographic location. NOTE: It’s a wise idea to have your lead facilitator(s) as part of your development team as a Subject Matter Expert. This way, the facilitators have more ownership in the content and will be more likely to deliver the course according to plan.
  • Participants: What would a course be without participants? Don’t forget to identify them as you plan your deliverables! This may require quite a bit of data gathering on the part of the Learning Technologist as well as Scheduling Support.
  • Scheduling Support: If learning is scheduled, then someone will be tasked with gathering and tracking participant contact info, reaching out via email or other methods to schedule training, and ensuring the participants’ travel plans are made in the case of classroom-based learning. This is typically a Learning Specialist skilled in administrative support.
  • Leadership: Some large-scale learning solutions require the support of the participants’ leadership (e.g., direct managers). Early in the project life cycle, stakeholders need to form a strategic communication plan directed at the leaders of the target audience to be sure those leaders understand and buy in to the value this learning solution will provide.

 
There you have it! All the members of a solid project development team! In our next installment, we’ll put the team to work as we look through the lens of the project life cycle!

About the Author: Teresa Davenport has been leading award-winning large-scale instructional design projects for...well, longer than she would like to say! Her love of inspiring and supporting all team members, helping stakeholders achieve goals, and providing innovative learning solutions all come together as conductor of the learning orchestra described in this series! You can reach Teresa at [email protected], and of course, on LinkedIn!