Year 3 (2019-2020) NYS 21CCLC Annual Evaluation Report Template Purpose of this This Year 3 Annual Evaluation Report (AER) Template and Guide for evaluators of local 21st CCLC programs in New York State was developed at the request of the State Program Coordinator. It is recognized, as stated in the Evaluation Manual, that “Evaluation first and foremost should be useful to the ...
Year 3 (2019-2020) NYS 21CCLC Annual Evaluation Report Template
Purpose of this Document
This Year 3 Annual Evaluation Report (AER) Template and Guide for evaluators of local 21st CCLC programs in New York State was developed at the request of the State Program Coordinator.
It is recognized, as stated in the Evaluation Manual, that "Evaluation first and foremost should be useful to the program managers at all levels of the system..." and that "The Annual Report's primary function is to present findings on the degree to which...objectives were met." The Evaluation Manual further specifies that the AER should report on the study methodology, findings, and recommendations and conclusions.
While these represent the report's "primary" functions, they do not reflect its only purpose. The AER also serves along with other data sources to inform NYSED Project Managers, Resource Center support specialists, and the Statewide Evaluator about program performance and accomplishments, which help guide the monitoring review and technical assistance processes. Indeed, many of the components of this report are directly aligned with NYSED policies and program expectations that are the focus of the monitoring visits that all programs receive. These alignments are highlighted throughout this template with references to required indicators and evidence in the revised Site Monitoring Visit Report ("SMV Report").1 Because NYSED and the Resource Centers review a program's AERs before each visit, information provided in this report that aligns with those indicators can be used to fulfill the documentation requirements of these visits.
Additional purposes of this report include helping to inform NYSED and the State Evaluator about trends across sub-grantees, which help to guide NYSED's policy decisions, as well as its mandated reporting to the U.S. Department of Education. In short, the AER supports program improvement at both the state and local levels, and contributes to evidence that the federal government needs to make funding decisions.
1 Retrieved from http://www.p12.nysed.gov/sss/documents/21C%20Onsite%20Monitoring%20Report%202017-19.doc.
Annual Evaluation Report (AER) Template Year 3 Final For all of these reasons, the information requested herein should be of interest to all stakeholders, and is consistent with that required by the Evaluation Manual 1 per the Request for Proposals for local program funding,2 as well as State monitoring guidelines.3 The purpose of this report guide and template is to clearly identify, and to organize within a consistent structure, the information that is necessary for each of the above stakeholders. The template has been designed with the varying needs of these different stakeholders in mind. It is designed to strike a compromise between the brevity and accessibility that program managers require, and the depth of detail that state and federal stakeholders require. Summaries or graphics that would be useful to program staff can always be included within the comments of each section or included in the appendices.
1 "New York State's 21st Century Community Learning Centers Evaluation Manual." Retrieved from: http://www.p12.nysed.gov/sss/21stCCLC/NYSEvaluationManual.pdf 2 Retrieved from http://www.p12.nysed.gov/funding/2017-2022-21st-cclc/2017-2022-21st-cclc-grant-application.pdf. 3 As outlined in New York State's revised 21st CCLC "Site Visit Monitoring Report," cited above.
Table of Contents
I. Project Information............................................................................................................................................................................................2 II. Evaluation Plan & Results.................................................................................................................................................................................3 Evaluation Plan and Results Tables ............................................................................................................................................................................ 3 III. Observation Results........................................................................................................................................................................................11
a. First visit .........................................................................................................................................................................................................11 b. Second visit: ...................................................................................................................................................................................................12 IV. Logic Model (LM) ............................................................................................................................................................................................13 V. Conclusions & Recommendations ..................................................................................................................................................................15 a. Status of the implementation of recommendations from the previous year......................................................................................................15 b. Strategies used to help ensure that evaluation findings were used to inform program improvement. ..............................................................15 c. Documented or perceived impacts of implementing prior year recommendations, if known ............................................................................16 d. Conclusions and recommendations based on the current year's evaluation findings ......................................................................................16 e. Conclusions and recommendations based on prior year evaluation findings that could not previously be addressed due to pending data, if applicable............................................................................................................................................................................................................... 16 VI. Appendices .....................................................................................................................................................................................................17
Annual Evaluation Report (AER) Template Year 3 Final
I. Project Information
Program Name Project Number Name of Lead Agency Name of Program Director Name(s) of Participating Site(s) and grade level(s) served at each site
Target Enrollment Evaluator Name and Company Evaluator Phone and Email
Boys & Girls Clubs of Buffalo, Charter Schools
0187-20-7014
Boys and Girls Clubs of Buffalo
Lorrie Ann Knight
Site 1: Buffalo United Charter School (BUCS)
Grade(s) Served: K-8
Site 2: Elmwood Village Charter School (EVCS)
Grade(s) Served: K-8
Site 3: South Buffalo Charter School (SBCS)
Grade(s) Served: K-8
Site 4: _________________________________________ Grade(s) Served: ____________________________
Site 5: _________________________________________ Grade(s) Served: ____________________________
Site 6: _________________________________________ Grade(s) Served: ____________________________
Site 7: _________________________________________ Grade(s) Served: ____________________________
Site 8: _________________________________________ Grade(s) Served: ____________________________
Site 9: _________________________________________ Grade(s) Served: ____________________________
Site 10: ________________________________________ Grade(s) Served: ____________________________
Site 11: ________________________________________ Grade(s) Served: ____________________________
Site 12: ________________________________________ Grade(s) Served: ____________________________
Total (Program-wide): 525
Actual # at/above 30 hours 537
Emily Hagstrom, Via Evaluation
(716) 362-0627 Emily@viaeval.com
2
II. Evaluation Plan & Results
Evaluation Plan and Results Tables
Objective 1: 21st CCLCs will offer a range of high-quality educational, developmental, and recreational services for students and their families.
Sub-Objective 1.1: Core educational services. 100% of Centers will offer high quality services in core academic areas, e.g., reading and literacy, mathematics, and science.
Program Objective 1.1-1 (specify): 100% of 21st CCLC students will participate in academic assistance and academic enrichment in the core areas during the school year and summer programs.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
During the school year, the program will offer targeted academic assistance/ enrichment activities to all participants for at least 30 minutes from MondayThursday.
Students
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
VIA reviewed the program
schedule and calendar and
discussed them with the
Power Hour,
Program
Project Director. At site visits,
Y
STEAM, Money
Schedule
VIA observed the timing and N/A
Matters, NASA
Site visit
quality of the activities. If any
Book Club
observations inconsistencies were noted,
VIA discussed them with the
project team for clarification.
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # sites where PI was met. If data pending, indicate when data expected. If not measured, explain why not.
Each site offers at least 30 minutes of
targeted academic assistance/enrichment
Yes
to all participants Monday-Thursday. Most
students participate in at least one hour
each day, Monday-Friday.
During the summer, the program will offer 3 hours of targeted academic activities to all participants every day of program.
Students
VIA reviewed the program
STEAM, Brain
Y
Gain Reading,
Program
schedule and calendar and N/A
Schedule
discussed them with the
Brain Gain Math
Project Director.
The program offered at least 3 hours of
Yes
targeted academic activities to all
participants each day during the summer.
Sub-Objective 1.2: Enrichment and support activities. 100% of Centers will offer enrichment and youth development activities such as nutrition and health, art, music, technology and recreation.
Program Objective 1.2-1 (specify): 100% of 21st CCLC students will participate in daily youth development activities designed to promote soft skills critical for students to succeed in school and life.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
The program will offer a variety of enrichment and youth development activities to all participants for at least 1 hour every day of program during the school year.
Students
The program will offer a variety of enrichment and youth development activities to all participants for at least 3 hours every day of program during the summer.
Students
VIA reviewed the program
schedule and calendar and
Arts and crafts,
discussed them with the
Program
Triple Play, Club
Project Director. At site visits,
Schedule
Y
Service, Positive
VIA observed the timing and N/A
Action, Smart
Site visit
quality of the activities. If any
Moves
observations
inconsistencies were noted,
VIA discussed them with the
project team for clarification.
Arts and crafts,
VIA reviewed the program
Triple Play, Club Program
schedule and calendar and
Y
N/A
Service, Positive
Schedule
discussed them with the
Action
Project Director.
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
Each site offered more than 1 hour of enrichment and youth development Yes activities to all participants each day during the school year.
The program offered at least 3 hours of enrichment and youth development Yes activities to all participants each day during the summer.
Annual Evaluation Report (AER) Template Year 3 Final
Sub-Objective 1.3: Community Involvement. 100% of Centers will establish and maintain partnerships within the community that continue to increase levels of community collaboration in planning, implementing and sustaining programs.1
Program Objective 1.3-1 (specify): BGCB will collaborate with the community to plan, implement, and sustain the after-school program and continue to expand services to students and their families.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
The program will hold quarterly
Advisory Committee meetings
VIA participated in meetings,
with the following participating:
AC Meeting
and sign-in sheets were
representatives from BGCB, Stakeholders
Y
AC Meetings
Minutes and
collected to monitor
N/A
EVCS, SBCS, BUCS, students,
Sign-in Sheets
stakeholder involvement in
parents, and community
meetings.
partners.
Each site held monthly meetings with all stakeholders participating and also Yes participated in two meetings with the other BGCB 21st CCLC program.
The program will offer services from at least two additional community agencies each program year.
Families
Community
Log of
The project director provided
agency-provided
community
the log of community agencies
Y
services (see list
agencies
providing services for
N/A
in explanation)
providing
evaluator review.
services
The following community agencies provided services at the three sites: Girls Scouts, Girls on the Run, ECMC Mobile Mammography, Write/Plan/Follow It Financial Literacy Program, Central Rock Yes Gym, Buffalo Zoo, Microsoft, Fisher Price, B-District Police, Albright Knox Art Gallery, Buffalo & Erie County Public Library, RICH Products, Americana music Program, and UB engineering
1 Note that this table might serve as a supplemental source of evidence documenting activities to engage and communicate with families, helping support grantees' compliance with Indicators in SMV Section G, particularly G-3, G-5, G-6, and G-7.
5
Annual Evaluation Report (AER) Template Year 3 Final
Sub-Objective 1.4: Services to parents and other adult community members. 100% of Centers will offer services to parents of participating children.1
Program Objective 1.4-1 (specify): Parents of students participating in the 21st Century Program will participate in programs to increase their capacity to support their students' education.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
The following events were offered at each site:
In conjunction with the schools,
the program will offer at least
10 family education and
Parents/Families
Y
engagement activities each
year.
Adult family members of at
least 40% of program students
will participate in family
Parents/Families
Y
education and engagement
activities each year.
Prior Year PIs for Objective 1.4-1
Family events (see Family event
list in explanation)
log, flyers
VIA reviewed the flyers and log of family events the
N/A program held throughout the year.
Family events (see Parent sign-in
list above)
sheets
The project director provided the number of parents who attended an event.
# targeted by PI: 564 # w data: 0
# targeted by PI: ___ # w data: ___
Yes Not measured
BUCS-Family Game Nights, Parent Info Nights, Annual Black History Month Program, Thanksgiving Day Program, Veterans Day Program, Santacade
EVCS- Family Halloween Night, Halloween Dance, Family Game Night, Family Craft Night, Santacade, Multicultural Celebration
SBCS-Potlucks snack meeting, Popcorn/raffle in conjunction with the school (2 times), Singing/ Dance Program , Hollowness Party, Thanksgiving, Santacade, Christmas Parents Teacher and Students Party
A total of 102 parents participated in events. However, parent names were not recorded and therefore could not be matched to students.
1 Note that this table might serve as a supplemental source of evidence documenting "Adult Learning Opportunities" helping to support grantees' compliance with SMV Indicator G-8(d).
6
Annual Evaluation Report (AER) Template Year 3 Final
# targeted by PI: ___ # w data: ___
Sub-Objective 1.5: Extended hours. More than 75% of Centers will offer services at least 15 hours a week on average and provide services when school is not in session, such as during the summer and on holidays.
Program Objective 1.5-1 (specify): 21st CCLC students will be offered quality programming during the summer and school year.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
During the school year, the program will offer 15 hours a week of programming to all students, from Monday to Friday.
Students
VIA reviewed the program
schedule and calendar and
discussed them with the
Program
Project Director. At site visits,
Calendar and
Y
Program activities
VIA observed the timing and N/A
Schedule
quality of the activities. If any
Site Visits
inconsistencies were noted,
VIA discussed them with the
project team for clarification.
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
EVCS and BUCS ran 3pm-6pm and SBCS
Yes
ran 3:30pm-6:30pm, Monday-Friday, for 15
hours a week each.
During the summer, the program will offer 40 hours a week of programming, from Monday to Friday for 6 weeks.
Students
VIA reviewed the program
Summer
Program
schedule and calendar and
Y
Calendar and
N/A
programming
discussed them with the
Schedule
Project Director.
During the summer, the program offered 50
Yes
hours a week of programming, 8am to 6pm
from Monday to Friday for 6 weeks.
Prior Year PIs for Objective 1.5-1
# targeted by PI: ___ # w data: ___ # targeted by PI: ___ # w data: ___
7
Annual Evaluation Report (AER) Template Year 3 Final Objective 2: Participants of 21st CCLC Programs will demonstrate educational and social benefits and exhibit positive behavioral changes.
Sub-Objective 2.1: Achievement. Students regularly participating in the program will show continuous improvement in achievement through measures such as test scores, grades and/or teacher reports.
Program Objective 2.1-1 (specify): Regular attendees will maintain or improve their literacy and math skills, the primary academic goals of the partner schools.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis conducted.
Include any longitudinal assessments conducted beyond one
program year.
(G) Response Rate/
% With Data (if applicable):
75% of regular attendees will meet grade level requirements in Math each year.
Students
75% of regular attendees will meet grade level requirements in ELA each year.
Students
95% of regular attendees will Students
progress to the next grade.
VIA received final ELA class grades
from BUVD. EVCS and SBCS
provided final quarter grades, which
Daily ELA-focused
were based on participation in online
Final ELA class
# targeted by PI: 487
Y
PowerHour
activities, and not mastery. VIA
grade
# w data: 199
activities
analyzed the data and calculated
the percentage of regular attendees
who passed their class according to
the schools' grading scales.
VIA received final Math class grades
from BUCS. EVCS and SBCS
provided final quarter grades, which
Daily Math-
were based on participation in online
focused
Final Math class
# targeted by PI: 487
Y
activities, and not mastery. VIA
PowerHour
grade
# w data: 199
analyzed the data and calculated
activities
the percentage of regular attendees
who passed their class according to
the schools' grading scales.
Daily academic Grade
Grade progression information was # targeted by PI: 487
Y
activities, Positive
progression
provided by the schools.
# w data: 467
Action
information
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
Not measured
Due to the COVID-19 shutdown, only BUCS assigned cumulative end-of-year grades to students. The other two schools based third trimester grades on participation in remote learning, and not mastery of the subject. Among the students at BUCS, 87% (173/199) passed ELA class.
Not measured Yes
Due to the COVID-19 shutdown, only BUCS assigned cumulative end-of-year grades to students. The other two schools based third trimester grades on participation in remote learning, and not mastery of the subject. Among the students at BUCS, 89% (178/199) passed Math class.
Of the 467 students with data, 451 (97%) progressed to the next grade. However, it should be noted that EVCS and SBCS promoted all students.
8
Annual Evaluation Report (AER) Template Year 3 Final
Sub-Objective 2.2: Behavior. Regular attendees in the program will show continuous improvements on measures such as school attendance, classroom performance and decreased disciplinary actions or other adverse behaviors.
Program Objective 2.2-1 (specify): Regular attendees will improve their achievement of indicators that positively impact school performance.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
At least 85% of regular attendees will exhibit positive school day attendance (missing less than 10% of eligible school days).
Students
At least 75% of regular attendees will demonstrate improvement in personal skills related to recreation, physical fitness, nutrition & health or conflict resolution.
Students
Student report cards that
included attendance
information were provided by
School day
the schools. Some schools # targeted by PI: 487
Y
Positive Action
attendance
included non-participation in # w data: 0
records
online activities as absences,
making the data not
comparable.
This objective was modified
this year, and the approval for
# targeted by PI: 487
Y
Positive Action
Staff survey
the modification came too late # w data: 0
to administer this survey to the
staff
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
Not measured
School day attendance could not be reliably measured this year, as each school handled attendance during virtual programming differently.
Not measured
This objective was modified this year, and the approval for the modification came too late to administer the survey to staff.
9
Provide a discussion of any particular strengths or limitations of above assessments or evaluation design, and describe any efforts or plans to minimize limitations (Required if there were limitations). (Optional): Additional comments on evaluation plan and Year 3 PI results.
This year was the first year with a new evaluator, so objectives and performance indicators were reviewed and modified. The modification approval was not granted until late July, when it was too late to administer staff surveys. This, combined with the early closure of schools, made it difficult to measure many objectives as planned.
Now that the objectives are in place, and with the new EZ Report system, the evaluator and program staff can ensure that data will be collected appropriately to measure outcomes next year.
Annual Evaluation Report (AER) Template Year 3 Final
III. Observation Results
In this section you are asked to provide data and findings from each of the two required annual evaluator visits per site, as specified in the Evaluation Manual. The specified purposes of these visits, as defined in the Evaluation Manual, include:
First visit: observe program implementation fidelity (Evaluation Manual, pp. 17-18). This visit includes verifying existence of, and alignment among,
the grant proposal (including the Table for Goals and Objectives), logic model, calendar and schedule of activities, program timeline, program handbook, parental consent forms, and procedures for entering/documenting evaluation data. This visit should also serve to identify any barriers to implementation.
Second visit: conduct point of service quality reviews (Evaluation Manual, p. 29). This visit, during which an observation instrument such as the OST is completed for selected activities, focuses on activity content and structure (including environmental context, participation, and instructional strategies), relationship building and the quality of interpersonal relationships, and the degree to which activities focus on skill development and mastery.
a. First visit
Append observation protocol results.1 Alternatively, you can paste on this page any summaries of findings on fidelity to program design from the first required visit.
Please specify approximate date(s) of first round of Year 3 observations (MM/YY): 12/20___________________________
Results:
See attached results
1 Copies of completed site observation protocols and/or other site visit summaries should be provided to program managers as a source of required supporting evidence to meet compliance for SMV Indicator H-1(c), "evidence of two site visits per site."
11
Annual Evaluation Report (AER) Template Year 3 Final
b. Second visit:
Append observation protocol results,1 or paste on this page, any summaries of findings on point of service quality review observations from the second observation conducted as part of the program evaluation.
Please specify approximate date(s) of second round of Year 3 observations (MM/YY): N/A__________________________ Observation protocol used for point of service observations:2
Out of School Time (OST) Protocol X Modified Out of School Time (OST) Protocol Other observation protocol (attach sample in Appendix, or if published, indicate name): _______________________________________ Results: While the program ran in-person child care programming after the closure, a site visit could not be conducted for safety reasons.
1 Copies of completed site observation protocols and/or other site visit summaries should be provided to program managers as a source of required supporting evidence to meet compliance for SMV Indicator H-1(c), "evidence of two site visits per site." 2 Note: As specified in SMV Indicator D-3, grantees are also required to conduct program activity implementation reviews, using a form consistent with the research-based OST observation instrument. Evidence of the activities specified in Indicator D-3 [see D-3(a) and (b)] can be strengthened if the evaluator and grantee collaborate on learning from the findings of these similar point-ofservice observations and grantee quality reviews.
12
IV. Logic Model (LM)
Annual Evaluation Report (AER) Template Year 3 Final
13
Annual Evaluation Report (AER) Template Year 3 Final Use the space below to summarize any aspects of the LM that have changed since the prior program year,1 or are still under development,
and if so, why. Comments:
1 Note that annual reviews of the logic model are required, as per SMV Indicator H-2(b).
14
Annual Evaluation Report (AER) Template Year 3 Final
V. Conclusions & Recommendations
Program's successes and lessons learned based on evaluation findings1
a. Status of the implementation of recommendations from the previous year
The following recommendations were made last year, followed by the actions taken by the program:
Improve parent engagement and advisory board participation, especially at SBCS BGCB staff met with the school administration team and teachers to collaborate on the various events the school sponsored during the year. BGCB participated in those events by assigning staff, purchasing supplies, and partnering with teachers to create games. BGCB staff helped to supervise the attendees.
Continue to improve staff training Coordinators in Sports/Rec and STEAM and the Director of Program completed observations of Youth Development Professionals. On-site training opportunities also were provided. Site Directors sent specific areas of improvement for their staff that allowed BGCB to create individual improvement plans. Teachers provided training in behavior management. BGCB required Director/Asst. Directors to attend grade level meetings, which helped to improve discussion of wrap-around services and improved relationships with the school staff.
Promote and strengthen academic confidence among students. BGCB increased the number of teachers assigned to assist during Power Hour. In some cases, teachers increased their hours so they could take part in enrichment activities as well.
Consider ways to address the issue of early removal from the afterschool program. BGCB adopted an attendance policy at all locations. Parents whose children were not meeting those requirements were contacted and BGCB reviewed attendance data with the parents. They were given another opportunity to meet the attendance requirements. If there were no improvements, the parents were informed their child would be placed on a wait list until the parent felt they could benefit from the program. This allowed BGCB to provide services to students on the waitlist, minimizing their wait time.
Address turnover BGCB did exit interviews with Youth Development Professionals who resigned from the organization. The overall concern expressed by these exiting staff was the low pay and part-time status. BGCB increased the hourly rate from $11.15 per hour to $15 per hour, and created succession planning guidance for those interested in moving up within the organization.
b. Strategies used to help ensure that evaluation findings were used to inform program improvement.
1 Note: as specified in SMV Indicator H-7, grantees are required to communicate evaluation findings to families and community stakeholders. Evidence of implementation of the activities specified in Indicator H-7(a) and (b) can be strengthened if the evaluator can help provide the grantee with a summary of sharable findings, such as reported in this summary.
15
Annual Evaluation Report (AER) Template Year 3 Final
In this first year with the new evaluator, regular meetings were held to discuss the plan for the evaluation and review objectives. The objectives were modified to better align to programming. Recommendations for program improvement were provided by VIA after the site visits. Student and parent surveys were completed, and the results presented to the project director.
c. Documented or perceived impacts of implementing prior year recommendations, if known
The program took many actions based on the recommendations last year, and they seemed to have a positive impact on the program. The early closure due to the COVID-19 pandemic prevented the evaluator from seeing some of these changes in action during second site visits.
d. Conclusions and recommendations based on the current year's evaluation findings
This program benefits from strong, organized leadership, as well as many years of experience running programming. When the schools had to shut down abruptly in March, the program quickly pivoted to offer in-person care for students of first responders. The transition with evaluators, and rewriting many performance indicators this year was a learning curve for both the program and evaluator, and unfortunately with the pandemic and early closure of schools, many of the typical evaluation activities could not take place as planned, such as second site visits, and some advisory meetings. However, the program had the benefit of experience, it was run as planned pre-closure, and most available data were collected as needed. Now that the objectives have been approved, and with the new statewide data collection system, the program will be able to ensure data are collected proactively in alignment with the objectives.
The following recommendations are provided to improve programming next year:
Work with Evaluator to ensure new objectives are measured
Now that the modifications have been approved, continue to work with the evaluator to refine data collection procedures to ensure data re collected as needed to measure the objectives.
Create a plan for virtual programming
It is likely in-person programming will be limited for some, if not all, of the 20-21 program year due to COVID-19. A plan for virtual programming should be created so that students and families can receive services even if in-person options are limited. Creating a plan now and getting necessary supplies will ensure that people have access to the valuable services the program provides. Some resources to inform this planning include experience gained during the spring closure, as well as found online through AfterSchool Works or the New York State Network for Youth Success.
e. Conclusions and recommendations based on prior year evaluation findings that could not previously be addressed due to pending data, if applicable
N/A
16
Annual Evaluation Report (AER) Template Year 3 Final
VI. Appendices
Required:
Copies of any locally developed measurement tools/assessments (surveys, observation tools, etc.) Full, tabulated results of any quantitative assessment tools (surveys,1 observation protocols, skills assessments, etc.)
Optional:
Sample of memo or weekly/monthly report used to share ongoing evaluation results/data with program2 Any additional narrative, analysis, graphics or other information that did not fit into any section in this report that you would like to include
1 Note: As specified in SMV Indicator H-4(a), local evaluators and program administrators are jointly responsible for administering annual surveys to student participants, and grantees are required to maintain documented evidence of this activity. 2 Note: As specified in SMV Indicator H-3(b), local evaluators and program administrators are jointly responsible for maintaining ongoing communication with each other, and grantees are required to maintain documented evidence of this activity.
17
2019-2020 21st Century Student Feedback Survey Report (Grades 3-8)
Boys and Girls Clubs of Buffalo - Chartter Schools
Number of Respondents
12 Overall
10 Elmwood Village Charter School 2 Missing
How often do survey respondents attend the program?
Less than once each month 0%
A few days each month
0%
A few days each week Every day program is offered
Grade Level of Respondents
44% 56%
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Missing
25% 17%
25% 8% 0% 8%
17%
What do students think of the program?
I like coming to this program. I tell my friends to come to this program. Staff at this program care about me. Staff at this program listen to me. I feel safe at this program. Students in this program are nice to each other. Activities in this program are fun. I can exercise or do sports activities in this program. I like the food in this program. Staff at the program treat me well. Staff at the program help me. If I can, I want to come back to this program next year.
Yes
Kind of
75%
50%
25%
83%
75%
83%
75%
83%
100%
27%
45%
92%
92%
83%
Not Really 17% 8% 25% 17% 17% 8% 17% 25% 8% 8%
27% 8% 8%
8% 8%
2019-2020 21st Century Student Feedback Survey Report (Grades 3-8)
Boys and Girls Clubs of Buffalo - Chartter Schools
What impact did the program have?
Yes
Kind of
Not Really
I come to school more often so I can go to this program.
50%
25%
25%
I learn new things in this program.
42%
50%
8%
This program helps me do better in school.
Coming to this program has helped me to learn I can do things I didn't think I could do before. Coming to this program has helped me to feel I am better at handling whatever comes my way. Coming to this program has helped me make better decisions.
58% 50% 45%
55%
17% 42% 55%
27%
25% 8%
18%
Themes from qualitative responses are summarized below. Complete survey comments are included at the end of this report.
What do students like best about the program? Caring staff and teachers Fun activities What would make the program better? Different activities/ability to choose from a range of activities Different food More time outside What would students do after school if they didn't attend the program? Dance class or bike ride Do homework Go home (or to a caregiver) What else did students share about the program? Enjoy playing with friends Excellent program Great staff
2019-2020 21st Century Student Feedback Survey Report (Grades 3-8)
Boys and Girls Clubs of Buffalo Complete Survey Comments
What do students like best about the program?
Everyone is kind to each other and the teachers are caring. Going to the art class and learning about computers. Gym I like the activities and the games are really fun. Its fun and something to do nothing Staff are always nice to me The staff.
What would make the program better?
different choices of food If we could play outside more It would be nice if the kids can pick their own activities like when we go to the gym some kids might want to play with the basketball instead of playing
kickball which the staff picked. more activities More activities miss the drama club. nothing Nothing I think that the program is very good.
What would students do after school if they didn't attend the program?
Be with a grandparent or other caretaker. Dance classes or ride my bike. do my homework at home in my room Go home and chill out I don't know I have no idea I would have to wait for my mom to finish all her work at school. It wouldn't be fun. play on my nintendo
What else did students share about the program?
For anybody who wants to attend this program should do it because it is an excellent program full of great staff. I am glad I can play with friends N/A no no thank you
2019-2020 21st Century Family Feedback Survey Report
Boys and Girls Clubs of Buffalo - Charter Schools
Number of Respondents
23 Overall
21 Elmwood Village Charter School 2 Missing
How often do survey respondents' children attend the program?
Less than once each month 0%
A few days each month
0%
A few days each week Every day program is offered
38% 62%
What do families think of the program?
Strongly Agree
Agree
Disagree
Strongly Disagree
Overall, this program is good for my child.
65%
30%
My child enjoys attending this program.
57%
39%
My child is safe at this program.
This program schedule meets the needs of my family.
I am satisfied with the level of communication with program staff.
School-day teachers want my child to come to this
23%
program.
I would recommend this program to my friends who have children.
70% 57% 59%
100% 59%
26%
39% 18%
36%
Usually
Sometimes
Rarely
Never
My child and I talk about this program.
77%
23%
My child says he/she likes program staff.
64%
32%
What do families think of the after-school family programs?
I am aware of after-school family programs. I attend after-school family programs. The after-school family programs are worthwhile.
Usually 32%
Sometimes 68%
71%
Rarely 50%
Never
23%
9%
14%
19%
10%
2019-2020 21st Century Family Feedback Survey Report
Boys and Girls Clubs of Buffalo - Charter Schools
What impact did the program have?
Since starting this program, my child has become interested in new areas (such as new subjects, sports, technology, etc.)
Since starting this program, my child has fewer behavior problems.
Since starting this program, my child is more interested in school.
Since starting this program, my child has improved in school.
Strongly Agree 30%
26% 26% 22%
Agree
Disagree
43%
57%
43%
48%
Strongly Disagree 26% 17%
30% 30%
Themes from qualitative responses are summarized below. Complete survey comments are included at the end of this report.
Why do families send their children to the program? Convenient with parents' work schedules Help students complete homework Students love the program What would make the program better? Greater focus on completing homework More activities More experienced staff (especially, experience dealing with behavioral issues) More frequent communiation with parents/guardians What family programs would be most helpful? Homework assistance Sports What else did families share about the program? Appreciation for the program Convenient location
2019-2020 21st Century Family Feedback Survey Report
Boys and Girls Clubs of Buffalo Complete Survey Comments
Why do families send their children to the program?
After school care is necessary due to work schedule and it is extremely convenient to have it right at the school. Because we work after school is over convenient for our family's work schedule Convenient location and times; allows children to complete homework For help with homework and still be at work Is at the school It is an easy program as the kids stay right at EVCS. The staff/supervisors know my kids and know what works and doesn't with them. It is super convenient because it's right at the school. Located at school My husband and I both work and are in need of the extra after school care. Plus we really enjoy the program! Needed the help with my work schedule. Program was located right inside of the school She has fun and we work so it is great for after school care. She loves the program and I love that she gets extra help with her homework. The activities they offer and time to finish homework or spend time with friends. It gives me time to finish my work after school as well. The instructors are great. We love Robin and Ms. Ebony We need trusted care after school when my husband and I are working. The kids enjoy their classmates and the Boys and Girls Club staff.
What would make the program better?
As per my child...the food lol Not that it's necessary but more school work focused on certain days would be great; though I also really appreciate the educational fun the kids have.
I think they do a great job already I'm not really sure right now because they do a lot. Issue with staff that are always addressed. Maybe more encouragement for the kids to complete their homework assignments while there. more activities More consistency with staff and have them better trained for student management especially if behaviors arise. More experienced staff, with more patience. Less hollering. More organization. More frequent communications from the staff to parents/guardians More of a focus on completing homework, and perhaps training staff to assist in that regard more. More structured activities. N/a Nothing at all Program was great have no complaints
What family programs would be most helpful?
Computer activities, homework, sports Home work assistance Sports/Gym. Homework help I can't think any. I'm not too aware of the family programs offered but normally after school we stay at home as the program ends at 6pm.
More sports Sports related. the homework hour helps me the most because I work a little later. They are all great Unsure Useful activities. Healthy meal ideas, organizing for busy parents,
What else did families share about the program?
conveniently located helps a ton. I really appreciate all that is provided at this program. It's a necessity for us! N/A No (2) This program is extremely helpful to our family - I do not know what we would do without it. Very thankful for this program We are grateful for the help and care offered!!
INITIAL SITE VISIT
Grantee: Boys and Girls Clubs of Buffalo Site: BUCS Date: 12/18/19 Site Coordinators: Rebecca Coccia and Andre Brown Easterling
Program summary:
Schedule: Monday-Friday, 3pm-6pm Total daily academics: 60 minutes Total daily enrichments: 90 minutes Total enrolled: 250
Recommendations:
· Continue to provide well-planned activities. · Continue meetings regularly with administration and
parents.
Participation Approximately 237 students attend each day, and the program is at capacity. Attendance method The staff are currently being trained on Comet so that student attendance can be entered by each staff member for their respective group via an iPad. The site coordinators currently enter the attendance and will check staff attendance for accuracy. Operations The program schedule is clear and includes all necessary information. Staff complete daily or weekly lesson plans, which are submitted to the site coordinators. Most activities take place in the central cafegymatorium, but additional classrooms are currently being approved by OCFS to be utilized for the older students. Academics Power hour is offered for 60 minutes each day. Money Matters is also offered. Enrichments A different program is offered each day of the week including SMART moves, Arts and Crafts, STEAM, and Positive Action. Triple Play is offered every day. Overall Atmosphere The observation took place before programming had begun for the day. The collaboration with the school was reported to be good. BGC staff currently meet weekly with the principal and deans of the school. Teachers, parents and students attend as needed or appropriate. Staffing Staffing is currently adequate. There are 12 staff including 1 school-day teacher. The site is currently in the process of hiring 3 additional staff. Community In addition to the weekly principal meetings, the site holds monthly parent meetings, as well as sending a parent newsletter each month.
INITIAL SITE VISIT
Grantee: Boys and Girls Clubs of Buffalo Date: 12/17/19
Site: EVCS Site Coordinator: Robin Roeser
Program summary:
Schedule: Monday-Friday, 3pm-6pm Total daily academics: 30-60 minutes Total daily enrichments: 75 minutes Total enrolled: 137
Recommendations:
· Continue to plan intentional activities and monitor attendance.
Participation There are currently only 2nd graders still on the waitlist, but that grade level is at capacity. Attendance is reviewed monthly and students not meeting participation requirements are removed from programming to allow space for students on waiting list. Attendance method Student time-in and time-out is tracked directly in Comet. Activity sheet attendance is kept as well and entered by staff into Comet after programming. Operations The program schedule is clear and includes all necessary information. All activities have detailed lesson plans. Academics Power hour is offered for 30-60 minutes each day, depending on the grade level. Money Matters is also offered. Enrichments Each day a different program is offered, including SMART moves, Arts and Crafts, STEAM, and Positive Action. Triple Play is offered every day. Flag Football is offered once a week for selected students. Overall Atmosphere The observation took place during programming, but program was not observed. The collaboration with the school was reported to be strong, and some BGC staff substitute teach during the school day. The site coordinator and her assistant have an office in the school and are there throughout the school day, which enables communication with school-day teachers. Staffing Staffing is currently adequate. There is one staff in process of being cleared to work. College students are also employed by the program. Community The site holds monthly/quarterly site-level meetings with parents, students, and the School's Special Education Coordinator participating. This coordinator also facilitates school-day linkages and is the data contact for the school.
INITIAL SITE VISIT
Grantee: Boys and Girls Clubs of Buffalo Date: 12/17/19
Site: SBCS Site Coordinator: Samir Lokhat
Program summary:
Schedule: Monday-Friday, 3:30pm-6:30pm Total daily academics: 45 minutes Total daily enrichments: 115 minutes Total enrolled: 125
Recommendations:
· Monitor student participation and offer spots of students who are not meeting attendance requirements to students on the waitlist.
Participation Participation is currently at capacity for the site. There is a waiting list of 50 students. Attendance method Student time-in and time-out is tracked manually and entered in Comet within 15 minutes. Activity sheet attendance is kept as well and entered by staff into Comet after programming. Operations The program schedule is clear and includes all necessary information. Academics Power hour is offered for 45 minutes every day. Additionally, Money Matters, STEAM, NASA, and Book Club are offered. Enrichments Each day a different program is offered, including SMART moves, Arts and Crafts, STEAM, and Positive Action. Triple Play is offered every day. Overall Atmosphere The observation took place before the beginning of programming. The collaboration with the school was reported to be strong, and school-day teachers are employed by the program to run Power Hour. BGC staff meet weekly with teachers in the school. As a result of the positive relationship with the school they have an office in the building, have access to multiple rooms, and can use the library for programming. Staffing There are currently 5 school-day teachers on staff in addition to BGC staff. There is a range of other staff, including some staff who have been working in the program for 3 or more years, and some college students. Additional staff have been hired and are waiting for fingerprint clearance to start. Community The site holds monthly site-level meetings with teachers, administration, parents, and students participating. There are multiple other partners who provide services in the program, including the Buffalo Zoo, Americana, NASA, Rich Products, and UB students who prepare engineering and science activities.
Year 3 (2019-2020) NYS 21CCLC Annual Evaluation Report Template
Purpose of this Document
This Year 3 Annual Evaluation Report (AER) Template and Guide for evaluators of local 21st CCLC programs in New York State was developed at the request of the State Program Coordinator.
It is recognized, as stated in the Evaluation Manual, that "Evaluation first and foremost should be useful to the program managers at all levels of the system..." and that "The Annual Report's primary function is to present findings on the degree to which...objectives were met." The Evaluation Manual further specifies that the AER should report on the study methodology, findings, and recommendations and conclusions.
While these represent the report's "primary" functions, they do not reflect its only purpose. The AER also serves along with other data sources to inform NYSED Project Managers, Resource Center support specialists, and the Statewide Evaluator about program performance and accomplishments, which help guide the monitoring review and technical assistance processes. Indeed, many of the components of this report are directly aligned with NYSED policies and program expectations that are the focus of the monitoring visits that all programs receive. These alignments are highlighted throughout this template with references to required indicators and evidence in the revised Site Monitoring Visit Report ("SMV Report").1 Because NYSED and the Resource Centers review a program's AERs before each visit, information provided in this report that aligns with those indicators can be used to fulfill the documentation requirements of these visits.
Additional purposes of this report include helping to inform NYSED and the State Evaluator about trends across sub-grantees, which help to guide NYSED's policy decisions, as well as its mandated reporting to the U.S. Department of Education. In short, the AER supports program improvement at both the state and local levels, and contributes to evidence that the federal government needs to make funding decisions.
1 Retrieved from http://www.p12.nysed.gov/sss/documents/21C%20Onsite%20Monitoring%20Report%202017-19.doc.
Annual Evaluation Report (AER) Template Year 3 Final For all of these reasons, the information requested herein should be of interest to all stakeholders, and is consistent with that required by the Evaluation Manual 1 per the Request for Proposals for local program funding,2 as well as State monitoring guidelines.3 The purpose of this report guide and template is to clearly identify, and to organize within a consistent structure, the information that is necessary for each of the above stakeholders. The template has been designed with the varying needs of these different stakeholders in mind. It is designed to strike a compromise between the brevity and accessibility that program managers require, and the depth of detail that state and federal stakeholders require. Summaries or graphics that would be useful to program staff can always be included within the comments of each section or included in the appendices.
1 "New York State's 21st Century Community Learning Centers Evaluation Manual." Retrieved from: http://www.p12.nysed.gov/sss/21stCCLC/NYSEvaluationManual.pdf 2 Retrieved from http://www.p12.nysed.gov/funding/2017-2022-21st-cclc/2017-2022-21st-cclc-grant-application.pdf. 3 As outlined in New York State's revised 21st CCLC "Site Visit Monitoring Report," cited above.
Table of Contents
I. Project Information............................................................................................................................................................................................2 II. Evaluation Plan & Results.................................................................................................................................................................................3 Evaluation Plan and Results Tables ............................................................................................................................................................................ 3 III. Observation Results........................................................................................................................................................................................11
a. First visit .........................................................................................................................................................................................................11 b. Second visit: ...................................................................................................................................................................................................12 IV. Logic Model (LM) ............................................................................................................................................................................................13 V. Conclusions & Recommendations ..................................................................................................................................................................14 a. Status of the implementation of recommendations from the previous year......................................................................................................14 b. Strategies used to help ensure that evaluation findings were used to inform program improvement. ..............................................................15 c. Documented or perceived impacts of implementing prior year recommendations, if known ............................................................................15 d. Conclusions and recommendations based on the current year's evaluation findings ......................................................................................15 e. Conclusions and recommendations based on prior year evaluation findings that could not previously be addressed due to pending data, if applicable............................................................................................................................................................................................................... 16 VI. Appendices .....................................................................................................................................................................................................17
Annual Evaluation Report (AER) Template Year 3 Final
I. Project Information
Program Name Project Number Name of Lead Agency Name of Program Director Name(s) of Participating Site(s) and grade level(s) served at each site
Target Enrollment Evaluator Name and Company Evaluator Phone and Email
Boys & Girls Clubs of Buffalo, Lovejoy
0187-20- 7015
Boys & Girls Club of Buffalo
Lorrie Ann Knight
Site 1: PS #43 Lovejoy Discovery School
Grade(s) Served: K-8
Site 2: _________________________________________ Grade(s) Served: ____________________________
Site 3: _________________________________________ Grade(s) Served: ____________________________
Site 4: _________________________________________ Grade(s) Served: ____________________________
Site 5: _________________________________________ Grade(s) Served: ____________________________
Site 6: _________________________________________ Grade(s) Served: ____________________________
Site 7: _________________________________________ Grade(s) Served: ____________________________
Site 8: _________________________________________ Grade(s) Served: ____________________________
Site 9: _________________________________________ Grade(s) Served: ____________________________
Site 10: ________________________________________ Grade(s) Served: ____________________________
Site 11: ________________________________________ Grade(s) Served: ____________________________
Site 12: ________________________________________ Grade(s) Served: ____________________________
Total (Program-wide): __175__________________ Actual # at/above 30 hours _100______________
Emily Hagstrom, Via Evaluation
(716) 362-0627 Emily@viaeval.com
2
II. Evaluation Plan & Results
Evaluation Plan and Results Tables
Objective 1: 21st CCLCs will offer a range of high-quality educational, developmental, and recreational services for students and their families.
Sub-Objective 1.1: Core educational services. 100% of Centers will offer high quality services in core academic areas, e.g., reading and literacy, mathematics, and science.
Program Objective 1.1-1 (specify): 100% of 21st CCLC students will participate in academic assistance and academic enrichment in the core areas during the school year and summer programs.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # sites where PI was met. If data pending, indicate when data expected. If not measured, explain why not.
During the school year, the program will offer targeted academic assistance/ enrichment activities to all participants for at least 30 minutes from MondayThursday.
Students
VIA reviewed the program
schedule and calendar and
discussed them with the
Program
Power Hour,
Project Director. At site visits,
Y
STEAM, Money
Schedule
VIA observed the timing and N/A
Matters
Site visit
quality of the activities. If any
observations inconsistencies were noted,
VIA discussed them with the
project team for clarification.
Lovejoy offers 30 minutes of Power Hour, STEAM, and Money Matters for a total of 90 Yes minutes of academics each day to all participants Monday-Thursday.
During the summer, the program will offer 3 hours of targeted academic activities to all participants every day of program.
Students
VIA reviewed the program
Y
STEAM, Brain Gain Reading,
Program
schedule and calendar and N/A
Brain Gain Math
Schedule
discussed them with the
Project Director.
The program offered at least 3 hours of
Yes
targeted academic activities to all
participants each day during the summer.
Sub-Objective 1.2: Enrichment and support activities. 100% of Centers will offer enrichment and youth development activities such as nutrition and health, art, music, technology and recreation.
Program Objective 1.2-1 (specify): 100% of 21st CCLC students will participate in daily youth development activities designed to promote soft skills critical for students to succeed in school and life.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
The program will offer a variety of enrichment and youth development activities to all participants for at least 1 hour every day of program during the school year.
Students
VIA reviewed the program
schedule and calendar and
Arts and crafts,
discussed them with the
Program
Triple Play, Club
Project Director. At site visits,
Schedule
Y
Service, Positive
VIA observed the timing and N/A
Action, Smart
Site visit
quality of the activities. If any
Moves
observations
inconsistencies were noted,
VIA discussed them with the
project team for clarification.
Lovejoy offered more than 1 hour of enrichment and youth development Yes activities to all participants each day during the school year.
The program will offer a variety of enrichment and youth development activities to all participants for at least 3 hours every day of program during the summer.
Students
Arts and crafts,
VIA reviewed the program
Y
Triple Play, Club Program
schedule and calendar and N/A
Service, Positive
Schedule
discussed them with the
Action
Project Director.
The program offered at least 3 hours of enrichment and youth development Yes activities to all participants each day during the summer.
Annual Evaluation Report (AER) Template Year 3 Final
Sub-Objective 1.3: Community Involvement. 100% of Centers will establish and maintain partnerships within the community that continue to increase levels of community collaboration in planning, implementing and sustaining programs.1
Program Objective 1.3-1 (specify): BGCB will collaborate with the community to plan, implement, and sustain the after-school program and continue to expand services to students and their families.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
The program will hold quarterly
Advisory Committee meetings
VIA participated in meetings,
with the following participating:
AC Meeting
and sign-in sheets were
Stakeholders representatives from BGCB,
Y
AC Meetings
Minutes Sign-in collected to monitor
N/A
Lovejoy, students, parents, and
Sheets
stakeholder involvement in
community partners.
meetings.
The program held quarterly advisory
meetings at the school with all stakeholders
Yes
participating, and also participated in two
meetings with the other BGCB 21st CCLC
program.
The program will offer services from at least two additional community agencies each program year.
Families
Log of
The project director provided
Community
community
the log of community agencies
Y
agency provided
agencies
providing services for
N/A
services
providing
evaluator review.
services
The following community agencies provided services at the school: Buffalo Zoo, UB Leadership Core, BPS Community Schools Yes Program, Youth Leadership Buffalo, Microsoft, Central Rock Gym, Rich Products, and Fisher Price.
1 Note that this table might serve as a supplemental source of evidence documenting activities to engage and communicate with families, helping support grantees' compliance with Indicators in SMV Section G, particularly G-3, G-5, G-6, and G-7.
5
Annual Evaluation Report (AER) Template Year 3 Final
Sub-Objective 1.4: Services to parents and other adult community members. 100% of Centers will offer services to parents of participating children.1
Program Objective 1.4-1 (specify): Parents of students participating in the 21st Century Program will participate in programs to increase their capacity to support their students' education.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
In conjunction with the school,
the program will offer at least
10 family education and
Parents/Families
Y
engagement activities each
year.
Open House,
Parent Meetings, Family event
Ice Cream Social,
log, flyers
Santacade
VIA reviewed the flyers and log of family events the
N/A program held throughout the year.
No
The program held 4 events this year.
Adult family members of at
least 40% of program students
will participate in family
Parents/Families
Y
education and engagement
activities each year.
The project director provided
Family events (see Parent sign-in the number of students who # targeted by PI: 135
No
list above)
sheets
had an adult family member # w data: 0
attend an event.
No parents of 21st CCLC Participants were reported as attending the events this year.
1 Note that this table might serve as a supplemental source of evidence documenting "Adult Learning Opportunities" helping to support grantees' compliance with SMV Indicator G-8(d).
6
Annual Evaluation Report (AER) Template Year 3 Final
Sub-Objective 1.5: Extended hours. More than 75% of Centers will offer services at least 15 hours a week on average and provide services when school is not in session, such as during the summer and on holidays.
Program Objective 1.5-1 (specify): 21st CCLC students will be offered quality programming during the summer and school year.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
During the school year, the program will offer 15 hours a week of programming to all students, from Monday to Friday.
Students
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
VIA reviewed the program
schedule and calendar and
discussed them with the
Program
Project Director. At site visits,
Calendar and
Y
Program activities
VIA observed the timing and N/A
Schedule
quality of the activities. If any
Site Visits
inconsistencies were noted,
VIA discussed them with the
project team for clarification.
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
The program runs 3:35pm-6:15pm, No
Monday-Friday, for 13.3 hours a week.
During the summer, the program will offer 40 hours a week of programming, from Monday to Friday for 6 weeks.
Students
Program
VIA reviewed the program
Summer Y
Calendar and schedule and calendar and
N/A
programming
Schedule
discussed them with the
Project Director.
During the summer, the program offered 50
Yes
hours a week of programming, 8am to 6pm
from Monday to Friday for 6 weeks.
7
Annual Evaluation Report (AER) Template Year 3 Final Objective 2: Participants of 21st CCLC Programs will demonstrate educational and social benefits and exhibit positive behavioral changes.
Sub-Objective 2.1: Achievement. Students regularly participating in the program will show continuous improvement in achievement through measures such as test scores, grades and/or teacher reports.
Program Objective 2.1-1 (specify): Regular attendees will maintain or improve their literacy and math skills, the primary academic goals of the partner schools.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
75% of regular attendees will meet grade level requirements in Math each year.
Students
75% of regular attendees will meet grade level requirements in ELA each year.
Students
95% of regular attendees will Students
progress to the next grade.
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
BPS provided final Math
grades for each student.
Students who scored below a
Daily ELA-focused
65 or who were given an
Final ELA class
# targeted by PI: 66
Y
PowerHour
incomplete were considered to
grade
# w data: 63
activities
not have met grade level
requirements. VIA matched
BPS data to program
attendance status.
BPS provided final ELA
grades for each student.
Students who scored below a
Daily Math-
65 or who were given an
focused
Final Math class
# targeted by PI: 66
Y
incomplete were considered to
PowerHour
grade
# w data: 61
not have met grade level
activities
requirements. VIA matched
BPS data to program
attendance status.
Grade progression information
Daily academic
was provided by BPS for all
Grade progression
# targeted by PI: 66
Y
activities, Positive
students. VIA matched BPS
information
# w data: 56
Action
data to program attendance
status.
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
Among the students with data, 61 (97%) Met
met grade level requirements in Math
Among the students with data, 58 (95%) Met
met grade level requirements in ELA. All 56 students with data (100%) Yes progressed to the next grade.
8
Annual Evaluation Report (AER) Template Year 3 Final
Sub-Objective 2.2: Behavior. Regular attendees in the program will show continuous improvements on measures such as school attendance, classroom performance and decreased disciplinary actions or other adverse behaviors.
Program Objective 2.2-1 (specify): Regular attendees will improve their achievement of indicators that positively impact school performance.
(A) Performance Indicator(s) (PI)
of success
(B) Target Population(s)
(C) PI Meets SMART Criteria?
(Y/N)
(D) Activity(ies) to support this
program objective
(E) PI Measures data collection instruments &
methods
(F) Describe the analysis
conducted. Include any longitudinal assessments conducted beyond one program year.
(G) Response Rate/
% With Data (if applicable):
At least 85% of regular attendees will exhibit positive school day attendance (missing less than 10% of eligible school days)
Students
BPS OSA provided the total
days absent and total
membership days for each
program participant through
the closure in March for the
19-20 school year. These
School day
# targeted by PI: 66
Y
Positive Action
were used to determine the
attendance records
# w data: 64
attendance rate. Students who
did not have data for either
year were excluded from
analyses. Only regular
attendees were included in the
analyses.
(H) Was this PI
Met? (Yes, No, Partial, Data Pending, Not Measured)
(I) EXPLAIN: If Yes, No or Partial: present results (expressed in the same metric as the PI) If Partial, indicate # of sites where PI was fully met. If data pending, indicate when data expected. If not measured, explain why not.
Among students with available data, 52
Met
(81%) had positive school day attendance
through March when the school closed.
At least 75% of regular attendees will demonstrate improvement in personal skills related to recreation, physical fitness, nutrition & health or conflict resolution.
Students
This objective was modified
this year, and the approval for
This objective was modified this year, and
# targeted by PI: 66
Y
Positive Action Staff survey
the modification came too late
Not measured approval for the modification came too late
# w data: 0
to administer this survey to the
to administer this survey to the staff.
staff.
9
Provide a discussion of any particular strengths or limitations of above assessments or evaluation design, and describe any efforts or plans to minimize limitations (Required if there were limitations). (Optional): Additional comments on evaluation plan and Year 3 PI results.
This year was the first year with a new evaluator, so objectives and performance indicators were reviewed and modified. The modification approval was not granted until late July, when it was too late to administer staff surveys. This, combined with the early closure of schools, made it difficult to measure some objectives as planned.
Now that the objectives are in place, and with the new EZ Report system, the evaluator and program staff can ensure that data will be collected appropriately to measure outcomes next year.
Annual Evaluation Report (AER) Template Year 3 Final
III. Observation Results
In this section you are asked to provide data and findings from each of the two required annual evaluator visits per site, as specified in the Evaluation Manual. The specified purposes of these visits, as defined in the Evaluation Manual, include:
First visit: observe program implementation fidelity (Evaluation Manual, pp. 17-18). This visit includes verifying existence of, and alignment among,
the grant proposal (including the Table for Goals and Objectives), logic model, calendar and schedule of activities, program timeline, program handbook, parental consent forms, and procedures for entering/documenting evaluation data. This visit should also serve to identify any barriers to implementation.
Second visit: conduct point of service quality reviews (Evaluation Manual, p. 29). This visit, during which an observation instrument such as the OST is completed for selected activities, focuses on activity content and structure (including environmental context, participation, and instructional strategies), relationship building and the quality of interpersonal relationships, and the degree to which activities focus on skill development and mastery.
a. First visit
Append observation protocol results.1 Alternatively, you can paste on this page any summaries of findings on fidelity to program design from the first required visit.
Please specify approximate date(s) of first round of Year 3 observations (MM/YY): 12/20
Results:
See attached results
1 Copies of completed site observation protocols and/or other site visit summaries should be provided to program managers as a source of required supporting evidence to meet compliance for SMV Indicator H-1(c), "evidence of two site visits per site."
11
Annual Evaluation Report (AER) Template Year 3 Final
b. Second visit:
Append observation protocol results,1 or paste on this page, any summaries of findings on point of service quality review observations from the second observation conducted as part of the program evaluation.
Please specify approximate date(s) of second round of Year 3 observations (MM/YY): N/A Observation protocol used for point of service observations:2
Out of School Time (OST) Protocol X Modified Out of School Time (OST) Protocol Other observation protocol (attach sample in Appendix, or if published, indicate name): _______________________________________ Results: Due to the early closure of schools, the second site visit could not take place.
1 Copies of completed site observation protocols and/or other site visit summaries should be provided to program managers as a source of required supporting evidence to meet compliance for SMV Indicator H-1(c), "evidence of two site visits per site." 2 Note: As specified in SMV Indicator D-3, grantees are also required to conduct program activity implementation reviews, using a form consistent with the research-based OST observation instrument. Evidence of the activities specified in Indicator D-3 [see D-3(a) and (b)] can be strengthened if the evaluator and grantee collaborate on learning from the findings of these similar point-ofservice observations and grantee quality reviews.
12
IV. Logic Model (LM)
Annual Evaluation Report (AER) Template Year 3 Final
13
Annual Evaluation Report (AER) Template Year 3 Final
V. Conclusions & Recommendations
Program's successes and lessons learned based on evaluation findings1
a. Status of the implementation of recommendations from the previous year
The following recommendations were made last year, followed by the actions taken by the program:
Improve parent engagement and advisory board participation BGCB staff met with the school administration team and teachers to collaborate on the various events the school sponsored during the year. BGCB participated in those events by assigning staff, purchasing supplies, and partnering with teachers to create games. BGCB staff helped to supervise the attendees.
Continue to improve staff training Coordinators in Sports/Rec and STEAM and the Director of Program completed observations of Youth Development Professionals. On-site training opportunities also were provided. Site Directors sent specific areas of improvement for their staff that allowed BGCB to create individual improvement plans. Teachers provided training in behavior management. BGCB required Director/Asst. Directors to attend grade level meetings, which helped to improve discussion of wrap-around services and improved relationships with the school staff.
Promote and strengthen academic confidence among students BGCB increased the number of teachers assigned to assist during Power Hour. In some cases, teachers increased their hours so they could take part in enrichment activities as well.
Consider ways to address the issue of early removal from the afterschool program BGCB adopted an attendance policy at all locations. Parents whose children were not meeting those requirements were contacted and BGCB reviewed attendance data with the parents. They were given another opportunity to meet the attendance requirements. If there were no improvements, the parents were informed their child would be placed on a wait list until the parent felt they could benefit from the program. This allowed BGCB to provide services to students on the waitlist, minimizing their wait time.
Address turnover BGCB did exit interviews with Youth Development Professionals who resigned from the organization. The overall concern expressed by these exiting staff was the low pay and part-time status. BGCB increased the hourly rate from $11.15 per hour to $15 per hour, and created succession planning guidance for those interested in moving up within the organization.
1 Note: as specified in SMV Indicator H-7, grantees are required to communicate evaluation findings to families and community stakeholders. Evidence of implementation of the activities specified in Indicator H-7(a) and (b) can be strengthened if the evaluator can help provide the grantee with a summary of sharable findings, such as reported in this summary.
14
Annual Evaluation Report (AER) Template Year 3 Final
b. Strategies used to help ensure that evaluation findings were used to inform program improvement.
In this first year with the new evaluator, regular meetings were held to discuss the plan for the evaluation and review objectives. The objectives were modified to better align to programming. Recommendations for program improvement were provided by VIA after the site visit. Student and parent surveys were completed, and the results presented to the project director.
c. Documented or perceived impacts of implementing prior year recommendations, if known
The program took many actions based on the recommendations last year, and they seemed to have a positive impact on the program. The early closure due to the COVID-19 pandemic prevented the evaluator from seeing some of these changes in action during second site visits.
d. Conclusions and recommendations based on the current year's evaluation findings
This program benefits from strong, organized leadership, as well as many years of experience running programming. When the schools had to shut down abruptly in March, the program quickly pivoted to offer in-person care for students of first responders. The transition with evaluators, and rewriting many performance indicators this year was a learning curve for both the program and evaluator, and unfortunately, with the pandemic and early closure of schools, many of the typical evaluation activities could not take place as planned, such as second site visits and some advisory meetings. However, the program had the benefit of experience, it was run as planned pre-closure, and most available data were collected as needed. Now that the objectives have been approved, and with the new statewide data collection system, the program will be able to ensure data are collected proactively in alignment with the objectives.
The following recommendations are provided to improve programming next year:
Work with Evaluator to ensure new objectives are measured
Now that the modifications have been approved, continue to work with the evaluator to refine data collection procedures to ensure data are collected as needed to measure the objectives.
Create a plan for virtual programming
It is likely in-person programming will be limited for some, if not all, of the 20-21 program year due to COVID-19. A plan for virtual programming should be created so that students and families can receive services even if in-person options are limited. Creating a plan now and getting necessary supplies will ensure that people have access to the valuable services the program provides. Some resources to inform this planning include experience gained during the spring closure, as well as found online through AfterSchool Works or the New York State Network for Youth Success.
15
Annual Evaluation Report (AER) Template Year 3 Final
e. Conclusions and recommendations based on prior year evaluation findings that could not previously be addressed due to pending data, if applicable
N/A
16
VI. Appendices
Annual Evaluation Report (AER) Template Year 3 Final
Required:
Copies of any locally developed measurement tools/assessments (surveys, observation tools, etc.) Full, tabulated results of any quantitative assessment tools (surveys,1 observation protocols, skills assessments, etc.)
Optional:
Sample of memo or weekly/monthly report used to share ongoing evaluation results/data with program2 Any additional narrative, analysis, graphics or other information that did not fit into any section in this report that you would like to include
1 Note: As specified in SMV Indicator H-4(a), local evaluators and program administrators are jointly responsible for administering annual surveys to student participants, and grantees are required to maintain documented evidence of this activity. 2 Note: As specified in SMV Indicator H-3(b), local evaluators and program administrators are jointly responsible for maintaining ongoing communication with each other, and grantees are required to maintain documented evidence of this activity.
17
21st Century Student Feedback Survey (Grades 3-8)
Boys and Girls Clubs of Buffalo @ Lovejoy School 43 2020
This is a survey about the activities that you attend after school at PS 43. Please tell us what you think about the program by answering the questions below. Please select only one response for each question.
Your responses are ANONYMOUS, which means that your name will not be collected, and we will not know who has said what about this program, so please be honest. Thank you for your feedback!
Tell us how you feel about the program. 1. I like coming to this program.
Not Yes Kind Of Really
Y
K
N
2. I tell my friends to come to this program.
Y
K
N
3. Staff at this program care about me.
Y
K
N
4. Staff at this program listen to me.
Y
K
N
5. I feel safe at this program.
Y
K
N
6. Students in this program are nice to each other.
Y
K
N
7. I come to school more often so I can go to this program.
Y
K
N
8. I learn new things in this program.
Y
K
N
9. Activities in this program are fun.
Y
K
N
10. I can exercise or do sports activities in this program.
Y
K
N
11. This program helps me do better in school.
12. Coming to this program has helped me to learn I can do things I didn't think I could do before.
13. Coming to this program has helped me to feel I am better at handling whatever comes my way.
14. Coming to this program has helped me make better decisions.
Y
K
N
Y
K
N
Y
K
N
Y
K
N
15. I like the food in this program.
Y
K
N
16. Staff at the program treat me well.
Y
K
N
17. Staff at the program help me.
Y
K
N
18. If I can, I want to come back to this program next year.
Y
K
N
PLEASE COMPLETE THE OTHER SIDE
1
21st Century Student Feedback Survey (Grades 3-8)
Boys and Girls Clubs of Buffalo @ Lovejoy School 43 2020
Please answer the questions below.
19. What grade are you in?
3rd 4th 5th 6th 7th 8th
20. How often do you attend this program?
Every day program is offered
A few days each month
A few days each week
Less than once each month
21. What do you like best about this program?
22. What would make this program better?
23. If you did not come to this program, what would you do after school?
24. Is there anything else that you would like to share about this program?
Thank you!
2
21st Century Family Feedback Survey
Boys and Girls Clubs of Buffalo @ Lovejoy School 43 2020
This survey asks about your experience with the 21st Century After-School Program that your child attends. This program includes the academic and enrichment activities that your child participates in outside of the regular school day. We would appreciate it if you took a few minutes to tell us about your experience with this program by answering the questions below. Please select only one response for each question.
Your responses are ANONYMOUS, which means that your name will not be collected, and we will not know who has said what about this program, so please be honest. Thank you for your feedback!
Tell us how much you agree or disagree with the statements below. 1. Overall, this program is good for my child.
Strongly Agree
A
Agree
a
Strongly Disagree Disagree
d
D
2. My child enjoys attending this program.
A
a
d
D
3. My child is safe at this program.
A
a
d
D
4. This program schedule meets the needs of my family.
A
a
d
D
5. I am satisfied with the level of communication with program staff.
A
a
d
D
6. Since starting this program, my child has become interested in new
A
a
d
D
areas (such as new subjects, sports, technology, etc.)
7. Since starting this program, my child has fewer behavior problems.
A
a
d
D
8. Since starting this program, my child is more interested in school.
A
a
d
D
9. Since starting this program, my child has improved in school.
A
a
d
D
10. School-day teachers want my child to come to this program.
A
a
d
D
11. I would recommend this program to my friends who have children.
A
a
d
D
Tell us how often the following things happen. 12. My child and I talk about this program. 13. My child says he/she likes program staff. 14. I am aware of after-school family programs. 15. I attend after-school family programs. 16. The after-school family programs are worthwhile.
Usually Sometimes
U
S
U
S
U
S
U
S
U
S
Rarely Never
R
N
R
N
R
N
R
N
R
N
PLEASE COMPLETE THE OTHER SIDE
21st Century Family Feedback Survey
Boys and Girls Clubs of Buffalo @ Lovejoy School 43 2020
Please answer the questions below.
17. How often does your child attend this program?
Every day program is offered A few days each month
A few days each week
Less than once each month
18. Why did you choose to send your child to this program?
19. What would make this program better?
20. What family programs or activities would be most helpful to you?
21. Is there anything else that you would like to share about this program?
Thank you!
INITIAL SITE VISIT
Grantee: Boys and Girls Clubs of Buffalo Date: 12/17/19
Site: PS 43 Site Coordinator: Amanda Palmieri
Program summary:
Schedule: Monday-Friday, 3:35pm-6:15pm Total daily academics: 30 minutes Total daily enrichments: 40 minutes Total enrolled: 73
Recommendations:
· Update schedule to include precise transition times and adequately represent total activity time.
· Ensure that the maximum amount of time reasonable is spent in activities.
Participation Participation is currently well below the target, due in part to insufficient staff. There is currently a waiting list of 50-60 students who can start once more staff is on board. Attendance method Student time-in and time-out is tracked in Comet as the students arrive and leave. Activity sheet attendance is kept as well and entered by staff into Comet after programming. Operations The program schedule includes general time ranges when activities could be starting and ending, so it is difficult to determine exact timing of activities. It currently appears that almost half of program time is spent in transitions, meals, and dismissal. Academics Power hour is offered for 30 minutes Monday-Thursday, and Money Matters is offered on Mondays. No academics are offered on Fridays. Enrichments Each day a different program is offered, including SMART moves, Arts and Crafts, STEAM, and Positive Action. Triple Play is offered every day. Overall Atmosphere The observation took place during mealtime at the beginning of programming, and program was not observed. The collaboration with the school was reported to be good, and school-day teachers are employed by the program to run Power Hour. Staffing There are currently 4 school-day teachers on staff, in addition to BGC staff. There are 3 new staff who just started last week. The new process for clearing staff has delayed bringing the additional staff needed to accommodate the wait list. Community The site holds quarterly site-level meetings with teachers, parents, and students participating. Youth leadership Buffalo is an additional partner who provides services in the program.
Nuance PDF Create