Audit of the Supporting the Documentary Heritage Communities Program

Table of Contents

Summary

Background

In 2015, Library and Archives Canada (LAC) began an approval process to reallocate LAC funds and establish a contribution program to assist local documentary heritage institutions in their mission to preserve and provide access to local collections. In May 2015, the Treasury Board (TB) approved LAC’s application for a contribution program, beginning in fiscal year 2015−2016. Total funding for the program is $8,698,000, including $7,500,000 for direct contributions and $1,198,000Footnote 1 to administer the program. The name of the program is Supporting the Documentary Heritage Communities Program (SDHCP).

As an integral part of the Program’s approval, LAC undertook to audit Program activities during fiscal year 2017−2018. Accordingly, Corporate Planning and Accountability Division included an audit of this Program in its 2017–2018 annual monitoring plan. The monitoring plan was approved by the Resource Management Committee on April 21, 2017.

Findings and recommendations

The following table provides an overview of compliance with TB policies and the Program’s terms and conditions.

Table 1: Compliance with the Program’s terms and conditions and the TB Policy on Transfer Payments

Activity

Compliant

Non-Compliant

Governance framework

X

 

Defined and documented roles and responsibilities

X

 

Evaluation criteria for documented and communicated funding applications

X

 

Documented evaluation processes

X

 

Evaluation process, including internal specialists and an external committee

X

 

Payments to recipients in accordance with established service standards

X

 

Administrative expenses for the Program

 

X

Compliance with the Policy on Transfer Payments

Mechanism to follow up with recipients

 

X

Compliance with the Financial Administration Act (FAA)

Section 34 of the FAA signed and dated by someone with relevant authority

X

 

Contribution agreements signed and dated before the issuance of payments (2017–2018)

X

 

In addition to compliance, observations were made related to the management of the Program, including:

  • Improvements could be made to optimize use of internal Subject Matter Advisors’ (SMA) analysis to support informed decision-making;
  • Discussions should be recorded during the final meeting of the External Advisory Committee (EAC) in order to have a better understanding of the decisions that led to the final selection of projects;
  • Some key documents were missing from recipients’ files in the Program workplace. The order and types of documents in the files varied from one file to the next, making it difficult to find relevant information; and
  • Insufficient training was provided for EAC members.

The Secretariat should:

  • review the evaluation process for funding applications in order to:
    • optimize the use of expertise provided by internal Subject Matter Advisors (SMA) and allow the sharing of all relevant information for the purposes of informed decision-making; and
    • better document discussions leading to the final selection of projects that will receive funding, thus ensuring the transparency of the evaluation process.
  • ensure that a structured recordkeeping system is in place to make it easier to locate documents and improve the efficiency of the analysis process;
  • review the process for analyzing applicant budgets to ensure that all costs are reasonable;
  • develop a risk-based monitoring mechanism to ensure that funds are used in accordance with the terms and conditions of funding agreements; and
  • ensure that all EAC members involved in evaluating funding applications receive the required training in a timely manner to do their work.  

Conclusion of the review

On the whole, the SDHCP complied with the conditions set out in the TB submission and with the Financial Administration Act (FAA). However, it partially complied with the TB Policy on Transfer Payments. In addition, some Program management deficiencies were identified, as indicated above.

Acknowledgements

We would like to express our appreciation to management and staff for their strong cooperative team spirit and assistance to the internal audit team throughout this engagement.

 

Introduction

Background

In 2015, Library and Archives Canada (LAC) began an approval process to reallocate LAC funds and establish a contribution program to assist local documentary heritage institutions in their mission to preserve and provide access to local collections.  In May 2015, the Treasury Board approved LAC’s application for a contribution program, beginning in fiscal year 2015−2016. Total funding for the program is $8,698,000, Footnote 2 including $7,500,000 for direct contributions. The name of the program is Supporting the Documentary Heritage Communities Program (SDHCP). The SDHCP’s annual budget to fund eligible projects is $1,500,000, leaving the rest of the TB-allocated funds to cover the Program’s administrative fees. The maximum eligible amount for each contribution is $100,000 per year. The following table indicates the number of applications received and funded by the Program since it was launched: Footnote 3

Table 2

Fiscal Year

Funding Applications Received

Funded Projects

2015–2016

136

64 Footnote 4

2016–2017

155

38

2017–2018

129

38

Total

420

140

SDHCP staff developed a collaboration-based approach with local documentary heritage communities so that Canada’s continuing memory is documented and accessible to current and future generations. This contribution program promotes the enhancement of library and archival communities by increasing their capacity to preserve, provide access to and promote local documentary heritage. Additionally, the Program provides these communities with opportunities to develop, remain viable and preserve their strategic importance.

The SDHCP provides financial assistance to the Canadian documentary heritage community for activities that:

  • Increase access to, and awareness of, Canada’s local documentary heritage institutions and their holdings; and
  • Increase the capacity of local documentary heritage institutions to better sustain and preserve Canada’s documentary heritage.Footnote 5

The Program is managed by LAC’s Corporate Secretariat. Many employees of branches within the Operations Sector participate in implementing the Program by providing expertise during the evaluation of applicant proposals in the area of archival and library sciences, digitization, preservation and client services. The Corporate Services sector supports and provides advice on financial issues.

As an integral part of the Program’s approval, LAC undertook to audit SDHCP activities during the 2017–2018 fiscal year. Accordingly, Corporate Planning and Accountability Division included an audit of this Program in its 2017–2018 annual monitoring plan. The monitoring plan was approved by the Resource Management Committee on April 21, 2017.

Risk assessment

A risk assessment was performed by the audit team. The following significant risks were identified and served as a basis for developing the audit criteria:

  • There is a risk that the governance structure may not allow the Program to meet its objectives;
  • There is a risk that the processes and guidelines may not be applied systematically, in accordance with the contribution agreement and the TB’s Policy on Transfer Payments; and
  • There is a risk that the tools, resources and training may not be in place to support sound Program management.

Objective and criteria

The objective of this audit was to provide assurance that the SDHCP is in compliance with Treasury Board policies and is consistent with the program objectives.

The following criteria were used during the SDHCP audit to test compliance with policies and program terms and conditions:

  • The SDHCP governance framework helps to meet the Program’s objectives;
  • A documented, fair and transparent evaluation process is used consistently;
  • The budgets submitted by applicants are analyzed and approved in accordance with Program terms and conditions and in compliance with policies;
  • Tools, guidance and training are developed and implemented to support effective Program management;
  • Payments to recipients are accurate, approved and issued in a timely manner in accordance with contribution agreements, standards and policies;
  • A monitoring mechanism is in place to help meet the Program’s objectives; and
  • Management has implemented an effective and complete recordkeeping system.

Scope

The audit covered SDHCP activities from the time of its implementation up to, and including, fiscal year 2017–2018. To be more specific, the tests focused on the following:

  • Governance framework;
  • Application evaluation process;
  • Program and payment management;
  • The mechanism used to follow up with recipients; and
  • Program records management.

Methodology

The methodology used for the audit included the following:

  • Review of the policies and directives related to the management of transfer payment programs, as well as the Program’s terms and conditions;
  • Analysis of the process used to evaluate funding applications;
  • Review of the population of recipients for the application of section 34 of the Financial Administration Act (FAA);
  • Sample review:
    • Funding applicant files;
    • Evaluations of funding applications;
    • Contribution agreements; and
    • Recipient expense reports.
  • Interviews with management, program management, the financial services team, internal Subject Matter Advisors (SMA), External Advisory Committee (EAC) members.

 

Findings and Recommendations

Contribution program governance

The contribution program governance framework complied with the Program’s terms and conditions. The roles and responsibilities of all parties concerned were defined, documented and communicated.

The audit team expected to find a functional governance framework to manage and control the contribution program, and enable the Program’s objectives to be met. We also expected this governance structure to comply with the Program’s terms and conditions, and the roles and responsibilities of all parties concerned to be clearly defined, documented and communicated.

We reviewed the governance framework in relation to what was planned when the Program was approved. We were able to confirm that the framework existed, that it was documented and communicated to the parties concerned, and that it complied with the Program’s terms and conditions. The Corporate Secretariat was responsible for Program management and administration. The SDHCP governance framework was composed of:

  • Subject Matter Advisors (SMA);
  • the External Advisory Committee (EAC); and
  • the Librarian and Archivist of Canada.

The SMA are LAC employees who were selected for their expertise in documentary heritage.Footnote 6 Their role was to evaluate funding applications from the perspective of feasibility, following the evaluation of proposals by Program employees based on the eligibility criteria. The  SMA were the first level in the Program’s governance framework.

EAC members are recruited from outside LAC and come from various backgrounds related to documentary heritage communities in Canada—librarians, archivists, authors, academics, historians, etc. EAC members were the second level in the governance framework for evaluating funding applications. The Committee meets once a year and submits recommendations directly to the Librarian and Archivist of Canada for the final approval of funding applications.

The duties and responsibilities were set out in the instructions for SMA,Footnote 7 as well as in the guide for EAC members.Footnote 8 These documents clearly identified the roles and responsibilities of SMA and those of EAC members. In addition, a section in the EAC guide described the roles and responsibilities of the Program’s team. The role of the Librarian and Archivist of Canada was documented and consisted in giving final approval for the projects selected by the EAC.

Moreover, interviews were conducted with various stakeholders to confirm that they understood their roles and responsibilities. The interviews with the Program’s employees, SMA and EAC members revealed that the roles and responsibilities were clearly defined, recorded, communicated and understood by all those concerned.

Evaluation of funding applications

A documented process for evaluating funding applications exists, however, improvements could be made to better utilize the analytical work done by SMA to support informed decision-making. In addition, the EAC’s evaluation process leading to the final selection of projects was not sufficiently documented.

We expected to see the following:

  • A documented and detailed process for evaluating funding applications that is fair, transparent and implemented consistently;
  • Applications received are processed within set time frames and in accordance with SDHCP terms and conditions; and
  • A priority process for funding applications is established, documented and shared with applicants.

LAC set up a three-step process for evaluating funding applications:

  1. Eligibility of the application based on the Program’s eligibility criteria;
  2. Feasibility analysis; and
  3. Scores assigned to selected projects.

Eligibility of applications based on Program eligibility criteria

The Program team, reporting to the Corporate Secretariat, was responsible for evaluating the eligibility of organizations’ applications, and ensuring, among other things, that they are received on time and that they meet all eligibility criteria. Applications that did not meet the eligibility criteria were rejected and not given further consideration. Some applications, for which the eligibility was uncertain, were referred to the Corporate Secretariat to determine their eligibility.

To be eligible, funding applications had to be received by a specific deadline. The Program directive states that the date recorded is the date the application was mailed, as evidenced by the postmark on the envelope containing the application. Since Program staff did not keep the envelopes after the applications were received, the audit team was unable to complete this test.

For this first step in the evaluation process, we observed that the eligibility evaluation process was documented and based on the eligibility criteria, as posted on the Program’s Web page. However, we were unable to confirm that this evaluation step was in full compliance because we were unable to conclude the application reception test.

Feasibility analysis

Applications that passed the first step in the evaluation were then sent to SMA, who were responsible for analyzing their feasibility based on their own expertise in library science, archives or any other field related to documentary heritage management. SMA had to answer the following three questions as part of their analysis:

  • Are the project and its outcomes realistic?
  • Are project timelines, implementation and milestones realistic and achievable?
  • Is the budget appropriate for delivering the project?

Once the SMA have completed their analysis, Program employees transcribe the results onto a triage sheet, which is the only analysis document that EAC members receive to carry out their own project evaluations. Tests were carried out on a sample of 50 applicant files to determine the quantity of information provided for EAC members on the triage sheet. The following table summarizes the test results:

Table 3: Number of Triage Sheets Containing Information from Internal Subject Matter Advisors

Fiscal Year

Number in Sample

Answers Only

Questions and Answers

Comments Only

Total

20152016Footnote 9

14

1

0

2Footnote 10

3

20162017

22

0

0

9

9

20172018

14

3

7

2

12

Total

50

4

7

13

24

The test results indicate that only 24 triage sheets contained information on the SMA’s analysis and that more than half of the triage sheets did not contain any information for EAC members. In addition, during interviews with SMA, many expressed doubts about the relevance of their analysis and said that they had never received comments or questions from EAC members regarding their work. Also, the tests found that in certain cases, some projects had received LAC funding despite receiving rather negative comments from SMA about their feasibility. Some EAC members said during interviews that they did not take into account the information contained on triage sheets in their own analysis.

These observations lead us to believe that the analysis of SMA are not systematically taken into account by the EAC in its selection of Program recipients.

Scores assigned to selected projects

The EAC is responsible for the third step in the project evaluation process. When the audit was conducted, all of the projects analyzed by SMA were forwarded to the EAC, regardless of the analysis results. The work of the EAC consists in allocating scores to projects to determine a final ranking for the projects, thereby identifying those that will receive funding, given that available funds are limited and cannot cover all eligible projects.

Starting in the 2017–2018 fiscal year, in order to increase the thoroughness and objectivity of the evaluations, each project is evaluated on the basis of merit by two EAC members. Each EAC member allocates a score to the projects he or she evaluates. In order to standardize the scores assigned to the various projects and avoid significant variances, a videoconference calibration session is held before the members do their evaluation.

Lastly, an annual meeting is held with all EAC members, during which the Committee makes its selections based on the best project scores. This final discussion is held to draw up the list of selected projects that will be sent to the Librarian and Archivist of Canada for approval. The analysis grid used by the EAC members contained very little information and did not make it possible to understand how they had assigned the final scores for the projects evaluated. In addition, the audit team was unable to obtain the minutes for these discussions because they were not available.

In terms of prioritization criteria, LAC undertook to define the Program’s funding priorities each year in order to support the prioritization and selection of funding applications. At the time the audit was conducted, no priorities were defined and documented since inception of the Program. Except for one member who liked the flexibility provided, there was consensus within the EAC on the need for clearer direction or guidance regarding Program priorities. According to Committee members, priorities would make it easier to select projects to be funded, and make decisions more objective and transparent. Some funding decisions were questioned by EAC members and SMA, as well as by some recipients and applicants.

Given the above observations, it was impossible to comment on the transparency and objectivity related to the final selection of contribution program recipients.

Recommendation 1
The Corporate Secretariat should review the evaluation process for funding applications in order to make optimum use of the expertise provided by SMA and allow the sharing of all relevant information for informed decision-making.

Recommendation 2
The Corporate Secretariat should review the evaluation process for funding applications in order to better document discussions leading to the final selection of projects that will receive funding, thus ensuring the transparency and fairness of the evaluation process.

Program records management

Those responsible for managing the program management did not develop a records management system. There were no guidelines or checklists to help Program employees, SMA and the EAC manage records. The content and order of records varied significantly in each file.

We expected an effective and complete recordkeeping system to be set up, and that all relevant documents for analysis and decision-making would be in the files.

In order to test records management, a random sample of 50 files was chosen, and interviews were held with Program team members. Of these files, 18 were also applicant files that had received funding. The sample covered the three-year period of the Program.

The Program team created a workspace in the Collaboration Portal (LAC’s designated repository for saving and managing its information). Electronic files were created for each applicant in the Collaboration Portal, and hard copies of the funding applications were scanned and included in the files. The documents were also copied in LAC Direct to allow EAC members to access them. Because the EAC is made up of members from outside LAC, they do not have access to the Collaboration Portal. The electronic files were consequently saved in duplicate, in the Collaboration Portal and in LAC Direct.

The review of the sample of files in the Program workspace found that essential documents were missing, including the acknowledgement of receipt, the acceptance or refusal letter, or progress or final reports. During the audit, of the 50 files reviewed, 16 were incomplete. The following table shows the breakdown of the incomplete files.

Table 4

Type of File

Number of Files in the Sample

Number of Incomplete Files

Recipient

18

12

Non-funded applicant

32

4

Total

50

16

Interviews conducted with EAC members revealed that records were not always placed in the same order in LAC Direct, which led EAC members to spend more time looking for the documents required for their analysis, thereby reducing the effectiveness of the process. The EAC members also said that there were sometimes unnecessary documents in the files, which also slowed down their work. These observations stem from the fact that there was no protocol for the content of files or the order in which they should be filed. There was also no methodology, such as a checklist, to ensure that all required documents had been received and put in file.

Program management has not put in place a structured management system for applicant and recipient files, as set out in the TB submission.

Recommendation 3
The Corporate Secretariat should ensure that a structured recordkeeping system is in place to make it easier to locate documents and improve the efficiency of the analysis process.

Program financial management

  • Payments to recipients complied with section 34 of the FAA and were issued in a timely manner.
  • Program staff did not carry out an analysis to determine the reasonableness of the budgets of funding applicants in each fiscal year.

We expected payments to recipients to be accurate, approved and issued in a timely manner, in accordance with contribution agreements, service standards and policies in effect. This means that section 34 of the FAA (which stipulates that a payment of any part of the federal public administration can only be issued, in addition to any other voucher or certificate that is required, when service is rendered) must be applied, that an authorized person signs and dates the necessary documents, that all required conditions must be met for the issuance of payments, and that the financial information contained in the reports submitted by the recipients must be accurate.

We also expected the Program’s administrative expenses to be coded so that they could be easily found and compiled. Lastly, we expected there to be an analysis mechanism in place to ensure that applicant budgets were reasonable and met the contribution program’s eligibility criteria.

The analysis of applicant files shows that the Program receives many more funding applications for projects than LAC is able to fund ($1.5 million). It is therefore important to exercise effective management to optimize use of the Program’s limited available funds.

Signatures on contribution agreements

As part of tests performed on contribution agreements, we wanted to confirm that they were signed by all parties concerned before the payment was issued, given that they are formal agreements. In total, for the three fiscal years reviewed, there were 140 contribution agreements concluded between LAC and SDHCP recipients.Footnote 11 Our tests indicated that all of the agreements were signed by an accredited LAC representative, however, one agreement was not signed by the recipient (in 2015-2016), and the payment was still issued for this agreement.

For the 2016–2017 fiscal year, we identified 23 cases where an agreement were signed but not dated. In addition, we found 13 cases where the date was added at a later time and retroactively dated before the recipient’s signature. It should be noted that the Program managers took corrective action since that fiscal year and that all 2017–2018 contribution agreements were signed and dated by both parties concerned, as per compliance requirements.

Section 34 of the FAA and timely payment

In order to test the application of section 34 of the FAA, we checked all recipient files covering the Program’s three fiscal years. The tests showed that all evidence of signature was on file, with the financial services team, and that all documents were signed and dated by a someone with financial delegation.

With regard to the standard-based payment test, out of the 50 files in the sample used for our analysis, 18 projects had received LAC funding. The analysis of these files revealed that 15 of the 18 payments were issued within 28 calendar days following the signing of the contribution agreement, thus meeting the established service standards. The three other payments were made three days or less late.

SDHCP administrative expenses

When the Program was approved, its administrators undertook to limit administrative expenses to $200,000 per yearFootnote 12 for each of the Program’s five fiscal years. LAC also planned on requiring 1.2 full-time equivalents (FTEs) to manage the Program.

A project code was established to account for these expenses. However, our analysis revealed that most of the administrative expenses, such as salaries, were not coded to this project. Consequently, we were unable to identify the exact amount of these expenses. A cost estimate was performed based on actual FTEs and the salaries of SDHCP management staff. The estimated administrative expenses for SDHCP exceeded the forecasts made when the Program was set. The following table provides information on the administrative expenses and the FTEs planned for the three audited fiscal years:Footnote 13

Table 5

Fiscal Year

Planned Administrative Fees

FTEs Planned

2015–2016

$210,380

2.25

2016–2017

$273,260

3.25

2017–2018

$361,217

4.00

As part of the program evaluation performed concurrently with the audit, a comparative analysis was carried out with other small cultural contribution programs. The analysis found that, although SDHCP administrative fees were higher than expected, they were within a range comparable to what is being done elsewhere, that is, administrative fees amounting to between 15% and 20%.Footnote 14

Budget analysis during funding application processing

When applicants submit an application for SDHCP funding, they must submit a budget detailing the costs of their project. The budget must undergo an analysis to confirm that:

  • the expenses indicated in the budget are eligible based on the Program criteria;
  • the project’s administrative expenses do not exceed 20% of the total cost of the project; and
  • the cost estimates submitted are reasonable.

The financial services group participated in the analysis during the first two fiscal years of the SDHCP. In seven of the 14 files, for which the financial services group confirmed the reasonableness of the costs, their expertise was used to significantly adjust some of the amounts submitted by the applicants, particularly the estimated rates to be paid to consultants, based on industry standards. However, since the 2017–2018 fiscal year, this analysis has been performed by SDHCP management and is limited to a summary analysis that is mostly based on the eligibility of the expenses submitted and not on the reasonableness of the expenses. Without performing a comparison with the rates in effect within the industry for similar projects, it is possible that LAC is paying more than the actual value of certain services.  

Recommendation 4
The Corporate Secretariat should review the process for analyzing applicant budgets to ensure that all costs are reasonable.

Monitoring mechanism

Recipients produced progress and final reports as required in the contribution agreements. However, no monitoring mechanism was in place to ensure that funds were spent as planned or reimbursed to LAC at the end of the fiscal year, when not used.

The Directive on Transfer Payments stipulates that managers in charge of managing transfer payment programs are responsible, among other things, for ensuring, through the timely assessment of recipient reports and other monitoring activities deemed necessary, that the recipient of a contribution has complied with the obligations and performance objectives in the funding agreement.Footnote 15 Monitoring activities are included in the best practices for transfer payment program management and constitute an effective control to ensure that the program’s objectives are met. We expected SDHCP management to have a monitoring mechanism in place to ensure that all terms and conditions set out in the contribution agreements are met and that funds allocated are used in accordance with these same terms and conditions.

Monitoring mechanism

The Program managers set up a series of controls to oversee the process for allocating funding to recipients. The controls set up by Program management include the reporting requirement and the related funding terms and conditions. The following table provides information on the reporting requirements:

Table 6
Contribution Amount

Progress Evaluation and Financial Report (January 15)

Final Evaluation and Financial Report

15% Holdback

Contribution of < $15,000

N/A

N/A

Contribution of > $15,000

Multi-year contributionsFootnote 16

Our tests were conducted using a sample of 50 recipient files, of which, nine required a progress report. The review of recipient files shows that in six files, the progress reports were submitted late, and that in three other files, the final reports were missing when it was time to do the tests. They were later provided by the Program team. In addition, there was no formal monitoring mechanism in place for recipients to ensure that reports were submitted or followed up on to ensure that funds were used for the activities indicated.

Financial reports of recipients

To ensure follow-up on the amounts allocated to recipients, a financial report must be submitted at mid-year and at the end of the fiscal year explaining how the funds were used. Using the same sample of 50 recipient files previously mentioned, our tests indicated that in four of the files analyzed, the financial information provided in the final reports by recipients did not match the amount allocated under the signed contribution agreement. At the end of the audit, no documentation was available to show that SDHCP management had followed up to explain these variances. It was not possible to complete the audit because of these deficiencies. The following table shows the variances between the contribution agreements and the financial reports submitted by the recipients:

Table 7

Fiscal Year

Contribution Agreement

Recipient’s Financial Report

Variance

2015–2016

$23,917

$20,329

$3,588

2015–2016

$69,946

$68,310

$1,636

2016–2017

$40,095

$39,971

$124

2016–2017

$71,530Footnote 17

$44,430

$27,100

A monitoring mechanism, proportionate to the organization’s needs, would enable Program management to ensure that all recipients submit reports, that the funds are used in accordance with the terms and conditions of the contribution agreements, and that unused funds are returned early enough to be allocated to other eligible projects, if applicable.

Recommendation 5
The Corporate Secretariat should develop a risk-based monitoring mechanism proportionate to its needs to ensure that funds are used in accordance with terms and conditions of the funding agreements.

Tools and training

Training given to EAC members was insufficient for their needs, while SMA were satisfied with the training received.

To support effective Program management, we expected that tools, guidance and/or training would be developed and implemented for those involved in managing or evaluating funding applications.

SDHCP management developed a range of templates, as well as operational directives for employees to use to manage the Program. Templates and guides were also developed for SMA and EAC members. These templates were reviewed at the end of each of Program’s funding cycles, and improvements were made in response to user suggestions. SMA and EAC members said that these tools helped them in their work.

SMA confirmed that the document with instructions for completing their analysis was clear and easy to understand. In addition, each year, SMA attended a guidance session on how to use the workspace in the Collaboration Portal to compile the results of their analysis. They confirmed that the guidance sessions and the instructions provided were enough to help them perform their duties.

EAC members had access to a guide providing them an overview of the Program’s mandate and their roles and responsibilities. In an interview, these people confirmed having received directives and documentation to help them in their work to evaluate eligible projects. At the launch of the Program, training was provided to EAC members, however, there was no training planned each year for them. Annual training identifying changes made to the Program would help EAC members better understand the evaluation process. For example, in an interview, some indicated that the SMA comments provided as part of their funding application analysis were not shared with them, which suggests deficiencies in the training for EAC members.  

Since the inception of the Program, the EAC has had difficulty keeping its members, which could be explained by the many responsibilities and heavy workload of these professionals, who are recruited to this committee as volunteers. In 2017–2018, the Program decided to double the number of EAC members – from 7 to 15 – to introduce peer evaluations, thus improving the funding application evaluation process. This change resulted in enhancing the challenge associated with recruiting and retaining EAC members. The high turnover rate for EAC members jeopardized the stability and effectiveness of the evaluation process as some members had to be recruited quickly to replace outgoing members for the 2017–2018 fiscal year, and thus could not receive adequate training to prepare them to do the evaluation work of projects submitted. Program management confirmed that they are aware of the problem and that they drafted an up-to-date list of potential candidates on a regular basis.

SDHCP management and employees took the Canada School of Public Service training on transfer payment program management, however only in 2017–2018. In previous fiscal years, Program staff interviewed indicated that they had to learn by themselves and did not have guidelines, such as a checklist, for managing applicant files.

Recommendation 6
The Corporate Secretariat should ensure that all EAC members participating in the evaluation of funding applications receive the required training in a timely manner to do their work. 


Annex A – Management action plan

Recommendations From the Review

Level of Risk

Action(s) to Take

Expected Completion Date

Responsibility

1 - The Corporate Secretariat should review the evaluation process for funding applications in order to optimize the use of expertise provided by Subject Matter advisors and allow the sharing of all relevant information for informed decision making.

High

For the 2018-2019 funding cycle, the Corporate Secretariat has implemented a new process for evaluating funding applications using an online platform. This new platform allows the various stakeholders to have easy access to all the relevant information for the evaluation of an application. During the training of the external committee members, in February 2018, among others, the importance of consulting the SMAs' comments was emphasized. In addition, the Secretariat has adjusted the evaluation criteria to eliminate duplication of questions among internal and external specialists to better target the expertise of each group. Finally, during the post-mortem exercise of the 2018-2019 fourth call, the Secretariat solicited feedback from SMAs and EAC members about the work and role of SMAs. The Secretariat will continue this practice in order to improve related processes and ensure the added value of the work of SMAs.

Completed

Corporate Secretariat

2 - The Corporate Secretariat should review the evaluation process for funding applications in order to better document discussions leading to the final selection of projects that will receive funding, ensuring the transparency and fairness of the evaluation process.

High

For the 2018-2019 funding cycle, the Corporate Secretariat has put in place a process to systematically document discussions at External Advisory Committee meetings.

The General Secretariat will work to further optimize the documentation of evaluators' reasoning for the 2019-2020 call for proposals, for example, by making comment fields mandatory in electronic evaluation forms.

Completed

 

 

 

December 2018

Corporate Secretariat

3 - The Corporate Secretariat should ensure that a structured recordkeeping system is in place to make it easier to locate documents and improve the efficiency of the analysis process.

Moderate

For the 2018-2019 funding cycle, the Corporate Secretariat has put in place a file structure organized in the LAC Collaboration Portal based on a defined protocol. It has established a naming convention for all tender documents.

The General Secretariat will continue to make improvements to this new structure, particularly in the use of a consistent nomenclature and checklist for safeguarding essential files. It is also working to further consolidate information on funding cycles and the Program to facilitate data analysis.

Completed

 

 

 

 

December 2018

Corporate Secretariat

4 - The Corporate Secretariat should review the process to analyze applicant budgets to ensure that all costs are reasonable.

Moderate

For the 2018-2019 funding cycle, the Corporate Secretariat, in collaboration with the Financial Services and Procurement Branch, reviewed the proposed budget assessment process and developed a checklist to be used by the program team at the triage stage. This process includes the assistance of a Financial Services and Procurement resource person for cost analysis, if required.

Completed

Corporate Secretariat and Procurement Branch

5 - The Corporate Secretariat should develop a risk-based monitoring mechanism proportionate to its needs to ensure that the funds are used in accordance with the terms and conditions of funding agreements. 

High

The Corporate Secretariat will review agency risks against a set of criteria (e.g. amount of contribution, former recipient) and develop a monitoring mechanism that targets medium-risk organizations. The Secretariat expects that only a limited number of organizations will be classified as “medium risk”, as the recipients are part of a low-risk sector and receive fairly moderate contributions (an average contribution of $32,000 over the first four funding cycles). The Secretariat expects only minimal changes in the profiles of recipients.

December 2018

Corporate Secretariat

6 - The Corporate Secretariat should ensure that all EAC members participating in the evaluation of funding applications receive the required training in a timely manner to do their work. 

Moderate

For the evaluation of 2018-2019 funding applications, the Secretariat has developed an online training (webinar) for all EAC members. Training and related materials distributed to members clarify members' roles and responsibilities and explain how to use the new tool to conduct online assessments.

Completed

Corporate Secretariat

Date modified: