Appendices - Evaluation of the Access to Documentary Heritage Program

On this page

Appendix A: Management Response and Action Plan

Recommendations Action Responsibility
1. Better coordination of activities and a prioritization of tasks among branches are needed to clarify the governance of the Access Program and the role of staff involved, regardless of the shape the program takes in the future.

We agree.

  1. The Operations Sector has reorganized its Branch Structure and clearly identify the functions of the Public Services Branch and the relationship of this branch to the others in the Sector.
    • Completed (April 2016)
  2. LAC has produced a new Strategy for Services to the Public.
    • Completed (April 2017)
  3. A Five Year Action Plan for Access (2017-2022) was developed in order to guide priorities, establish responsibilities and coordinate access initiatives.
    • Management Board approval expected November 2017

1) Chief Operating Officer

2) and 3) Director General, Public Services Branch

2. Program managers should undertake a review of output and outcome indicators to ensure that they are collected on an ongoing basis, that the indicators identified are useful for decision making, and that data collection is possible and practical so that the program’s progress and outcomes can be measured.

We agree.

  1. Develop and implement a new Performance Information Profile and Logic Model for the Public Services Program, establishing a comprehensive set of new indicators, results, and outputs to inform the annual Departmental Plan / Departmental Performance Report and other evaluation and accountability instruments.
    • Expected completion date: April 2018
  2. Public Services Branch will create and implement internal tools to centralize data gathering and reporting relating Public Services Program indicators and key operations.
    • Expected completion date: April 2018
  3. Develop and implement a new Performance Information Profile and Logic Model for the Outreach and Support to Communities Program, establishing a comprehensive set of new indicators, results, and outputs to inform the annual Departmental Plan / Departmental Performance Report and other evaluation and accountability instruments.
    • Expected completion date: April 2018
  4. Communications Branch will create and implement internal tools to centralize data gathering and reporting relating Outreach and Support to Communities Program indicators and key operations.
    • Expected completion date: April 2018

1) and 2) Director General, Public Services Branch

3) and 4 ) Director General, Communications Branch

3. Efforts should be made to complete the digitization of finding aids.

We agree.

  1. Scope requirements for the digitization of finding aids
    • Expected completion date: April 2018
  2. Begin the process to secure funding.
    • Expected completion date: April 2018

1) and 2) Director General, Preservation and Digital Operations Branch and Director General, Public Services Branch

4. To facilitate access to the collection on its website, LAC should improve the search tools found there as well as navigation

We agree.

  1. Develop an integrated web search tool for streamlined access to all LAC databases.
    • Expected completion date: March 2019
  2. Improve LAC Website architecture and navigation.
    • Expected completion date: March 2019

1) Director General Public Services Branch; Senior Director General and Chief Information Officer, and Director General Communications Branch

2) Director General, Communications Branch; Director General Public Services Branch, and Senior Director General and Chief Information Officer

Appendix B: Case Study #1: Access to Public Accounts of Canada

Type of case study: Low complexity - Published heritage

As stated in the mandate for the evaluation of the Access to Documentary Heritage Program (PAA 2.3), which was approved in spring 2016 by LAC’s DPEC, case studies were to be the mechanism used for further evaluation of how easy it is to navigate through Reference Services; the efficiency of the services provided to clients when they make access to information and privacy requests; and the discoverability, availability and accessibility of the LAC collection.

Case studies were identified on the basis of the following criteria:

1. Coverage of the following LAC collection areas (to be selected at the time of the evaluation):

  • Government Records;
  • Private Archives; and
  • Published Heritage.

2. Complexity of the LAC collection (to be selected at the time of evaluation):

  • Simple;
  • Medium; and
  • Complex.

Taking the example of the Government of Canada's Public Accounts, this case study on published heritage is intended to illustrate the discoverability of the LAC collection and the availability and accessibility of the descriptions and the terms and conditions set out in the three LAC policy instruments, namely, the Access Policy Framework, the Policy on Making Holdings Discoverable and the Policy on Making Holdings Available.

Evaluation questions:

Performance:

  • To what extent has the program improved access to documentary heritage?
  • How well do clients find what they are looking for in the LAC collection?

Background:

In reviewing LAC’s RPPs and DPRs for 2011-2012 to 2015-16 (the period of the Access to Documentary Heritage Program review), it was noted that the three commonly used concepts regarding access to the LAC collection (discoverability, availability and accessibility) are not clearly explained or used consistently. There are in fact frequent references in the RPPs and DPRs to LAC’s desire to make the entire collection accessible. That commitment in itself is understandable, but it is not easy to achieve. In fact, a certain portion of LAC's collection is not accessible, as LAC must comply with policies and with administrative or statutory restrictions, with respect to copyrighted materials in particular. The Access Policy Framework indicates that LAC takes into account the legislation, regulations and policies in place within the Government of Canada and the organization. It further states that, in certain circumstances, the legal and political environment in which LAC exists will temporarily delay access to LAC’s documentary heritage holdings or will limit options in that regard.

As discussed below, we expected that it would be easy to find the Government of Canada’s Public Accounts on the LAC website. Our experience in this case study demonstrates that the situation is more complex.

Methodology

As mentioned earlier, the concepts of discoverability, availability and accessibility will be defined first. Access mechanisms will then be illustrated in relation to those three concepts, using the example of the Government of Canada Public Accounts. The first step will entail finding this publication on the LAC website. The next step will be to see whether the Public Accounts are available and, finally, whether they are accessible, online or otherwise. The conclusion will present the associated challenges, observations and findings.

The Public Accounts of Canada are prepared annually by the Receiver General, as required under section 64 of the Financial Administration Act, and they present the Government of Canada’s financial transactions. This publication is acquired by LAC under the Legal Deposit Regulations.

For this case study, the evaluation team acted as any external client would to conduct a search. No contact was made with and no advice was sought from a staff member at LAC.

Definitions and descriptions

Definitions of the three concepts used in this case study are presented below.

Discoverability

Documentary heritage and other information sources can be discoverable when it is possible to establish their existence, description, location and availability. (Library and Archives Canada, Policy on Making Holdings Discoverable, December 9, 2013)

The Policy on Making Holdings Discoverable (2013) is part of the Access Policy Framework (2016). Activities relating to discoverability are the first ones to be carried out so that anyone can identify a document, photo or other item in the LAC collection. The preparation of a description (descriptive metadata) is the essential tool for discoverability. Items become discoverable when a first-level description is created in an LAC system (such as Amicus or MIKAN) endnote64 at the time of acquisition.

The description created at that time is helpful to LAC employees in locating items in the collection. A description as well as the creation of search instruments and indexing are essential in order for items in the collection to be found. This work requires resources and time; depending on the size and complexity of the materials acquired and the number of resources dedicated to processing, it can take several weeks or even months. Archivists and librarians also adhere to national and international standards for descriptions, cataloguing and the arrangement of archival holdings. Such standards were first developed to facilitate management of the collection, but they are not always easy for people who do not work in archival or library science to understand. The average Canadian could easily become lost in the descriptions and in the structure of the holdings in particular.

Availability

Documentary heritage and other information sources are available if they are not subject to any legal or political constraints and if users are able to consult them. (Library and Archives Canada, Policy on Making Holdings Available, June 9, 2014)

As with the above-mentioned policy, the Policy on Making Holdings Available (2013) falls under the Access Policy Framework. Although LAC must ensure that the entire collection is discoverable, it is not possible for the organization to make its entire collection available. LAC must comply with various statutes, regulations and restrictions on access to its collection, including the Privacy Act, the Access to Information Act, the Policy on Copyright Management and the agreements with donors of private archives. Similarly, records of federal institutions may entail certain access restrictions.

That said, in recent years an effort has been initiated to reduce access restrictions through such initiatives as Open Government, the aim of which is to make government records open by default. Otherwise, a date on which restrictions are to be lifted should be identified as soon as an acquisition is made in order to make the records available. Block review of government records involves evaluating a set of restricted documents with a view to making them available. Through block review, it was possible to process 9 million pages of historical government documents and make them available between 2011–12 and 2013–14 (2013-14 DPR).

Accessibility

Documentary heritage and other sources of information are accessible if physical, technological and geographic barriers to obtaining content are removed and if they can be used by as many people as possible. (Library and Archives Canada, Access Policy Framework, August 30, 2011)

As noted above, there are a number of barriers to accessing the LAC collection. Accessibility of the online collection is strongly desired or is quite simply necessary for individuals living outside the National Capital Region. To consult documents in analog format, clients must travel to 395 Wellington Street in Ottawa. To address this constraint, in recent years LAC has implemented a digitization plan based on topics of interest to clients, such as military heritage and indigenous issues. LAC has also implemented a digitization plan in collaboration with partners and has successfully completed the digitization of a large number of documents, photographs, films and documentaries. 

Data collection

Below are all the steps that the evaluation team completed in order to access the Government of Canada’s Public Accounts.

Step 1: Searching on the LAC website

The data collection first consisted of finding the documents. Using the “Search All” menu on the LAC website, the term “Public Accounts of Canada” was entered, along with the term “Les comptes publics du Canada”. The “Discover the Collection” menu was also explored, and there were searches by “Topic” and by “Type” and using the A-Z Index. This search found 277 items in French and 2,199 items in English. Even when the “Sort” feature was used, a number of pages had to be explored before a document that seemed to be the desired document was found. At that stage of the evaluation, having only the title and some other basic information found on the LAC website was not enough to say whether the document was relevant.

Even when a specific year was added to the search on the LAC website, the results did not provide direct links to documents, or they provided links that were not working. Therefore, searching on the LAC website was a long and complex process. A number of pages had to be sorted before the evaluation team could access what it was looking for and before the Government of Canada Public Accounts could be found.

Step 2: Searching outside the LAC website

The evaluation team conducted another test to see whether it was possible to access the Public Accounts of Canada more easily. From the LAC website, clicking on the “Electronic locations” link leads to an external Internet Archives platform, the Public Services and Procurement Canada (PSPC) site and the Receiver General for Canada site. On the latter site it was possible to locate the last available report (2015) for the Public Accounts of Canada. The site then refers to the Government of Canada’s Open Data portal and to Library and Archives Canada for reports produced between 1995 and 2014. By following this link, it was possible to find the reports archived on the LAC website via a system called Amicus, which is used to search for published documents (Amicus Nos. 16986958 and 16987016).

It was therefore possible to locate and access the archived reports (from 1995 to 2014) of the Public Accounts of Canada on the LAC website, but by going through an external site. As mentioned, some descriptions bring the client to external platforms such as Archive.org (e.g.: Amicus 7007032).

Step 3: Other search possibilities that were tested

Another way to find the Government of Canada Public Accounts was to use the general Government of Canada site at canada.ca. Once on this site, it was necessary to click on “Departments and agencies” to find the TBS site. The subject “Public finances” could then be found. Clicking on “Public finances” takes the user to the “Reporting of government spending” link and ultimately the “Public Accounts of Canada” link. Ultimately it was possible to find the following: The Public Accounts of Canada for previous years (starting with 1995) are available in a PDF format and in HTML format since 2012 from the Library and Archives Canada. In clicking on this link it is possible to find the Public Accounts of Canada archived (from 1995 to 2014) on the LAC website. However, there is an assumption that the client knows the information is available on the TBS site in order to proceed in this way.

The quickest and most efficient method remains the use of a search engine such as Google for published documents. This would have immediately yielded, as the first search result, the link to the Receiver General site, which presents all relevant links to the Public Accounts of Canada. All of these sites lead to the LAC website for archived documents between the years 1995 and 2014.

Conclusion

The description with the relevant links can be found in the list of search results on the LAC website, but the steps needed to sort and find the correct result take time. Access to the documents required sorting a large number of results obtained on the LAC website. In addition, the lack of detail in the descriptions made research more complicated. Moreover, some item titles were confusing, as the title indicated "electronic" or "online" but the link was either missing or not working. It was therefore possible to find and access the most recent report available (2015) for the Public Accounts of Canada on the PSPC, Receiver General for Canada or TBS websites. It was also possible to find the Public Accounts of Canada reports archived on the LAC website, but it was first necessary to go through an external platform and then follow the links to access the documents. A client who was not persistent could have given up on searching the LAC website and then chosen to use a search engine such as Google.

Challenges

As mentioned, it was difficult to find the Government of Canada Public Accounts archived on the LAC website. For archived documents, such as the Public Accounts of Canada prior to 2015, the above‑mentioned sites lead us to the LAC website. This case study therefore demonstrates the importance of having a detailed and complete description in order to make documents on the LAC website easier to find and access without having to search on external sites. Without a detailed and complete description, clients may find it difficult to find precisely what they are looking for, as there may be a large number of search results. In this case, the descriptions did not make it easy to find the document being looked for.

Observations

While LAC is focusing on increasing the accessibility of its collection, it would be advised to better explain the concepts of availability and accessibility to its clients, partners and employees. The terms used in the RPPs and DPRs and in business plans may be confusing. In fact, those documents should not suggest that the entire LAC collection will eventually be accessible; the limits to access should be clearly presented and the appropriate terms (discoverable, available and accessible) should be used and explained.

Findings

To what extent has the program improved access to documentary heritage?

In the case of the study on the Public Accounts, an improvement in access through the LAC website could not be established.

How well do clients find what they are looking for in the LAC collection?

Access to government publications takes time, and searching on the LAC website is a long process. It is quicker and easier to go through external sites such as PSPC, Receiver General of Canada or TBS sites and to use a search engine such as Google to find archived documents from the Public Accounts of Canada even if they send the client back to the LAC site.

Appendix C: Case Study #2: Access to Records on the Ground-Breaking Ceremony for the St. Lawrence Seaway on June 25, 1959

Type of case study: High complexity - Government record

As stated in the mandate for the evaluation of the Access to Documentary Heritage Program (PAA 2.3), which was approved in spring 2016 by LAC’s DPEC, case studies were to be the mechanism used for further evaluation of how easy it is to navigate through Reference Services; the efficiency of the services provided to clients when they make access to information and privacy requests; and the discoverability, availability and accessibility of the LAC collection.

Case studies were identified on the basis of the following criteria:

1. Coverage of the following LAC collection areas (to be selected at the time of the evaluation):

  • Government records;
  • Private archives; and
  • Published heritage.

2. Complexity of the LAC collection (to be selected at the time of the evaluation):

  • Simple;
  • Medium; and
  • Complex.

For the choice of case study #2, which involves a government record, the following additional criteria were developed: (1) a government record that did not relate to a topic of current interest, such as World War I; (2) a file that dated back more than 25 but not more than 100 years, to ensure that the documents had been transferred to LAC; (3) a file that was sufficiently diverse, with documents, photos and sound, video and documentary recordings; (4) an event that was important to Canada; and (5) a file containing restricted-access documents. The ground-breaking ceremony for the St. Lawrence Seaway met all of those criteria.

The purpose of this case study is thus to illustrate the extent to which Government of Canada records are discoverable, available and accessible. In particular, this case study covers the issue of the ease of use and effectiveness of LAC’s reference services and looks at the way that requests are made under the Access to Information Act.

Evaluation questions

Performance:

  • To what extent has the program improved access to documentary heritage?
  • Is it easy for a client to use the services offered by Reference Services?
  • How effective are the services offered to clients under the Access to Information Act?
  • To what extent is it possible for clients to find what they are looking for in the LAC collection?
  • To what extent do clients find what they are looking for in the LAC collection?

Background

The ground-breaking ceremony for the St. Lawrence Seaway is an important historical event for Canada. It took place on June 25, 1959, and was attended by the Prime Minister of Canada, the Right Honourable John Diefenbaker, U.S. President Dwight D. Eisenhower and Queen Elizabeth II of England.

Methodology

This case study involved determining whether government records can be found and accessed easily. The first step would be to find and attempt to access documents on the LAC website. If the documents were available but not accessible online, they could be consulted on site at Reference Services, located at 395 Wellington Street in Ottawa. For restricted documents, an access to information request would be prepared to assess how the services are working and to validate whether those services are effective and whether the requested information is provided in accordance with the terms and conditions of the Access to Information Act.

This research was done by presenting ourselves as outside persons interested in researching a particular topic. Data collection took place in a number of stages. The case study was to highlight the challenges encountered and the observations and findings that came out of it.

Data collection

Step 1: LAC website

Online research was done by limiting the search to government records and excluding private publications and archives. A search using "Discover the Collection", "Search Online" and "Search All" was performed by typing in the words “Inauguration de la Voie maritime du Saint-Laurent" [Ground-breaking ceremony for the St. Lawrence Seaway]. A list of relevant documents was found through this search. The results were sorted by the team of evaluators and a list of records that seemed relevant to the search theme was prepared. At that stage of the research, it was not possible to determine whether the documents were relevant from only the title and some other basic information. A search on the website turned up a number of items related to the opening of the St. Lawrence Seaway. However, some of the results did not pertain to the June 1959 opening ceremony even though the search was done using the key words “ceremony” and “opening of the St. Lawrence Seaway”. Further analysis was required. In addition, the English and French searches did not yield the same results. The research on the LAC website turned up 24 government archive items. When duplicates or irrelevant items were deleted, seven potentially interesting items remained. Of those seven items, only the photos were accessible online.

The evaluation team also found that, although the records pertaining to the opening of the St. Lawrence Seaway dated back to June 1959, a number of them were still restricted under the Access to Information Act. Two open files were found:

  • “St. Lawrence Seaway Authority (Canada) (graphic material), 1955-1972. (R1195-O-X-E)”, which contained 808 photographs and 16 postcards.
  • “Water Resources and Development - St. Lawrence Water and Power Project - Ground Breaking Ceremonies for Seaway Project (1953-1959)”.

The other five files were restricted under the Access to Information Act. It was noted that the restrictions varied. For four of those files, there was the following notation: “Warning: Descriptive record is in process. These materials may not yet be available for consultation.”

Step 2: LAC Reference Services at 395 Wellington, Ottawa

A user card was obtained by completing the electronic form and contacting Reference Services directly. The card was used to request an appointment at Reference Services by completing a form on the website. Reference Services offered two appointments, one with an archivist and one with a librarian. Both meetings provided an opportunity to learn more about the research topic. The professionals did some research in advance and answered the questions. They provided quick service of excellent quality. The appointment with the archivist provided an opportunity to learn about the finding aids. When a file contains a large number of items, a finding aid may have been created to help clients find what they are looking for. Through the finding aid, clients can order the articles they are interested in consulting. For example, the "The St. Lawrence Seaway Authority fonds [multiple media], 1817-1996" item found on the LAC website contains 46.65 metres of textual records, 3,830 technical drawings, 864 maps and other cartographic materials, 295 architectural drawings and 807 photographs. Therefore, finding aids are essential to locating relevant material in large archival holdings, as with the records found for this case study. However, not many of these finding aids are digitized and accessible online. They are primarily paper-based and therefore can be accessed only on site through Reference Services. This is a significant barrier to access for clients who do not reside in the National Capital Region.

An appointment with a librarian also took place. She did some research prior to our visit and provided a list of 13 books relevant to the opening ceremony for the St. Lawrence Seaway. Because these were not government records, they were not added to the elements evaluated in this case study.

A second visit to Reference Services was required to learn more about the finding aids and to order the material to be consulted. Files that are "open" can be ordered either online or on site at 395 Wellington Street in Ottawa by filling out a paper form (archival request form). Three boxes were ordered (RG 52, Volumes 173, 174 and 175; MIKAN No. 3807514). They had been previously identified with the archivist’s assistance. A request was submitted to the employee responsible for ordering. Two of the three boxes could be ordered, but Box 175 could not. The employee was unable to provide the specific reasons but suggested that the material could be undergoing processing or that there were restrictions on access. 

Step 3: Return to Reference Services

A third visit was made to Reference Services to consult the materials that had been ordered and to validate the process for submitting a request under the Access to Information Act. The finding aids were consulted in order to locate the information on the “Dossiers de référence se rapportant aux cérémonies d'ouverture officielle (document textuel), 1955-1964”. One finding aid in particular was interesting: paper finding aid 52-2. It was difficult to physically locate this finding aid, but it was possible to find it. In the 52-2 finding aid, volume 5 was found, and it contained 12 documents that appeared interesting for the research. Again, at this stage of the research, having only the title of the documents made it difficult to say whether they were relevant to our case study. Unfortunately, no file number or information regarding access to the documents in question was available in the finding aid.

A visit to the Reference Services reception desk was made with the list of 12 documents to seek assistance in finding the file numbers. The person at the reception desk confirmed that there were none. There was a “Restrictions vary” notation on the LAC website, but the person at the reception desk did not have any information on the files available and those that were restricted. There were three possible options: 1) making a new request to the Reference Services clerk on the third floor with the titles of the 12 documents; 2) asking to meet with someone who handles Access to Information Act requests in order to clarify the access conditions; or 3) submitting an access to information request for all of the files. The latter option was chosen, and a formal access to information request was prepared for volume 175. Since the St. Lawrence Seaway opening ceremony was held in June 1959, we were confident that we would find some relevant documents and photos in this file (archival reference R1195-5-9-F or E; MIKAN number 164935 or 160020).

However, during this second visit to Reference Services it was possible to look at both boxes ordered during the first visit. The service was efficient in that both boxes were already available for consultation. It was possible to go to the 3rd floor of 395 Wellington Street in Ottawa to consult the two boxes. After the order card was shown, the person responsible brought the two boxes for us to look at. She asked us to put on a pair of the white gloves available on all of the tables, and then the boxes could be opened. There were photographs in the first box. A photographer had in fact taken some pictures of the St. Lawrence Seaway under construction, but most of the photos depicted the day-to-day lives of Canadians across the country. There were very few photos relating to the St. Lawrence Seaway and none of the opening ceremony. The second box was more interesting because it contained photos (808 photos, according to the item description) showing the construction of the St. Lawrence Seaway, and the Saint-Lambert locks and the Welland Canal in particular. Other photos showed Canadian and American officials involved in the construction of the St. Lawrence Seaway. However, although these photos were interesting, they were not relevant to the research. After the analysis was completed, the photos were meticulously put back into the boxes, with an attempt to follow the same order in which they had been found. The person responsible was advised that the visit was over and that the boxes could be returned to the Preservation Centre. We left the consultation room without any questions being asked. To our knowledge, the contents of the two boxes that we were returning were not checked at that point. No one checked to ensure that we were not taking any items from the boxes with us.

Step 4: Access to restricted documents

The online form for an access to information request was completed, printed and sent to 395 Wellington Street in Ottawa with a $5.00 cheque in order to comply with the terms of the Access to Information Act. Four days later, confirmation of receipt was received. This confirmation indicated that a response to the access to information request would be received within 30 days.

Less than 30 days later a CD containing the 12 documents ordered under the Access to Information Act was in fact received. These 12 documents were very relevant, as they pertained to various aspects of the opening ceremony for the St. Lawrence Seaway: the Queen's ship, the Britannia, and all of the logistics pertaining to the ship; the minutes of various planning meetings for the opening ceremony;  the RCMP’s involvement in the opening ceremony as well as the Queen’s travel elsewhere in Quebec and in the rest of Canada; the involvement of National Defence, including  the Royal Canadian Air Force, in the opening ceremony; different versions of the highly detailed agenda for the opening ceremony; the hosting of U.S. dignitaries, including the U.S. President; and many other useful documents on that day.

Conclusion 

In conclusion, it was possible to identify a number of items pertaining to the opening ceremony for the St. Lawrence Seaway, although certain items that were either duplicates or not relevant had to be rejected. Some photos were available online but other types of documents, such as textual records, maps and technical drawings, were not. They were available for consultation through Reference Services, with the exception of certain items that were restricted. However, it may be difficult to identify what is restricted as opposed to what is available on the LAC website. A finding aid is useful to help determine which of the group of documents is relevant, but not all finding aids are online and those that we consulted had not been updated. Nevertheless, Reference Services provided access to more documentation and other information, such as books we had not identified during the initial research. Since it is not always possible to know what is open or restricted, an access to information request had to be completed, adding an additional step before the documents could be accessed. If that information had been available on the LAC website or in the finding aid, it would have been possible to consult them directly by requesting them from Reference Services.

It should be noted that, as a result of our research on the ground-breaking ceremony for the St. Lawrence Seaway, the access service determined that the records requested under the Access to Information Act no longer had any access restrictions. LAC therefore updated the information on the LAC website, and in early January 2017 the site indicated that these documents were open. Therefore, if another client wants to see these documents it will be possible to do so without going through the Access to Information Act. Finally, it appears that monitoring should be in place to minimize the risk of loss or theft of pieces of the LAC collection when clients consult documents on site in Ottawa.

It therefore appeared that it may be difficult to locate government records with only a title and a primary description. However, once a document with no access restrictions has been located, it is available. If the document has been digitized, it can then be accessible online.

Challenges

Research such as this presents is a challenge for clients who are not familiar with the LAC website. Clients need to identify what they are looking for in order to select the proper items displayed in the search results. Clients need to think about searching in French and English, because the results are not the same. In addition, it is necessary to go to Reference Services, in many cases several times, and to be able to use the finding aids, as many are analog (paper format) and are therefore not accessible for online consultation.

Observation 

Conducting a search in the LAC collection is much easier when there are finding aids, such as finding aid 52-2, which contains a typed list of files pertaining to the opening of the St. Lawrence Seaway. Since not all finding aids are digitized, clients from outside the National Capital Region are unable to consult them easily unless they plan to travel. These finding aids are also difficult to understand without the help of professionals and they do not provide certain pieces of relevant information, such as the file number, creating a degree of frustration for clients. Furthermore, they do not indicate whether the file is open or closed (restricted). Such information would be helpful in avoiding the need to file a request under the Access to Information Act.

The following specific list of observations is provided for consideration:

  • Finding aids are primarily paper-based and are available for on-site use only. Those that we consulted were in paper format, which meant they were not accessible online and did not provide any information regarding access conditions.
  • The templates for the finding aids are very different from one another; they are not standardized, which does not make them easier to understand.
  • A lack of information regarding access conditions may lead clients to submit access to information requests when documents are open. Clear identification of access conditions and an indication of the date on which a restricted document will become available would provide quicker access to documents for clients.
  • During on-site visits, we did not notice any specific monitoring to prevent the loss or theft of material from the LAC collection.

Findings

To what extent has the program improved access to documentary heritage?

This case study shows that it is possible to access information on topics as specific as the St. Lawrence Seaway opening ceremony on June 25, 1959. However, a current and comprehensive finding aid would have facilitated the search in LAC’s documentary heritage. 

Is it easy for a client to use the services offered by Reference Services?

The experience with this case study demonstrates that Reference Services provides fast, efficient and high-quality service with regard to obtaining an access card or meeting with archivists or librarians.

How effective are the services offered to clients under the Access to Information Act?

Through this case study it was possible to evaluate the services under the Access to Information Act. The online request was simple to complete. An acknowledgement of receipt as well as the documents that had been requested were received within the service standards mentioned on the LAC website.

How possible is it for clients to find what they looking for in the LAC collection?

It may be difficult to interpret the results obtained through a search on the LAC website. The title of a document and a summary description of the document do not actually indicate whether a particular document is relevant. In many cases, documents are not relevant or are duplicates. Moreover, the website does not suggest that clients conduct their research in both of Canada's official languages in order to obtain more results. Finding aids are useful and can help identify what clients are looking for as long as they are aware that these aids are available.

To what extent do clients find what they are looking for in the LAC collection?

This case study shows that making a search, even on a specific topic, presents challenges for a client who is unfamiliar with the LAC website. It is necessary to clearly identify the subject of the search in order to select the correct items to be displayed in the search results, and to think of searching in both French and English. In addition, it is necessary to go to Reference Services located in Ottawa to be able to view the available documents and use the finding aids, as many are analog (paper) and therefore not accessible for online consultation.

Appendix D: Case Study #3: Access to the Burton Cummings Fonds

Type of case study: Average complexity – Private archives

As stated in the mandate for the evaluation of the Access to Documentary Heritage Program (PAA 2.3), which was approved in spring 2016 by LAC’s DPEC, case studies were to be the mechanism used for further evaluation of how easy it is to navigate through Reference Services; the efficiency of the services provided to clients when they make access to information and privacy requests; and the discoverability, availability and accessibility of the LAC collection.

Case studies were identified on the basis of the following criteria:

1. Coverage of the following LAC collection areas (to be selected at the time of the evaluation):   

  • Government records;
  • Private archives; and
  • Published heritage.

2. Complexity of the LAC collection (to be selected at the time of the evaluation):

  • Simple;
  • Medium; and
  • Complex.

This case study covers private archives and is designed to illustrate how easy it is to find private archives held by LAC and how available and accessible they are on line. The study focuses on the comprehensiveness of descriptions and how they are used. Description is an activity that LAC has been wanting to improve for a number of years. endnote65 This case study will make it a focus of special attention so that it can be determined whether description facilitates research and access to the private archives collection.

Evaluation questions:

Performance

  • How possible is it for clients to find what they are looking for in the LAC collection?
  • How well do clients find what they are looking for in the LAC collection?

Background:

As there were many possible research topics, a list of criteria was drawn up to select the purpose of this case study. The topic that was chosen was not the focus of special attention at LAC or a matter of current interest. An individual was selected at random from the list of the top 100 Canadian personalities prepared by the CBC/Radio-Canada television network in 2004. This case study deals with documentation related to Burton Cummings, a major Canadian musical personality who played an important role in Canada. 

For illustrative purposes, this private archive search is not exhaustive or representative of all searches that can be made on the LAC website. Nevertheless, it illustrates the process a client might go through in a similar search of LAC’s private archives.

Methodology

This case study began with an exploration of the LAC website. Different sections of the website, such as “Browse by Type”, “Browse by Topic” and “Search All” were explored in order to find the documentation relating to Burton Cummings. The description was carefully reviewed to determine whether the description made it easier to search and whether it could be used to pinpoint what was being looked for on the LAC website. Following those searches, the evaluators sorted and analyzed the results to identify the relevant elements. Availability and online access of the items in question were then validated.

The conclusions with regard to searching private archives will illustrate the level of complexity of a search on the LAC website. Other observations and findings pertain to the completeness of the descriptions, the relevance of the results obtained and the online accessibility of the private archives in this case.

For this case study, the evaluation team acted as any external client would to conduct a search. No contact was made with and no advice was sought from a staff member at LAC.

Data collection

The data collection first entailed finding documents on the LAC website. Different searches were carried out, using “Search All”, “Discover the Collection”, “Navigate by Topic” and “Navigate by Type” as well as in the “Portrait Portal” under “Burton Cummings”. A search was also made on the various social media used by LAC, such as Facebook, Twitter, Flickr and blogs.

Step 1: Exploration

Using the “Search All” function on the LAC website, it was possible to find 183 library items and 49 archival items (including 30 images). Of the 49 archival items, 47 were from private holdings and two were government records. For the purposes of this case study, it was more important to focus on the 47 items from private holdings. Of those 47 items, 30 were photographs and were accessible online. Two items were identified as restricted access. A fairly high level of duplication of the same items with French and English descriptions was discovered. In the end, the number of archival items that were found was 29 and not 47. In four cases, the search results indicated that the documents were open. For the others, consultation of the detailed sheets revealed that access varied depending on the item (open, closed for processing, restricted or “Restrictions vary”). Clicking on the links for the items that were open showed that they were available but not accessible online. To consult them, it was necessary to submit a request to Reference Services at 395 Wellington Street in Ottawa.

Aside from the title and the brief description, this initial exploration found that there was not much information to assess the relevance of the items found. A detailed and comprehensive description would have helped identify the desired items more quickly.

Step 2: Discover the Collection

A search in the “Discover the Collection” section turned up a very large number of links on the LAC website as well as a number of repetitions of the same links (type, topic, index) that increased the chances of achieving results. However, it did make searching cumbersome, as it was necessary to carefully examine each link to determine whether it met the search criteria. Sometimes the link title was not sufficient: it was necessary to go on to the next page to determine whether the link was relevant. Even if the “Sort” feature was used, it was necessary to explore several pages before the desired document could finally be identified. Again, a more detailed description would have indicated whether the links obtained from the search were relevant.

Step 3: Search by Type

By performing a search by type, it was possible to find the "Music Archives at the National Library of Canada: A Guide (1994, 2003) - and photos, archived", but this search did not yield any relevant results. Similarly, from the “Discover the Collection” menu, it was possible to arrive at “Dictionary of Canadian Biography Online”, which is a link to an external research platform. However, accessing this link did not turn up any available information on Burton Cummings.  

Step 4: Available biographies

A search in “Available biographies” did not yield any results for Burton Cummings. However, 583 results were obtained from the “Music, Films, Videos and Sound Recordings” section. These were mostly interviews by the artist with various media. All of the items, without exception, had an ISBN, meaning they were published documentary heritage, which was not covered by this case study. Also, in the Portraits Portal there were links to 14 photographs that were accessible online.

Step 5: Social media

A search of LAC social media (Flickr, Facebook, Twitter and blog) did not produce any information about Burton Cummings.

Step 6: Google

Finally, using the Google search engine it was possible to find a personal profile of Burton Cummings on the LAC web site, with 14 links to online archival items. Despite the earlier efforts to access those items through the LAC Portraits Portal, it was not possible to do so. The only way to access them was through Google. The following message to the public on the Portrait Portal page came up when it was accessed through Google: “As part of its Modernization, Library and Archives Canada has decided to post as much material as possible online. Given that this material has been acquired and described over several decades, there is a likelihood that some descriptions could be incomplete in light of the large scale of the collection and the wide range of sources. Any contributions that could complete our descriptions can be sent to us via email at BAC.Portraits.LAC@canada.ca.”

Conclusion

As the case study on Burton Cummings shows, searching takes time. It was in fact necessary to go through various steps and links such as “Search All”, “Discover the Collection” and Google before achieving results. As noted above, searches using “Browse by Type” as well as “Available biographies” and “Social Media” did not produce any results. It may therefore be difficult to access relevant information, especially for someone who is new to this type of thing. Despite those difficulties, it was possible to locate items in the LAC collection relating to Burton Cummings' private archives by following the appropriate links and accessing the photos, which were available online. Other types of textual records were available but were not accessible online.

Observations

Once the research had been completed, the summary description with the links was found in a list of search results on the LAC website, but the steps for sorting and finding the correct result took time. To access the documents, it was sometimes necessary to sort a large number of search results and to try to understand descriptions that were lacking in detail. Instead of persevering with the LAC website, a client could choose to use a search engine such as Google.

Although textual records were available, they were not accessible online. To consult the documents, a client must submit a request to Reference Services, located at 395 Wellington Street in Ottawa. Documents can usually be consulted the following day as long as the person is able to go to Ottawa. For someone who does not live in Ottawa, this is not very practical.

It would be beneficial for LAC to clearly explain the concepts of “availability” and “accessibility” to clients to help them understand the search results.

Findings

How possible is it for clients to find what they are looking for in the LAC collection?

This case study showed that it was possible to locate photos and textual records in Burton Cummings’ private archives on the LAC website via the “Search All” and “Discover the Collection” functions. Photos could be accessed online. Textual records were available for on-site consultation but were not accessible online.

How well do clients find what they are looking for in the LAC collection?

This case study shows that it is possible to locate material in the private archives of the LAC collection. It also demonstrates the importance of having a detailed and complete description to facilitate the discoverability of particular items in the collection. Without such a description, it is difficult for clients to find precisely what they are looking for, as there may be a large number of search results. 

Appendix E: Logic Model, Access to Documentary Heritage Program

Appendix E: Logic Model, Access to Documentary Heritage Program
  • Logic Model: text version

    Appendix E shows the logic model for the Access to Documentary Heritage Program. To read the logic model, it should start from bottom-up by the activities, then the outputs (products of the Program) and the outcomes (what the Program try to change).

    Activities

    Access to Documentary Heritage (PAA 2.3)

    Outputs

    • Descriptions (metadata, authority, standards)
    • Services (responses to enquiries, ATIP, reference, website visits, etc.)

    Immediate Outcome

    Improved access to Canada's documentary heritage

    Intermediate Outcome

    Improved use and engagement with Canada's documentary heritage among the general public.

    Ultimate Outcome

    Canada's continuing memory is documented and accessible to current and future generations (PAA 2.0)

Appendix F: LAC Priority Activities in Reports on Plans and Priorities (RPPs)

Topic (Priority Activity) RPP 2011–2012 RPP 2012–2013 RPP 2013–2014 RPP 2014–2015 RPP 2015–2016

Descriptions

Implement a single metadata-based descriptive structure Have better descriptions New approach to the description of documentary heritage Describe content as quickly and clearly as possible to facilitate discovery and access Describe the content of a maximum number of documentary resources as quickly and as clearly as possible to facilitate discovery and access
Finding aids -

Digitize finding aids

-

Enhance access to documentary heritage by expanding online finding aids

Create and maintain digital toolkits and finding aids to facilitate discovery of documentary resources

LAC will continue to create new search tools and finding aids
Digital content and online access to the collection -

Ensure that Canadians are able to discover, consult and share LAC content when, where and how they want

Increase the amount of digital content available to Canadians

Expand access to collections through the implementation of a new digital services model to support clients through self-service access to content

Implement a multi-year digitization strategy

Facilitate access to LAC’s documentary resources and facilitate consultation of the most popular material

In conjunction with partners, LAC is leading digitization initiatives to provide Canadians with better access to its collection and increase the amount of content available online

Continue to implement the strategy involving digitization of the most frequently requested documents

Increase access to documentary heritage through digitization initiatives and increasing the amount of online content

 

Maximize the amount of content accessible to LAC clients online
Direct client services -

Reassign employees from in‑person to online services

New service model for Reference Services

Continue the renewal of LAC services so clients have access to quality services and a maximum amount of content online Also provide in-person services geared to the needs of researchers travelling to the public spaces at 395 Wellington Street in Ottawa
Improved access Shorten time between the acquisition of documentary resources and their access - - - Lift restrictions on access to documents in the LAC collection
Reprography Change the way that documentary resources are reproduced and the way that copies are sent in favour of digital format, including requests received under the Access to Information Act Implement a digital-by-default approach and digitize the most frequently requested documents

Focus on digital format as the primary channel for service delivery

- -
LAC visibility - - -

Continue to organize and participate in various exhibitions and initiatives in collaboration with interested communities to help promote LAC’s collection across Canada

Continue to share content on social networks to maximize the number of clients reached and make the collection accessible through a wide variety of channels

Increase the visibility of the collection through public events held in collaboration with other memory institutions

Continue to share content on social networks to maximize the number of clients reached and make the collection discoverable through a wide variety of channels

Appendix G: Performance Measurement Strategy

Outputs

Logic Model Element Indicator Definition Data Source and Frequency Responsible for Collecting

Description (metadata, authorities and standards)

Number of new notices in Amicus

This output is related to basic or enhanced metadata created by LAC or its users.

Amicus database

Monthly

Director General, Published Heritage

Description (metadata, authorities and standards)

Number of descriptions created by others and integrated into LAC corporate systems (by source)

This output is related to basic or enhanced metadata created by LAC or its users.

LAC administrative data

Monthly

Directors General,

- Published Heritage;

- Private Archives;

- Government Records

Services (responses to enquiries, ATIP requests, reference, website visits, etc.)

Number of responses of various types (reference, phone calls, ATIP requests, copyright) provided by type of enquiry (LAC Performance Report)

This indicator may include:

number of letters/emails/faxes replied to;

number of in-person questions answered;

number of telephone calls answered;

number of formal and informal ATIP requests; and

number of copyright-related enquiries received per month

This indicator include a number of outputs, such as number of responses to various enquiries ranging from basic requests for information about LAC holdings to ATIP requests, number of documents or photocopies delivered to clients in paper or digital format, number of items consulted by clients across the various service channels, amount of content made accessible to Canadians through exhibitions and loans or through digitization by LAC or its partners.

LAC administrative data

Various frequencies: Monthly, quarterly and annual

Director General, Public Services

Services (responses to enquiries, ATIP requests, reference, website visits, etc.)

Number of pages/images delivered to clients by type (copies, formal and informal ATIP requests) (LAC DPR)

This indicator include a number of outputs, such as number of responses to various enquiries ranging from basic requests for information about LAC holdings to ATIP requests, number of documents or photocopies delivered to clients in paper or clients across the various service channels, amount of content made accessible to Canadians through exhibitions and loans or through digitization by LAC or its partners.

LAC administrative data, and

Performance Report

Monthly (both)

Director General, Public Services

Services (responses to enquiries, ATIP requests, reference, website visits, etc.)

Traffic on websites of partners such as Canadiana and Ancestry and number of queries on z39.50 protocol

This indicator include a number of outputs, such as number of responses to various enquiries ranging from basic requests for information about LAC holdings to ATIP requests, number of documents or photocopies delivered to clients in paper or digital format, number of items consulted by clients across the various service channels, amount of content made accessible to Canadians through exhibitions and loans or through digitization by LAC or its partners.

LAC administrative data, and

Performance Report

Monthly, and

agreements with partners

Director General, Public Services

Services (responses to enquiries, ATIP requests, reference, website visits, etc.)

Number of unique Internet Protocol (IP) visits on LAC site per month (LAC DPR)

This indicator include a number of outputs, such as number of responses to various enquiries ranging from basic requests for information about LAC holdings to ATIP requests, number of documents or photocopies delivered to clients in paper or digital format, number of items consulted by clients across the various service channels, amount of content made accessible to Canadians through exhibitions and loans or through digitization by LAC or its partners.

Reports

Quarterly

Director General, Public Services

Immediate outcome

Logic Model Element Indicator Definition Data Source and Frequency Responsible for Collecting

Improved access to  Canada’s documentary heritage

Increase in number of titles published online

This indicator is related to descriptions. The indicator can be defined in a number of ways: for example, enabling clients to access documentary heritage directly across one or more access channels; responding to enquiries that will facilitate future access; or providing new content in areas where clients have shown an interest. This indicator is also related to compliance with LAC standards.

LAC administrative data, and reports based on agreements with partners

Quarterly

Director General, Published Heritage

Improved access to  Canada’s documentary heritage

Increase in amount of digitized material available online

This indicator is related to services. The indicator can be defined in a number of ways: for example, enabling clients to access documentary heritage directly across one or more access channels; responding to enquiries that will facilitate future access; or providing new content in areas where clients have shown an interest. This indicator is also related to compliance with LAC standards.

LAC administrative data, and reports based on agreements with partners

Quarterly

Director General, Public Services

Improved access to  Canada’s documentary heritage

Proportion of published material described within the three-month performance standards for the Access Program

This indicator is related to the 2014–15 Performance Measurement Framework.

Expected result: Heritage materials are described in such a way as to facilitate client research.

Statistics from AMICUS

Annually

Director General, Published Heritage

Improved access to  Canada’s documentary heritage

Percentage of ATIP services provided that met service standards (formal requests) This indicator is related to the 2014–15 Performance Measurement Framework.

ATIP software

Quarterly

Director General, Public Services

Improved access to  Canada’s documentary heritage

Percentage of “Purchase of reproductions from photographs” services that met the service standards

Expected result: Clients are able to access the collection through LAC services.

Statistics from the Brechin group report

Quarterly

Director General, Public Services

Improved access to  Canada’s documentary heritage

Percentage of services offered that met service standards. This indicator is related to “Purchase of photocopies” -

Administrative report

Quarterly

Director General, Public Services

Intermediate outcome

Logic Model Element Indicator Definition Data Source and Frequency Responsible for Collecting

Improved use and engagement with Canada’s documentary heritage among the general public

Increase (as a percentage) in visits to LAC’s and partners’ websites

This indicator contributes to improved use and engagement with Canada’s documentary heritage among the general public 

LAC administrative data, and reports based on agreements with LAC partners

Monthly, and Performance Report

Director General, Public Services

Cost-effectiveness and efficiency

Logic Model Element Indicator Definition Data Source and Frequency Responsible for Collecting

-

Most recent cost for creating a description of a published title

2014–15 Performance Measurement Framework.

Statistics from Amicus

Annually

Baseline data from 2014–15

Director General, Published Heritage
-

Ratio of electronic services (Internet Protocols (IP)) to traditional channels (mediated services)

This indicator is related to Public Services output. LAC administrative data Director General, Public Services

Appendix H: Evaluation Questions

Relevance

Ongoing need for program

Do the programs and services under PAA 2.3 continue to be relevant?

To what extent does the program continue to meet the evolving needs of clients?

Alignment with government priorities

To what extent does the PAA 2.3 align with the priorities of LAC and the Government of Canada?

Harmonization with government roles and responsibilities

Are the roles and responsibilities of the various parties involved in implementation of the program clearly defined?

Performance (efficiency, effectiveness and cost-effectiveness)

Attainment of desired outcomes

Has the performance measurement strategy for the program been implemented as planned?

Has data on outputs and outcomes been collected?

How is the performance-related data collected under PAA 2.3 being used?

To what extent does the data collection system provide reliable, consistent and useful performance information?

To what degree have the expected immediate and intermediate outcomes of PAA 2.3 been achieved?

How easy it is to navigate throughout LAC’s Reference Services?

How efficient are the ATIP services provided to clients?

To what extent are clients able to find what they are looking for in LAC’s collections?

Demonstration of efficiency and cost‑effectiveness

Have other ways of achieving the expected outcomes of PAA 2.3 been considered?

How have financial resources been invested amongst program activities and outputs?

Are financial details available (by activity and by output)?

Are program costs in line with the minimum needed to achieve the expected outcomes of PAA 2.3?

Appendix I: List of Acronyms and Abbreviations

ADM
Assistant Deputy Minister
DPEC
Departmental Program Evaluation Committee
DPMPEC
Departmental Performance Measurement and Program Evaluation Committee
DPR
Departmental Performance Report
FTE
Full-time equivalent
GC
Government of Canada
LAC
Library and Archives Canada
PAA
Program Alignment Architecture
RPP
Report on Plans and Priorities
TBS
Treasury Board Secretariat
TD
Toronto-Dominion Bank
TRC
Truth and Reconciliation Commission of Canada

Appendix J: Bibliography

Access Policy Framework, Library and Archives Canada, August 30, 2011.

Assurance report - processes to promote and make documentary heritage available, Library and Archives Canada.

Canada’s Anglo-Celtic Connections, John D. Reid, Wednesday, July 23, 2014.

Communication, LAC begins implementation of new approach to service delivery, Library and Archives Canada, May 9, 2012.

Departmental Performance Report 2011-2012, Library and Archives Canada.

Departmental Performance Report 2012-2013, Library and Archives Canada.

Departmental Performance Report 2013-2014, Library and Archives Canada.

Departmental Performance Report 2014-2015, Library and Archives Canada.

Departmental Performance Report 2015-2016, Library and Archives Canada.

Directive on the Evaluation Function, Treasury Board Secretariat, April 1, 2009.

Evaluation and Acquisition Policy Framework, Library and Archives Canada.

Evaluation Framework for Program Alignment Architecture 2.3, Access to Documentary Heritage, July 16, 2014, prepared by Content Access Branch and Audit and Evaluation Directorate, Library and Archives Canada.

Guide for the Review of Evaluation Reports, Treasury Board Secretariat, 2009.

LAC Blog, from 2011 to 2016, Library and Archives Canada.

LAC Discovery Facility, Library and Archives Canada, CIOB, September 30, 2016.

LAC News Releases, from 2011 to 2016, Library and Archives Canada.

Les cinq priorités de BAC ; Nos quatre engagements, Library and Archives Canada.

Management Board (MB) Meeting / Rencontre du Conseil de direction (CD), Record of Decisions,
February 9, 2016, Library and Archives Canada.

Policy on Copyright Management, Library and Archives Canada, May 18, 2015.

Policy on Making Holdings Available, Library and Archives Canada, April 2, 2013, and June 9, 2014.

Policy on Making Holdings Discoverable, Library and Archives Canada, December 9, 2013.

Policy on Evaluation, Treasury Board Secretariat, April 1, 2009.

Presentation to Johanna Smith, DG Public Services, Review of Access Methods to LAC’s Collection, Summary of Results, April 21, 2016.

Priorities and Strategic Planning Support, submitted by Nanos to Library and Archives Canada, December 2015 (Submission 2015-697C).

2015-2016 Annual Report, Library and Archives Canada.

Report of the Auditor General of Canada, Chapter 7—Documentary Heritage of the Government of Canada – Library and Archives Canada. Fall 2014. Office of the Auditor General of Canada.

Report on Plans and Priorities, 2011-2012, Library and Archives Canada.

Report on Plans and Priorities, 2012-2013, Library and Archives Canada.

Report on Plans and Priorities, 2013-2014, Library and Archives Canada.

Report on Plans and Priorities, 2014-2015, Library and Archives Canada.

Report on Plans and Priorities, 2015-2016, Library and Archives Canada.

Road to 2017, Library and Archives Canada Contribution, June 25, 2014.

Speech from the Throne, December 2015.

Speeches of the Librarian and Archivist of Canada, from 2011 to 2016, Library and Archives Canada.

Standard on Evaluation for the Government of Canada, Treasury Board Secretariat, April 1, 2009.

Terms of Reference, Evaluation of Access to Documentary Heritage (PAA 2.3), Program Evaluation, Monitoring and Audit Liaison Directorate, January 2016.

Users give Library and Archives Canada low marks for public access to collection, Don Butler, Ottawa Citizen, June 21, 2016.

W.B. Coltman Report, Transcription, September 30, 2016.

Date modified: