Outsourcing digital data storage

Storing Commonwealth records in data centres, digital repositories and the cloud

Outsourcing digital data storage can relieve pressure on storage capacity and ICT resources while reducing costs. However, there are potential risks to the viability, access and use of the records. It may also be difficult to ensure that all copies of digital records are accounted for when they are destroyed or removed from storage.

Many of the records management risks can be mitigated once they are recognised. One of the strategies to mitigate risks is to perform due diligence; ie take care when evaluating a provider before entering into a contract. A records management risk assessment template is provided that can be used as a starting point to evaluate digital storage options.

Records management risk assessment template (pdf, 76kb) (Word, 1.1mb)

The Records Issues for Outsourcing including General Disposal Authority 25 (pdf, 111kb) authorises Australian Government agencies to transfer custody of Commonwealth records to contractors and sets out the terms and conditions that apply.

National Archives general advice on outsourcing.

Some of the main records management risks are outlined below.

Regulatory requirements may be breached

A number of laws affect how Australian Government agencies create and manage their records and information. Some, such as the Privacy Act 1988, the Archives Act 1983, the Australia Information Commissioner Act 2010 and the Freedom of Information Act 1982, apply to most Australian Government agencies, while other laws are agency specific. Agency-specific legislation can cover diverse requirements. For example, it may require certain information to be created, determine the format in which it is to be kept, how or where it is to be captured, and how and to whom it may be disclosed.

Agencies must ensure that their choice of offsite storage provider does not put them in breach of any of the legislation to which they are subject.

Outsourcing storage does not lessen an agency's obligation to ensure that its records are created, managed, made accessible and disposed of accountably. Management of the information by the storage provider must also be accountable and in accordance with legal requirements.

To reduce the risk of breaching regulatory requirements:

• consider what information is to be relocated to offsite storage and the legislation that applies to that information
• investigate any legislative impediments to offsite storage location and control of the information
• ensure adequate management and control of the records stored offsite, including those records created by the storage provider
• ensure the storage provider understands its responsibilities in relation to the agency's information.

The storage location may be unknown, with legal consequences

It may be difficult to identify precisely where an agency's information is stored, particularly in the case of cloud storage. The location of the cloud’s server may not be stated in the terms of service of the contract. This can be further complicated if a storage provider changes the location of records without notifying the agency, or uses a subcontractor. Additional problems can occur if data is stored in multiple locations at the same time.

Risks arise if data is stored in a jurisdiction that does not maintain appropriate standards or is not legislatively comparable to that of the agency. For example, some jurisdictions may have the power to demand access to all information, including classified information, stored in that jurisdiction. Different jurisdictions may have different privacy laws that apply to information stored within the jurisdiction, even if the information did not originate there. Storing information in such a jurisdiction may be a breach of legislation to which the agency is subject.

To reduce the risks associated with storing information in unknown jurisdictions:

• give careful consideration to the nature of data that is stored in other jurisdictional regions
• ensure contracts specify where records are stored, for example, the location of servers and their main operations
• determine if the storage provider uses subcontractors, and if so, where the operations of the subcontractor are located
• determine whether information can be stored legally in jurisdictions outside the Commonwealth of Australia or if there are restrictions relating to particular jurisdictions
• consider whether the agency may unintentionally become subject to legislative and regulatory requirements of other jurisdictions and how that may affect the security of the data
• ensure the contract provides that:
  • information is stored in locations only specified within the contract (that is, locations acceptable to the agency's information needs)
  • information stored outside the jurisdiction of the Commonwealth of Australia is managed in accordance with Commonwealth law, as well as with the law of the non-Commonwealth jurisdiction. First and foremost, the laws of the Commonwealth of Australia must be adhered to.

Unauthorised access to, or use of, information

The consequences of information being accessed without authorisation can be very damaging, particularly if the information is sensitive or personal. This may occur if a storage provider accidentally or deliberately discloses information to parties outside those specified in the contract.

In addition, new information is created when stored information is accessed or updated, or when changes are made to equipment in the storage facility. This transactional and relationship information may include details of the person(s) who has accessed the information, including location (if accessing information through internet portals or intranets), name or identification, and activities carried out. This information also needs to be protected from inappropriate access.

To reduce the risk of unauthorised access to, or use of, information:

• ensure that the contract outlines conditions for access to, and use of, information, including that:
  • information cannot be used for any purpose other than that outlined in the contract
  • transactional and relationship information created through the use of the information is the property of the agency, and may only be accessed and used by people nominated by the agency
• ensure the storage provider maintains system logs and audits, and has the capability to detect unauthorised access
• ensure the storage provider understands requirements for authentication and authorisation.

Privacy may be compromised

Consider whether records that contain private or sensitive information should be stored offsite, or if it is more appropriate for them to remain under agency control. If the information is stored offsite, the storage provider must have adequate access controls and security measures, and be able to provide an appropriate level of security for personally sensitive material.

To reduce the risk of privacy being compromised:

• know where the information is physically stored and who has access to the storage area
• assess storage providers’ ability to protect the sensitivity of the information
• consider retaining sensitive information under the control of the agency on agency servers.

Access to the information may be lost

Access to information may be lost as a result of a disaster, including fire or flood, if the provider has not performed appropriate backup. Cloud computing services may be affected by internet service disruption.

To reduce the risk of loss of access to information:

• ensure that the contract outlines access requirements, including that:
  • the provider can retrieve and make available sufficient copies of the original data ('sufficient' should be defined in the contract)
  • there is an agreed period within which access requests are satisfied
• ensure proper backups are created and maintained by the provider, and that disaster recovery and business continuity plans are in place and are actively maintained
• ensure agency access requirements can be met by the provider.

The storage provider may cease business

Timely access to information may be lost if the storage provider, whether a data centre, digital repository or cloud computing service, goes out of business or is taken over by another company.

When a storage provider goes out of business, agencies may not be able to access their information and could lose control of vital business records.

If new owners of the storage facility do not honour previous arrangements, agencies may not know who has access to their information.

To reduce the risk arising from a storage provider ceasing business:

• perform due diligence when selecting a provider, i.e. undertake appropriate checks
• maintain all contracts, agreements, licences and transfer documentation throughout the period of the contract
• ensure the storage provider has sound policies and procedures that reflect, and are appropriate for, the type of information stored
• ensure that data can be easily migrated to other providers without impediment if the provider has gone out of business or because an agency wishes to change providers at the end of a contract. Are there costs involved, will the information be exported in an open format, and how long will it take before you can access your data again?

Information readability and usability may be compromised

Over time, a storage provider may need to upgrade its hardware or software. There is a risk that when records are returned to an agency, its software may no longer be compatible with that used in the provider’s facility. The information would therefore be unreadable and unusable.

The use of open formats supports readability of information over time.

To reduce the risk of data becoming unreadable or unusable:

• specify in the contract the format in which the information should be returned once the contract has ended
• specify that formats used in storage are to be based on open formats
• require notice of any software or hardware changes, including migration of information, that may be undertaken by the storage provider
• ensure proper processes are followed when information is migrated, including keeping a copy of the information in the original format until the migration is proved successful.

The evidential value of the information may be diminished

Government records must be authentic and reliable. The evidential value of government records can diminish if their authenticity cannot be proven.

Risks to evidential value when using a storage provider may arise from unsuccessful migration undertaken by the provider, the information not being properly secured, unknown access or access that is not logged, as well as backups not being routinely performed.

To reduce the risk of evidential damage for the information:

• ensure the contract provides that:

  • the storage provider is compliant with information security requirements in the Australian Government Protective Security Policy Framework and the Australian Government Information Security Manual
  • the storage provider's information security policies and procedures are documented and accessible
  • instances of data corruption or loss, and steps taken to repair or negate further loss, are reported to the agency in a timely manner
  • migration or refreshment is undertaken with the agency's knowledge and acceptance, and is completed with agreed precautions against data loss
• assign appropriate security classifications or protective markings to their information
• choose a storage provider that can provide security, including physical security, at a level suitable for the information being stored.

Metadata may not be appropriately maintained

Metadata is the means by which digital information is confirmed as complete and authentic, and by which information is made findable and usable. Metadata is information that describes an object’s structure, context, content, and management through time. Mismanaged metadata may result in information that is unusable because it is difficult to find the information, understand its purpose, or be sure of its integrity.

To reduce the risk of metadata not being appropriately maintained:

• determine requirements for metadata, and assign metadata to records in accordance with those requirements before they are located with the storage provider (preferably at point of creation)
• ensure that the metadata assigned is sufficient and reliable enough to confirm the authenticity of information while being stored offsite
• ensure that the contract includes minimum metadata requirements for process management of the information while being stored offsite.

Destruction or removal of records

Disposal is the means by which Australian Government information, after it has satisfied minimum retention requirements, can be transferred to the National Archives of Australia, or destroyed or disposed of in other ways. Disposal of Australian Government records is authorised by the Archives through records authorities.

When storing information offsite, it is vital that any disposal actions, including transfer or destruction, are transparent and appropriately carried out. Factors to be considered, regardless of who is conducting the disposal, include the removal of copies created during the transfer process and ensuring that removal or destruction of information is in accordance with the appropriate records authority.

The Australian Government Protective Security Policy Framework and the Australian Government Information Security Manual set out requirements for rendering backups or copies unreadable and irretrievable, with requirements becoming more stringent as the security classification of the information increases. There are risks of information leaks, confusion about the validity of records, possible discovery costs and embarrassment to the agency if even unclassified records are not permanently removed from the storage provider's systems. Cloud storage poses a particular problem because, by design, it creates multiple, geographically distributed copies to maintain availability. It may not be possible to verify that records have been destroyed and agencies must factor this risk into their use of the cloud.

When managing the disposal of records stored offsite consider:

• if disposal is to be carried out by the storage provider, ensure the contract stipulates what disposal action is to take place and when, which disposal processes are permitted, and requirements for recording disposal actions (for more information on the recording of disposal actions, see Administrative Functions Disposal Authority (pdf, 5787kb) – Information Management Control [class 1940]).
• if records are to be returned to the agency, ensure no other version remains with the storage provider
• if records are to be destroyed by the storage provider, ensure no other version remains with the provider
• records that have been classed in a records authority as 'retain as national archives' should be transferred to the National Archives of Australia at the appropriate time.

Due diligence assessment and service level agreements

Australian Government agencies should undertake appropriate assessment and checks when entering into any contract for an outsourced service. When storing digital information, questions that should be asked include:

• where will the records and information be located?
• can the provider meet the records management requirements of the Australian Government?
• is the storage provider aware of the laws and regulations surrounding Australian Government information, such as the Privacy Act 1988?
• will records be returned in a timely manner with no copy of the original records retained once the contract has ended?
• who has access to the records and information, storage location and systems?
• what backup plans and processes are in place?
• is the provider certified in, or audited against, any standards?

Following this assessment, agencies can specify what should be included in a service level agreement. Some providers may have standard service level agreements, however the terms need to be acceptable not only to the provider but also to the agency.

A service level agreement is used to outline specific metrics the storage arrangements can be measured against to ensure that an agency's information is managed appropriately. As data storage is increasingly being offered as a service rather than a product specified in an end-user agreement, it is important that agencies use service level agreements to specify metrics. Doing so ensures that the information stored is protected, costs involved beyond the contract fees and charges are agreed, and what happens during server downtime is outlined. The agency may need to seek legal and accounting expertise to assist in specifying and assessing requirements in a service level agreement.

Storage option definitions

What is a data centre?
A data centre houses computer systems and associated components such as servers, networks and data storage systems. Data centres are purpose built, permanent, shared enterprise facilities that can contain a full range of ICT equipment for agency use.

What is a digital repository?
A digital repository is used to retain and manage digital information, and aims to ensure the usability of stored digital objects over time. The term 'digital repository' is often interchangeable with 'institutional repository' and 'digital archive'. Common types of digital repositories include national libraries and archives, subject-based repositories or scientific-data archives.

What is cloud computing?
Cloud computing can be described as information technology resources delivered as a service through a network. These services may include procurement of software, platforms, infrastructure, or a combination of these. Outsourced cloud storage services may involve sharing, creating or storing information on remote servers accessed through the internet.