Commonwealth procurements are complex, require judgement, and are subject to a range of accountability, legislative, policy and probity obligations. AI is rapidly becoming embedded in day-to-day work activities and practices. AI tools have the potential to make procurement processes more efficient. However, using AI tools in procurement does not dilute or abrogate the responsibility of government officials to comply with all applicable obligations. The use of AI in procurements is likely to attract greater scrutiny so the potential efficiency benefits do not justify introducing AI into a procurement without careful planning to ensure compliance.

This article outlines 6 probity and process tips to help ensure your procurement remains fair and defensible from a probity perspective if AI is intended to be used.

1. Planning 

The use of AI in a procurement must be carefully planned. AI must never be used unless its proposed use has been expressly contemplated and appropriately authorised. If AI was not contemplated and approved at the planning stage, procurement personnel must not use it merely because it is available or may expedite the process.

Agencies could consider conducting a pilot or test case before rolling use of AI in procurements out more broadly. 

In relation to each procurement, consider the following:

• What specific aspects of the procurement will AI be used for (for example, preparation of the approach to market (ATM) documents, undertaking evaluation, etc)?
• All procurement personnel who will use the AI tool should understand its risks and limitations. Is training in use of the relevant AI platform needed for those procurement personnel?
• Should there be an AI ‘super user’ available to oversee use of the AI platform in the procurement, monitor its use, and answer any questions?
• Are any specific approvals needed for the use of AI in the procurement?
• Any decision to use AI in a procurement should be documented in detail in the procurement plan and relevant procurement governance documents, including the probity plan and evaluation plan.

2. Transparency

Paragraph 7.1 of the Commonwealth Procurement Rules (CPRs) states that the Australian Government is committed to ensuring accountability and transparency in its procurement activities. The requirement to treat tenderers fairly is also a principle underpinning ethics and probity in Australian Government procurements (see Ethics and Probity in Procurement | Department of Finance).

As such, the intended use of AI in a procurement should be disclosed in the ATM documentation. The ATM documentation should explain:

• the fact that the agency intends using AI in the procurement
• the AI platform or tool that will be used. This can help reassure tenderers that tender-related information will not be available more generally via a consumer or public AI platform
• what procurement activities AI will be used for. If AI will be used to support tender evaluation, consider explaining the limits or parameters of that use. For example, consider informing potential tenderers that AI will only be used to support the analysis of tenders but that agency procurement personnel will be ultimately responsible and accountable for reading and evaluating tenders, awarding scores or ratings, determining value for money, and all decision-making.

3. Confidentiality and data protection

Information entered into AI systems can be processed or stored in ways that are not visible to the user. This raises obvious concerns about confidentiality.

The CPRs include a mandatory obligation to treat all tenders as confidential before and after the award of a contract (paragraph 7.23). Tenders and evaluation-related materials commonly contain commercially sensitive, proprietary, personal or security-classified information. 

For these reasons:

• it is critical to only use AI platforms or tools that are approved by your agency as these will have undergone rigorous privacy and information security assessments. Do not enter tenders or evaluation material into consumer, public or otherwise unapproved AI products. 
• if AI is used in evaluation, all AI-generated outputs must be treated as confidential and stored/used in the same way as human-generated evaluation information. This includes implementing information security measures so that only procurement personnel with a ‘need to know’ can access the AI-generated outputs.
• all storage, use and access arrangements documented in the probity plan and evaluation plan should expressly identify any additional requirements implemented for AI-generated outputs. This includes detailing any additional roles or responsibilities established for the procurement due to the use of AI.

4. Human judgement and quality assurance 

Procurement officials using an AI tool must understand its limitations. 

For example, AI outputs often look complete, accurate and authoritative even when that is not the case. That creates a risk of misplaced confidence. 

Reliance on AI-generated outputs can also result in officials effectively abrogating their responsibility to comply with applicable legal, probity and policy requirements. The role that AI plays in the procurement should not displace the role of humans on matters requiring judgement and compliance.

This means that:

• The specific uses of an AI tool should be documented in detail (including in the procurement plan, the evaluation plan, and the probity plan) together with the parameters or limitations of the use of AI, including express documentation of the roles of procurement personnel. For example, in evaluation, the AI tool could be used to produce an initial overview of tenders (such as key strengths and weaknesses against the evaluation criteria) but the evaluation plan should expressly state that evaluation personnel must:
  • read all tenders themselves
  • use the AI-generated outputs as a guide (only) but that they should check the accuracy (or otherwise) of those outputs and must evaluate the tenders against each evaluation criterion themselves
  • undertake scoring and rating of tenders themselves (ie AI should not be used)
  • undertake the value for money assessment themselves (ie AI should not be used).
     
• If AI is used to help prepare ATM documents or with administrative tasks, the AI-generated outputs must be independently checked by procurement personnel against the procurement requirement and approved procurement plan, the CPRs, applicable procurement-connected policies, source material and the approved evaluation methodology.

5. Process compliance and fairness 

The procurement process and evaluation methodology should be structured to ensure compliance with all applicable legal, policy and probity obligations. In this context, it is important to remember that an AI platform is simply an efficiency tool.

AI must not be used in a way that alters the approved evaluation process, introduces new evaluation criteria, or results in inequitable treatment of tenderers. 

If an AI tool is to be used to support evaluation, the evaluation plan should include details of all AI-related matters, potentially in a specific annex, including:

• the risks and limitations of the AI tool
• what the AI tool can and cannot be used for
• the roles that human evaluation personnel will retain exclusively (that is, without resorting to AI use)
• the quality assurance mechanisms that will be implemented
• evidence that the risks of AI use in evaluation have been addressed
• who has AI-related roles in relation to the procurement, and the nature of their responsibilities.
• disclosure requirements regarding use of AI. For example, if evaluation personnel use AI to help write the evaluation report, the evaluation plan should require the report to state that AI was used for this purpose. The evaluation report should also detail how the AI tool was used in evaluation more generally, to demonstrate compliance with the evaluation plan. 

Any procurement risk assessment should include the risks and mitigation measures associated with use of the AI tool, together with ownership of the risks and mitigations.

• The approved delegate must remain the actual decision-maker. AI must not displace independent judgement, and a decision must not be adopted merely because an AI tool has been used in evaluation or to help write the evaluation report or decision brief.

6. Auditability and record-keeping 

The CPRs include a mandatory obligation requiring officials to maintain for the procurement a level of documentation consistent with its scale, scope and risk (paragraph 7.2).

In relation to AI use, the procurement record should provide information detailing:

• the reasons an AI tool was used, which AI tool was used, and the purposes for which AI was used throughout the procurement
• the processes followed in relation to the use of the AI tool (as detailed in the procurement plan, the probity plan and the evaluation plan)
• relevant approvals obtained for use of the AI tool
• the actual use of AI (to demonstrate that this complied with the approved processes) including audit logs or records capturing the detailed prompts that were used.

For transparency, where AI was used to help prepare documents (such as the ATM documentation, plans, reports, briefs) those documents should include a statement to the effect that AI was used in their preparation.

• All prompts and AI-generated outputs must be retained in accordance with the Archives Act 1983 (Cth) (CPRs, paragraph 7.5). 

The bottom line

AI is a tool only, not a procurement officer. It must never be used in a procurement as an undisclosed or unplanned shortcut. Its use must be planned, properly authorised, clearly documented and transparent, including by communicating its use in the ATM documentation. Responsibility for the integrity, fairness, compliance and defensibility of the procurement always remains with the relevant procurement personnel.

Need guidance on AI in your next procurement?

As government agencies increasingly look to integrate AI into procurement processes, the stakes and the scrutiny will only grow. Getting the probity settings right from the outset is critical to ensuring your procurement remains fair, transparent and defensible.

Maddocks' probity team advises Commonwealth and state government agencies on the full spectrum of procurement, probity and governance challenges; from planning and evaluation through to audit and review. Our team has deep experience helping agencies navigate emerging issues at the intersection of technology, accountability and compliance, including the responsible use of AI.