Test agreement customizations

Testing enables you to measure the accuracy of Agreement Manager’s AI extractions before deploying your configuration to production. By comparing AI-extracted values against known correct values, you can identify fields or agreement types that need improvement and iterate on your configuration until the extraction accuracy meets your requirements.

The testing process has four steps:

1. Set up test agreements
2. Generate the testing template file
3. Update the testing template file with ground truth values
4. Run the test to compare AI-extracted values against those ground truth values

For a walkthrough of the testing process, see this video:

Prerequisites and test agreement requirements

Before you begin testing, make sure that:

• You are authenticated to the target environment. See Authenticate for details.
• You have uploaded your Agreement Manager package to configure the agreement types and fields you want to test.

For the test agreements themselves, follow these requirements and recommendations:

• Add your test agreements to the following directory: {workspaceName}/{projectName}/agreement-manager/files/test/

• Test agreements must be different from the training agreements used to train AI extraction. They should represent realistic agreements expected in production.

• Use a maximum of 400 agreements per test run. This limit is based on guidance from the Docusign Applied Science team. A diverse set of 400 agreements produces accuracy measurements that are statistically equivalent to testing across hundreds of millions of agreements.

• Include a diverse mix of agreement types and field values to ensure accuracy measurements reflect real-world conditions.

• The following file formats are supported:

  • PDF
  • Word: DOC, DOCX, RTF
  • Excel: XLS, XLSX, XLSB
  • PowerPoint: PPT, PPTX, PPSX
  • HTML
  • Images: JPG, PNG, TIF, TIFF

Sample agreements

Docusign provides sample agreements that you can use as training or test documents while you are learning how the Docusign CLI works. Because these agreements are generic, you should not use them to prepare Agreement Manager customizations for the production environment. When working toward a production deployment, use training and test documents that are representative of your organization’s or customer’s agreements.

Step 1. Set up test agreements

The first step in the testing process is uploading test agreements to Agreement Manager. After the upload, Docusign extracts agreement values for standard fields and AI-enabled custom fields. These extracted values will be included in the testing template file if you prefill values when generating the template.

See Prerequisites and test agreement requirements for information about preparing test agreements for upload.

To upload test agreements, run this command:

$ ds agm test --setup

When you run the test --setup command, the CLI checks the {workspaceName}/{projectName}/agreement-manager/files/test/ directory and detects whether any of the files have been uploaded previously. The CLI then displays these prompts:

• Found N test agreements that have not been ingested into Agreement Manager yet. Do you want to ingest N test agreements into Agreement Manager? Respond with y to upload agreements that have not been uploaded before.

  Note: When the CLI generates the testing template file, it includes all files in the test directory, regardless of whether they have been uploaded. If you try to run a test when the testing template file contains files that have not been uploaded, the test --run command will return an error.

• Found N agreements in Agreement Manager that need AI reprocessing. Do you want to reprocess N agreements to ensure AI extracted values are accurate? Respond with y to reprocess previously uploaded test agreements. Docusign recommends reprocessing test agreements if you’ve made any of the following updates to agreement-manager-manifest.json since the agreements were uploaded:

  • Added new custom fields
  • Added standard or custom fields to agreement types
  • Made changes to improve AI extraction accuracy

Step 2. Generate the testing template file

The testing template file, testing.csv, defines the structure of your test run. It contains one row per test agreement and one column for each standard and custom field. The contents of the file are based on the test agreements you uploaded using the ds agm test --setup command, described in Step 1. Set up test agreements.

You populate testing.csv with the expected (ground truth) values that the AI should extract, and the CLI uses it as the basis for comparison when you run the test.

To generate testing.csv, run the following command:

$ ds agm test --generate-test-template

Additional command options

When you run test --generate-test-template, you can include these additional options:

What happens when the command runs

When you run test --generate-test-template, the CLI generates the testing.csv file to the default location {workspaceName}/{projectName}/agreement-manager/tests/ or to the location specified in the --output option.

If a testing.csv file exists in the output location, the CLI prompts you to confirm whether the existing file should be overwritten. Save a backup of any manually entered ground truth values before overwriting the file.

Prefill the testing template

The --prefill-extractions option instructs the CLI to populate testing.csv with the actual AI-extracted values for all relevant standard and custom fields, rather than leaving the cells blank. This gives you a starting point and can help you identify which values the AI has already extracted correctly.

Example command:

$ ds agm test --generate-test-template --prefill-extractions

This option offers a useful advantage: instead of entering every ground truth value from scratch, you can review the prefilled values, correct any that are wrong, and leave accurate ones unchanged. This can significantly reduce the time required to prepare the testing file.

Use this option with care. If you run the test without reviewing and correcting the prefilled values, the test results will always show 100 percent accuracy for the prefilled cells because the ground truth and the extracted values will be identical. Always review every prefilled cell and update any values that do not match the actual content of the agreement.

Step 3. Update the testing template file with ground truth values

Open testing.csv and enter the expected values for the relevant fields in each test agreement. The values you enter are treated as the ground truth and compared against the AI-extracted values during the test run.

Each row in testing.csv corresponds to one test agreement and includes the agreement name, the agreement type, and columns for all standard and custom fields. You do not need to populate every field for every agreement. Leave a cell blank if the field is not present in that agreement or is not relevant for testing. Blank cells are ignored during the test run.

Keep the following formatting rules in mind when entering values:

• For standard duration fields (such as Liability Cap Duration, Renewal Notice Period, and Renewal Term), enter values in the format # {days/months/years}. Examples: 30 days, 6 months, 3 years.
• DateTime values can be entered in ISO format or MM/DD/YYYY format. For example, if an agreement has an expected start date of 03/28/2026, enter that value in the Start Date column for that row.

Step 4. Run the test

After you have populated testing.csv with ground truth values, run the test with the following command:

$ ds agm test --run

Additional command options

When you run test --run, you can include this additional option:

What happens when the command runs

When you run ds agm test --run, the CLI fetches the AI-extracted values from your test agreements in Agreement Manager, compares them against the ground truth values in testing.csv, and calculates accuracy results. Upon completion, the CLI outputs a summary of field-level accuracy:

The CLI also generates two result files in {workspaceName}/{projectName}/agreement-manager/tests/:

• {current date-time}-field-test-results.csv: Contains a row for every tested field. For each test agreement, this file shows whether the AI-extracted value matched the ground truth value in testing.csv.
• {current date-time}-agreement-type-test-results.csv: Contains a row for every test agreement. For each agreement, this file shows whether the AI-extracted agreement type matched the agreement type in testing.csv.

Improve AI extraction accuracy

If the accuracy results from your test run do not meet your requirements, you can improve them by refining the configuration in your agreement-manager-manifest.json file and retraining the AI. The available improvements depend on the type of extraction.

For custom fields:

• Improve the field’s aiDefinition so the AI has a clearer understanding of what the field represents.
• Add more training examples to the examples section (3–5 examples are recommended).
• Use full paragraphs in the exampleText property to provide richer context for the AI.
• Refine existing examples that may be ambiguous or unrepresentative.

For custom agreement types:

• Improve the agreement type’s aiDefinition to give the AI a clearer understanding of the type.
• Add more diverse training agreements (5–10 documents are recommended).
• Ensure training agreements represent real-world variations of the agreements you expect in production.

Note: Improving the accuracy of standard extractions is not supported.

After making changes to the package to improve accuracy, follow these steps to check the results:

1. Upload the updated package. Changes to the manifest do not take effect until the package is uploaded.
2. Complete the test agreement setup to apply the updated configuration and reprocess test files.
3. If you’ve added test agreements, update the testing template file.
4. Rerun the test.

Frequently asked questions

How many test agreements should I use?

You can use up to 400 agreements per test run. The Applied Science team has determined that a diverse set of 400 agreements produces accuracy results that are statistically equivalent to testing across hundreds of millions of agreements. Prioritize diversity of agreement types and field values over quantity.

What does it mean if a cell in testing.csv is blank after using —prefill-extractions?

A blank cell after using --prefill-extractions typically means the AI has not yet finished processing that agreement or field. This can happen when agreements were recently uploaded. Wait a few minutes and rerun the test --generate-test-template command to allow extraction to complete.

Can I reuse testing.csv across multiple test runs?

Yes. If you rerun test --generate-test-template and a testing.csv file exists in the output location, the CLI prompts you to confirm whether the existing file should be overwritten. Save a backup of any manually entered ground truth values before overwriting the file.

Why does my test run show 100 percent accuracy?

If you used the --prefill-extractions option and did not review and update the prefilled values before running the test, the ground truth and extracted values will be identical, resulting in 100 percent accuracy. Always review every prefilled cell and correct values that do not match the actual agreement content.

Can I improve the accuracy of standard field extractions?

No. Improving the accuracy of standard field extractions is not supported. You can only improve accuracy for custom fields and custom agreement types where customExtractionEnabled is set to true.

Next steps

• Once accuracy meets your requirements, deploy your agreement customizations to a production account.
• Learn about bulk ingesting agreements in production, including supported file types and how duplicate detection works.
• If you deploy customizations to an environment where agreements have already been ingested, initiate a reprocess to apply the customizations to existing agreements.