Investigating Leaked Artifacts via API

Investigations resolve a leaked artifact to a particular marked copy and thus a recipient. Here is the intended workflow for the investigation API (called “detect” as a verb). Your organization identifies a potential leak of data. You gather a set of "artifacts" that are leaked. These are supposed copies, renderings, or photographs of marked content. For each artifact, you should identify an original document (or a set of possible originals) from where it could be derived. Additionally, helpful hints such as page numbers can be optionally provided. Investigation involves passing an artifact, a plausible set of originals, and a set of hints to a detect endpoint.

After running a leak investigation, you'll be presented with a leak report. The leak report gives a high level match result,  including details about the results.

Match Confidence

Each result comes with an associated chance of error. The chance of error measures EchoMark’s confidence in the accuracy of the results. It uses a combination of statistical analysis and extensive offline simulation to ensure trust in the results.

The chance of error is a measure of statistical significance called a p-value, which is a statistical tool to evaluate whether differences between noisy measurements are attributable to random chance. EchoMark measures the spatial correlation of marks in a leaked artifact against the marks we introduced into the email. We use a statistical test called the chi-squared test to determine whether the difference in correlation between the top-ranked and second-ranked copies is statistically significant. The confidence score is a defense against distortions in the artifact, i.e., whether the marks that we were unable to detect could change the results of the investigation.

The API will yield a “Possible Match” when an artifact is uploaded for the leak investigation and our detection system finds that it matches a marked copy when the chance of error is low. Sometimes our computer vision detection system is not confident in determining a clear match and will yield a “No Match”. 

The meaning of a result

It is important to note that while an investigation result can be used to confidently determine that the likely source of the leaked content was from the copy originally sent to the recipient indicated in the results, EchoMark cannot determine the ultimate path it took to getting leaked. It's possible that the leak was not the fault of that recipient holder. Some non-exhaustive possible explanations include (in non-ranked order):

• The recipient leaked the content directly
• Someone else who had permissible or compromised access to the account owner's computer(s) or account leaked the content.
• A copy of the marked content was shared with someone else who ultimately leaked the content.

EchoMark 'detect' results represent one confident piece of evidence in a broad forensic analysis of your potential leaks. A full investigation should be run, including the gathering of additional forensics information, to help validate or rule out various possible explanations for your result.

Learn more about match results.

For Marked Documents 

/detect/ endpoint

Detect requests are also passed as multipart forms. It expects a single artifact file named "artifact". It can take any amount of originals. The API server uses the glob "original*" to determine whether a file is an original. Each original is hashed and checked against our internal content store to determine if it has been marked before. Detect will ignore any original that hasn't been marked by our system.

Content-Type: multipart/form-data; boundary=----FormBoundary7MA4YWxkTrZu0gW 
------FormBoundary7MA4YWxkTrZu0gW 
Content-Disposition: form-data; name="artifact" 
Content-Type: image/png 
 
[Binary content of png image]

------FormBoundary7MA4YWxkTrZu0gW 
Content-Disposition: form-data; name="original_1" 
Content-Type: application/pdf 
 
[Binary content of the pdf document]

------FormBoundary7MA4YWxkTrZu0gW 
Content-Disposition: form-data; name="original_2" 
Content-Type: application/pdf 
 
[Binary content of the pdf document] 
------FormBoundary7MA4YWxkTrZu0gW—j

Content-Disposition: form-data; name="detect_options" 
Content-Type: application/json

{

"investigation_hints": {

    "page_idx": 0,

    "orientation_idx": 1

  },

"background": "True"

}

 
------FormBoundary7MA4YWxkTrZu0gW--j

Detect optionally takes in two options: investigation hints, background. 

Investigation hints provide hints to the investigation engine to speed up the analysis time.

If page_idx is provided, it is assumed that all originals provided are page-aligned. If not, leave this blank. Background determines whether the detect task runs synchronously or asynchronously. More on this a bit later.

Detect returns a response that looks like this.

Scored copies are tuples of names and scores. The confidence score measures whether the highest ranked (highest absolute score) is a statistical winner or not.

To make this clearer, here is an example /detect payload.

{ 
  "scores": [ 
    { 
      "label": "secure-Alice-Bar.pdf", 
      "score": 0.95 
    },
    { 
      "label": "secure-Bob-Car.pdf", 
      "score": 0.65 
    } 
  ], 
  "confidence": "possible_match" 
}

For Background Detect

Detect tasks can take a substantial amount of time.  The background option gives users the option to trigger a detect asynchronously. When false the result is synchronously returned to the requestor, when true the request returns synchronously with an analysis_id to query at a later time. Synchronous detect is useful for debugging or assessing that the system is working correctly. When background detect is set to true, detect returns almost immediately with an analysis id:

{ 
  “analysis_id” : ”b94d27b99a1139c96512636b6” 
}

This analysis id can be passed into another API endpoint to query the report.

JSON Mode

By default, your container will accept multipart form requests. But, you can use json requests if pass in a URL argument ?json=True to the detect endpoint. Detect requests can similarly be sent in JSON mode. The structure is slightly different than that of above with the multipart forms. 

Here is the basic schema:

Content-Type: application/json
{
“artifact": <base64 encoded string>
“content_type”: <the content type for the artifact above>
“investigation_hints”: <investigation hints following the structure above>
“originals”: [ <a list of original structs>
                 {
                    “original”: <base64 encoded string of original document>
                    “marked_copies”: <an optional list of marking schemas if storage mode is set to none> 
                 } 
             ]
“background”: <whether to execute in background mode>
}

Note: This structure is exactly the same as EchoMark’s V1 JSON structure with the only change being additional options of background execution and orientation correction.

Note: Algorithm versioning and background execution with state is not supported in JSON mode.

/reportjson/<analysis id>

The developer can run a get request against a particular analysis id, for example, “GET /reportjson/b94d27b99a1139c96512636b6”. The end point will return responses in the following schema.

The user can poll the endpoint until the status returns complete. The result of the report is in the same structure as one expects with synchronous detect.

Example Detect Request:

As described above, to initiate a detect request, you first need to have a few things.

1. An artifact (e.g. an image or PDF) which represents the leaked (marked) copy.
2. A list of all originals from which the artifact could have originated.
3. The marking schema associated with all markings ever created from those originals.

We provide an example detect flow in Python below. First, we provide an example of marking a few documents, in order to clearly illuminate how the marking schemas from step 3  are used in the detect flow.

Example of marking several PDFs

original_filename = os.path.join(working_dir, "important.pdf")
base, ext = os.path.splitext(original_filename)

# Mark the PDF for each recipient.
names = ["John Lennon", "Paul McCartney", "George Harrison", "Ringo Starr"]
marked_copy_schema_names = {}
for name in names:
    # Set up mark options and a unique marked filename for each recipient.
    mark_options = default_mark_options.copy()
    mark_options["recipient"] = name
    mark_options["recipient_name"] = name
    marked_filename = os.path.join(working_dir, f"{base}.marked.{name}{ext}")

    # Open the PDF and mark it.
    with open(original_filename, "rb") as fileobj:
        files = {
            "mark_options": (None, json.dumps(mark_options), "application/json"),
            "original": (
                os.path.basename(original_filename),
                fileobj,
                "application/pdf",
            ),
        }
        response = requests.post(
            url=f"{API_URL}/document/mark/pdf",
            files=files,
            headers=create_headers(),
            timeout=TIMEOUT_SECONDS,
        )

    # Save the marked output and retain the schema filenames for use in detect example below.
    # marked_output.keys() == ["marked_copy", "marking_schema", "marked_copy_schema_name"]
    marked_output = parse_response_and_save_marked_copy(response, marked_filename)
    marked_copy_schema_names[(original_filename, name)] = marked_output["marked_copy_schema_name"]

The mark example uses the helper function parse_response_and_save_marked_copy. While not particularly illuminating for the purposes of using the mark and detect APIs, it is still useful to have in place, so we copy below for completeness.

def parse_response_and_save_marked_copy(response: requests.Response, output_filename: str):
    """
    Save the marked output to a file, along with the marking schema.

    Args:
        response: requests.Response - API response containing marked copy and marking schema
        output_filename: str - name of the output file to save the marked copy
    Output:
        marked_copy: BinaryIO - binary marked copy
        marking_schema: str - marking schema
        marked_copy_schema_name: str - name of the marked copy schema file
    """
    if response.status_code != 200:
        raise ValueError(f"Request failed with status code {response.status_code}: {response.text}")

    # Extract the marked copy and marking schema from the response
    marked_copy = None
    marking_schema = None
    decoder = MultipartDecoder.from_response(response)
    for part in decoder.parts:
        content_disposition = part.headers.get(b"Content-Disposition", b"").decode()
        if 'name="markingSchema"' in content_disposition:
            marking_schema = part.text
        elif 'name="markedCopy"' in content_disposition:
            marked_copy = part.content  # Binary data

    if marked_copy is None:
        raise ValueError("Marked output not found in response")
    if marking_schema is None:
        raise ValueError("Marking schema not found in response")

    # Save the marked copy and marking schema to files
    marked_copy_schema_name = f"{output_filename}.marking_schema"
    with open(marked_copy_schema_name, "w", encoding="utf8") as schema_file:
        schema_file.write(marking_schema)
    with open(output_filename, "wb") as marked_file:
        marked_file.write(marked_copy)

    return {
        "marked_copy": marked_copy,
        "marking_schema": marking_schema,
        "marked_copy_schema_name": marked_copy_schema_name,
    }

Example detect flow

Notice that the example below is only using a single original in originals_to_process, but the example is written to allow for the possibility of multiple originals. The majority of the work in the example is setting up the FILES dictionary which is passed into the detect request. Creation of the FILES dictionary is where we make use of the marking schema stored from the mark example above.

# Construct detect options
hints = {}
page_idx = 0
hints = {"pageIdx": page_idx}
detect_options = {
    "investigationHints": hints,
    "background": False,
}

# Artificially create a leaked document, called an artifact.
leaker_name = names[0]
artifact = os.path.join(working_dir, f"important.marked.{leaker_name}.pdf")
artifact_content_type = "image/jpeg"
if artifact.endswith(".pdf"):
    artifact_content_type = "application/pdf"
if artifact.endswith(".png"):
    artifact_content_type = "image/png"
if artifact.endswith(".txt"):
    artifact_content_type = "text/plain"
if artifact.endswith(".html"):
    artifact_content_type = "text/html"

# FILES holds all files to be sent to the API
FILES = {}
FILES["detect_options"] = (None, json.dumps(detect_options), "application/json")

# Artifact file to be sent to the API
with open(artifact, "rb") as artifact_handle:
    artifact_data = artifact_handle.read()
    FILES["artifact"] = (
        os.path.basename(artifact),
        artifact_data,
        artifact_content_type,
    )

# Construct originals. Note that we are only assuming one original for this
# artifact, but in practice there could be many.
originals_to_process = [original_filename]
originals = []
for idx, original in enumerate(originals_to_process):
    with open(original, "rb") as original_handle:
        original_data = original_handle.read()
        FILES[f"original_{idx}"] = (
            "none.bin",
            original_data,
            "application/octet-stream",
        )

    # Pass the marked copies for each original.
    marked_copies = []
    marking_schema_filenames = [marked_copy_schema_names[(original, name)] for name in names]
    for marking_schema_filename in marking_schema_filenames:
        with open(marking_schema_filename, "r", encoding="utf8") as marking_schema_handle:
            marking_schema = marking_schema_handle.read()
            marked_copies.append(
                {
                    "label": marking_schema_filename,
                    "markingSchema": marking_schema,
                }
            )
    FILES[f"original_{idx}/marked_copies"] = (
        None,
        json.dumps(marked_copies),
        "application/json",
    )

# Create the detect request
response = requests.post(
    url=f"{API_URL}/static/detect",
    files=FILES,
    headers=create_headers(),
    timeout=TIMEOUT_SECONDS,
)