Best Practice for high-res/large file sync (TIFF, ProRes > 2GB) using the V4 API

Hello Charlie/Team,

Following our last successful exchange on token authentication, I’ve managed to stabilize a local sync script (NAS → Frame io) for our photo and video assets.

The sync works perfectly for small and medium files (e.g., NEF files around 50MB). However, I’m encountering SignatureDoesNotMatch (403) errors, consistently, on very large files (TIFF, large video files > 2GB) during the multi-part upload stage.

I’ve already implemented the best-practice fixes for S3 signature issues (handling headers, checking server time, token refresh). Since these large files require very long connections for hundreds of parts, I suspect the issue is related to network resilience and connection timeouts during the long S3 pre-signed URL window, rather than a bug in the signature itself.

My Question to the Team is:

What is the officially recommended, most robust, and resilient strategy for developers handling a large volume of very big files (> 2GB) via the V4 API, in an unmanaged network environment (like a NAS Docker container)?

  1. Should we rely on the standard multi-part upload initiated by the V4 API?
  2. Or is it better to flag these large files in the sync script and pass them off to a specialized tool like the Frame io Transfer App or FIO Uploader CLI, which are likely optimized for this exact challenge?

The goal is 100% data integrity and resilience for multi-hour uploads.

Thank you for your guidance on this workflow design choice!

Kind regards,
Freddy

Hi @freddy you’re going to want to use AWS multi-part upload for uploading to Frame.io. When you make the call to Frame.io for a local upload the server will send back a series of URLs for you to upload to which means you then need to chunk your file for the amount of URLs that are returned and then upload those chunks to those URLs. Frame will assemble them all together in the background so you dont have to worry about that part.

While we don’t yet have the upload logic in our V4 SDK, here’s some pseudo code using our SDK to help:

import os
import time
import mimetypes
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from frameio import Frameio


def build_retrying_session() -> requests.Session:
    session = requests.Session()

    # Retries for transient failures (network hiccups, 5xx, throttling, etc.)
    retry = Retry(
        total=8,
        connect=8,
        read=8,
        backoff_factor=0.75,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["PUT"],
        raise_on_status=False,
    )
    adapter = HTTPAdapter(max_retries=retry, pool_connections=8, pool_maxsize=8)
    session.mount("https://", adapter)
    return session


def upload_local_file_v4(
    *,
    client: Frameio,
    account_id: str,
    folder_id: str,
    file_path: str,
    poll_interval_s: int = 5,
) -> str:
    file_size = os.path.getsize(file_path)
    file_name = os.path.basename(file_path)

    # 1) Create placeholder file + get presigned upload_urls
    create_resp = client.files.create_local_upload(
        account_id=account_id,
        folder_id=folder_id,
        data={"file_size": file_size, "name": file_name},
    )

    # SDK typically returns a dict-like object with "data"
    file_obj = create_resp["data"] if isinstance(create_resp, dict) else create_resp.data
    file_id = file_obj["id"]
    media_type = file_obj.get("media_type") or (mimetypes.guess_type(file_path)[0] or "application/octet-stream")
    upload_urls = file_obj["upload_urls"]  # list of { "size": int, "url": str }

    # 2) Upload each part (IMPORTANT: use the per-part size returned by the API)
    session = build_retrying_session()

    # Timeouts: (connect_timeout, read_timeout). Tune read_timeout for slow links.
    timeout = (10, 300)

    with open(file_path, "rb") as f:
        for idx, part in enumerate(upload_urls, start=1):
            part_size = int(part["size"])
            url = part["url"]

            chunk = f.read(part_size)
            if len(chunk) != part_size:
                raise RuntimeError(
                    f"Read {len(chunk)} bytes for part {idx}, expected {part_size}. "
                    "File changed during upload or sizing mismatch."
                )

            resp = session.put(
                url,
                data=chunk,  # bytes to avoid chunked transfer encoding surprises
                headers={
                    "Content-Type": media_type,
                    "x-amz-acl": "private",
                },
                timeout=timeout,
            )

            if resp.status_code != 200:
                raise RuntimeError(
                    f"Part {idx} upload failed: HTTP {resp.status_code} - {resp.text}"
                )

    # 3) Poll for completion (Frame stitches + processes automatically)
    while True:
        status_resp = client.files.show_file_upload_status(account_id=account_id, file_id=file_id)
        status_obj = status_resp["data"] if isinstance(status_resp, dict) else status_resp.data

        if status_obj.get("upload_failed"):
            raise RuntimeError(f"Frame.io reports upload_failed=true for file_id={file_id}")

        if status_obj.get("upload_complete"):
            return file_id

        time.sleep(poll_interval_s)


# Example usage:
# client = Frameio(base_url="https://api.frame.io", token="YOUR_ACCESS_TOKEN")
# file_id = upload_local_file_v4(
#     client=client,
#     account_id="...",
#     folder_id="...",
#     file_path="/mnt/nas/huge_video.mov",
# )
# print("Upload complete:", file_id)

If you’re using AI to help you code, you can use our “Open in ChatGPT” function on our developer docs: Create file (local upload) | Frame.io API Documentation

We don’t have an uploader CLI at this time nor do we have auto-upload from Transfer just yet (we do have the MacOS app on the app store that has a “watch folder” style upload if you want to use that instead.

Hi Charlie,

Thank you very much for your detailed explanation and for taking the time to share the multipart upload approach. That clarified exactly what was missing on our side and helped us move forward.

Everything is clear for now and we don’t have any further questions at this stage. Really appreciate your support.

Best regards,

Freddy

Hi Charlie,

I am currently developing a synchronization service between Frame.io v4 (Next) and a local NAS. I am encountering a persistent issue regarding the retrieval of Ratings (Stars) via the API.

The Context:

  • In the web interface, my assets (e.g., protypes-1143.NEF) are clearly marked with a 5-star rating.

  • I am using the Account ID: *************7.

  • Target Asset ID: **************cecdf0ee.

The Problem: When querying the asset details, the rating information is missing or empty. Specifically:

  1. Standard Query: A GET request to /v4/accounts/{{account_id}}/files/{{asset_id}} returns the expected file metadata, but the rating or label fields are absent from the root of the data object.

  2. essentials Block: Following your previous advice, I checked the essentials block. However, for these assets, it returns an empty array [].

  3. Invalid Parameters: Attempting to force the inclusion of metadata using ?include=metadata or ?include=custom_fields results in an “Invalid value: Unexpected field” error (422 Unprocessable Entity).

My Questions:

  • Has the schema for Ratings changed recently in the v4 “Next” architecture?

  • Is there a specific include parameter or a different endpoint I should use to reveal “Creative” metadata like stars?

  • Is it possible that certain assets (like .NEF RAW files) store these values in a different sub-object than the one found in essentials?

Thank you for your time and help!

Best regards,

freddy

hi @freddy if you haven’t resolved this issue you need to be sure to use the /metadata endpoint and not the include=metadata, confusing I know.