正文 · 原文
该语言的正文暂不可用,当前显示已有版本。
This post is co-written by Mariah Miller and Sirajus Salekin from AWS and Nicholas Chase from Social Department. Entertainment marketing teams manage tens of thousands of video assets across studios, networks, and streaming applications. Most campaigns begin with the same video discovery problem: finding the one moment in the footage that will resonate with an audience. When a brief lands, the clock starts, and what they need is almost never something a database can describe. Semantic search, on its own, has a ceiling: It answers the question you can articulate. Ask it for a character holding a coffee mug and it will find every coffee mug in a thousand episodes. The queries that actually drive entertainment campaigns look nothing like that. They are questions of judgment: which moment in a show will travel, which line is quotable, and which single frame will work as a meme. Those questions can’t be typed into a search box. Answering them requires two things the search index doesn’t have: an editorial definition of what “quotable” or “memeable” means for a given library, and a continuously refreshed view of what is moving outside the library. Historically, answering them meant a person with deep knowledge of the show watching footage for a week, and that person doesn’t scale across a catalog. In this post, we describe how Social Department, the AI operating system for entertainment marketing, set out to encode those editorial questions as executable retrieval pipelines. The result is QuickSearch, a set of single-action queries that return the clips a marketer is looking for without requiring them to describe it: Shareable Scenes, Quotable Quotes, Memeable Moments, and Trend Align. The models in this solution run on Amazon Bedrock, so customer data stays within the AWS data boundary. It isn’t shared with model providers and isn’t used to train the underlying models. Social Department’s approach to responsible AI is that AI proposes and people decide. In production, code checks generated output against the source material (for example, a quote is discarded if it isn’t in the episode transcript). Amazon Bedrock Guardrails helps screen customer search text for prompt attacks, and a person on the customer’s team approves every post before it publishes. Social Department owns the pipeline architecture, the retrieval system, and the editorial criteria behind each action. TrackIt, an AWS Premier Tier Services Partner, implemented and delivered against that design. TwelveLabs’ Marengo 3.0 and Pegasus 1.2 models on Amazon Bedrock provide the scene-level video reasoning the pipeline builds on. AWS supplies the infrastructure, the model access including Anthropic’s Claude models, and Amazon OpenSearch Service, using its serverless option, Amazon OpenSearch Serverless, for retrieval. A note on scope: This post describes the AWS portions of the system: the ingest and indexing pipeline, the retrieval path, and the services that run them. The components Social Department owns, including its media and data backbone, content analysis and discovery engine, editorial criteria, and evaluation program, are described only to the depth needed to follow the architecture. The named services that follow are the ones this post is about, not an inventory of Social Department’s full system. About Social Department Social Department is the AI operating system for entertainment marketing, serving major studios, networks, streamers, and FAST applications. It automates campaign strategy, asset creation, publishing, and performance optimization across a client’s full content library. Most of those functions depend on locating the right moment inside the footage first, and semantic clip discovery has been core to the product from the beginning. Why semantic search alone was insufficient Entertainment marketing operates under two constraints that a general-purpose search layer can’t satisfy. The valuable queries are the ones nobody can phrase A marketer can describe an object, a location, or a line of dialogue, but not what will travel. That judgment, this moment lands and that one doesn’t, is the actual skill in the job. It was never automatable because it depends on knowing how an audience behaves rather than what a video contains. A search layer that only understands content requires a human to supply the intent, and supplying that intent well requires having already watched the show. Trend signal lives outside the library and moves on a different clock No amount of video understanding tells you what audiences are talking about this morning. That information is created outside the library and changes hourly, while a library changes only when new content is added. A system that answers these queries has to acquire trend signal continuously and reconcile it against the library at request time. That distinction drove the core architectural decision: content understanding and trend acquisition run as independent pipelines on their own schedules and are joined only when a user selects a QuickSearch action. Solution overview With QuickSearch, teams can interact with a video library in a different way, one where the system proposes rather than waits. It provides four actions, each with its own retrieval strategy, editorial criteria, and ranking model. The walkthrough that follows describes how each is implemented. The list here introduces what they do. Shareable Scenes surfaces substantial, self-contained scenes such as extended confrontations, climaxes, and set pieces, biased toward developed sequences rather than quick reactions. Quotable Quotes finds punchy, quotable dialogue capped at short clip lengths, weighting up lines that have demonstrably circulated where that signal is available for the title. Memeable Moments identifies individual frames that will function as memes and generates a caption grounded in the actual frame. A dedicated extraction and scoring step chooses the frame rather than taking a thumbnail. Trend Align takes what is trending right now and finds the moments in the library that legitimately connect to it, so a title that finished airing years ago can re-enter a live conversation. Alongside these, Social Department keeps natural-language search for everything the actions don’t cover, and image-based search, where a reference image returns visually similar content across the library. The following diagram shows how the three parts of the architecture connect. The content understanding pipeline makes the library searchable, and the trend acquisition lanes pull in external trend signal. The QuickSearch job layer joins the two at request time. Figure 1: High-level architecture Architecture walkthrough The system has three parts that run on independent schedules. Two acquire and structure information. The third resolves a user’s single selection into a multi-stage retrieval operation across both. Content understanding pipeline This pipeline makes the library legible. It works asynchronously on ingest. An AWS Step Functions state machine orchestrates roughly two dozen Python AWS Lambda functions end to end. The following steps trace one video from upload to indexed clips. Upload and trigger – Video lands in a per-team Amazon Simple Storage Service (Amazon S3) bucket. The S3 Object Created event, routed through Amazon EventBridge, starts the Step Functions execution. Metadata – The first Lambda function resolves title, team, episode, and public identifiers against Social Department’s media and data backbone, writes video metadata to Amazon DynamoDB, and reports the asset as processing. Transcode – AWS Elemental MediaConvert transcodes the asset to 480p H.264 with adaptive deinterlacing, using a Step Functions task token so the state machine waits on the job rather than polling. Shot detection – Amazon Rekognition performs asynchronous segment detection with an Amazon Simple Notification Service (Amazon SNS) callback. A second Lambda function merges and splits the raw shots into content-aware ranges of roughly 10–20 seconds. Those ranges are the unit the rest of the pipeline fans out over. Within a range, Social Department segments embeddings at a fixed interval, a design choice tuned to the pacing of episodic television. The indexed clip is short and fixed-length, but the range it sits in follows the cut. Embedding – Ranges are processed in a Step Functions Distributed Map. Each call passes a start and end time to TwelveLabs Marengo 3.0 on Amazon Bedrock through asynchronous invocation. The call produces multimodal embeddings across visual, audio, and transcript dimensions, with no file splitting needed on this path. Scene description – TwelveLabs Pegasus 1.2 on Amazon Bedrock runs per segment in a parallel Distributed Map, producing scene-level narrative descriptions, emotions, themes, dialogue type, and narrative position, which form the text layer the ranking models later reason over. Pegasus takes pre-cut files rather than time ranges, so a separate Lambda function cuts each range first, with a few seconds of padding on either side to preserve scene transitions. Transcription and enrichment – In parallel with embedding, the pipeline runs speech-to-text with automatic fallback, and Amazon Rekognition for on-screen text, visual labels, and face collections used in character grounding. Anthropic’s Claude models on Amazon Bedrock extract the show’s character map and generate show-level concepts and character roles. Indexing – Amazon Bedrock writes embedding output to Amazon S3. A final Distributed Map then indexes embeddings, transcript, on-screen text, scene description fields, and labels into Amazon OpenSearch Serverless. The pipeline is event-driven end to end. Every Lambda task retries with exponential backoff. The scene-description stage adds a dedicated throttling retry with full jitter. Non-critical enrichment steps degrade gracefully rather than failing the run, and a hard failure routes to an error handler that records the reason and reports the asset as failed. Same-Region staging for Marengo. Marengo is available in a specific set of AWS Regions. Asynchronous invocation in Amazon Bedrock reads input from and writes output to an Amazon S3 bucket in the same Region as the endpoint. The pipeline is built around those parameters. It stages each video into a dedicated bucket in US East (N. Virginia) before invoking Marengo, then reads the embedding output back from the same bucket. The rest of the infrastructure stays in its home Region while each embedding job runs alongside the model. For model availability by Region, see Model support by AWS Region in Amazon Bedrock. Trend acquisition This pipeline operates continuously and independently of any library, and it is what makes Trend Align possible. Two lanes, one feed – Trend data is acquired through two lanes with deliberately different mechanisms. The daily lane uses an Amazon EventBridge cron, one scheduled fetch per region per day, timed to each region’s local morning rather than a fixed UTC hour. The heat lane (Heat Seeker) reads early-stage, momentum-ordered trends, and is fetched only when a user explicitly asks for it, reusing what is already stored if it is recent enough. The split is a cost decision. A cron pays for every region on every tick whether or not anyone is working, while a demand-shaped fetch behind a TTL costs nothing on a quiet day. Regional feeds are separate populations – Trends are acquired per geographic region, currently four feeds: worldwide, the United States, the United Kingdom, and Australia. A region isn’t a filter over a global list. Each is discovered separately on its own schedule, so the same day’s trends genuinely differ between them. Interpretation and the editorial gate – Raw trend data isn’t shown to a customer. Anthropic’s Claude Haiku on Amazon Bedrock normalizes each trend onto Social Department’s own category schema, standardizes its description, deduplicates near-matches, and caps how many survive per category. In the same pass, an editorial gate drops trends whose primary subject is death, violence, self-harm, natural disaster, politics, protest, crime, or a public-health crisis. A marketing customer isn’t offered these, regardless of how large the trend is. Rejected trends are remembered so each fetch doesn’t re-screen them. Storage as first-class rows – Interpreted trends are upserted by stable fingerprint, so the same trend day-over-day reuses its row and its cached derivations. Trends that stop appearing age out and are eventually retired. Refresh runs take a claim so two cannot overlap. Quotable Quotes draws on a second, narrower input: an external dialogue popularity signal, used strictly to weight lines that have already circulated. It’s optional by design, so if the signal is unavailable a line gets no boost rather than failing the job. The QuickSearch job layer This layer is where a single selection becomes a retrieval strategy. It executes each action asynchronously and returns a ranked set of clips. Request path – The web application is a React single-page application served from Amazon S3 through Amazon CloudFront. Its API is a Fastify service running on AWS Lambda behind an Amazon API Gateway HTTP API, with Amazon Aurora PostgreSQL-Compatible Edition for relational data. Asynchronous execution – Selecting a QuickSearch action writes a job record to Amazon DynamoDB and enqueues an Amazon Simple Queue Service (Amazon SQS) message. A worker Lambda function picks it up and runs the pipeline for that action. Transient failures retry through Amazon SQS, and errors classified as non-retriable go straight to a dead-letter queue. Query formulation – Each action expands into a set of retrieval queries shaped by its editorial criteria. For Trend Align, this is where most of the proprietary work sits: A trend arrives as a cultural topic, and what a video index can match is what appears on screen and in the audio. Bridging those two is a translation problem, which Claude models on Amazon Bedrock handle. The output is a set of queries the index can act on. Multi-channel retrieval – Each query goes to the search Lambda function, which handles hybrid retrieval. Marengo embeds the query once, as text, an image, or a composed text-plus-image query. The embedded query then executes across six k-NN channels (visual, audio, transcript, on-screen text, scene description, and labels) and seven BM25 text channels. Per-modality weights come from the query itself. Query-type routing adjusts channel boosts so dialogue queries lean on transcript and emotion channels while visual queries suppress them. Fusion and reranking – The channels are combined with Reciprocal Rank Fusion, which uses rank positions rather than raw scores and so avoids the scale mismatch between k-NN and BM25 similarity. The top candidates then pass through the Amazon Rerank model, a cross-encoder called through the Rerank API in Amazon Bedrock, for a final text-to-text relevance pass, followed by a Claude validation step over the top results. Aggregation – The index stores short fixed-length clips. Before results are returned, adjacent clips with high visual-embedding similarity are merged into coherent moments, so a query for a scene returns one 20–60 second result rather than five fragments of the same scene. Editorial ranking – A Claude model on Amazon Bedrock ranks the deduplicated, enriched candidates against the action’s editorial criteria. The pool is capped so the model’s output can’t truncate. Model choice is per action: Claude Sonnet where cultural recall changes the answer, Claude Haiku where the task is bounded and cost-sensitive. Frame extraction – For Memeable Moments, a dedicated Lambda function extracts several candidate frames from the winning clip and scores them for face quality using Amazon Rekognition and for image sharpness. It then re-extracts the winner at full resolution, trims letterboxing, and returns a signed Amazon CloudFront URL. Caching – Completed results are stored in Amazon DynamoDB and reused for 30 days. The cache key includes a per-action pipeline version and, for Trend Align, a hash of the active trend set, so a criteria change, a model upgrade, or a refreshed trend catalog invalidates cleanly rather than serving a stale answer. Design decisions Three choices made culturally informed retrieval tractable. Separating trend acquisition from content understanding A library changes when new content is added. Trends change daily, and early-stage trends faster than that. Coupling the two would mean either reprocessing content against fresh signal, which is computationally untenable, or serving stale trend judgment. Keeping them as independent pipelines joined at request time means signal stays current without the library ever being touched again. A title that finished airing years ago can re-enter a live conversation. Splitting each action into cheap retrieval and expensive judgment The naive design puts one large model in front of the whole library. QuickSearch instead uses inexpensive, high-recall vector and keyword retrieval to build a candidate pool, then spends model budget only on ranking a capped set of real candidates. The caps are sized against the ranking model’s output token limit, because a ranker that truncates mid-response silently drops results. This also keeps the per-query cost of Amazon Bedrock predictable: one embedding call, one rerank call, and a small fixed number of Claude invocations, regardless of library size. Making cache invalidation an explicit editorial act Results are cached for 30 days. Without intervention, a change to an action’s criteria or model would stay invisible to users until the cache expired. Each action carries a pipeline version that is bumped in the same change that alters its behavior. That bump invalidates prior results immediately, so editorial judgment is refined as a versioned change to the product rather than relitigated on every query. Results Social Department reports the following outcomes from customer use of QuickSearch. These are the company’s own observations rather than a controlled benchmark. Queries that were previously impossible are now a single selection – Identifying the moments a fandom has canonized, or the frames that will function as memes for a specific audience, previously required a person with deep familiarity with the title watching footage, and it didn’t scale across a catalog. Discovery no longer depends on manual scrubbing – According to Social Department, campaign discovery work that used to take 3–4 days now takes about 10 minutes with QuickSearch. The time a marketer spent scrubbing through episodes for the right moment goes back to higher-value campaign work. Marketers reach candidate clips without composing a query – That removes the requirement that the person searching already knows the library well enough to describe what they want, the constraint that made discovery depend on the few people who had watched the show. Archived catalog becomes addressable – Content that was effectively unreachable, because it was finished, archived, and invisible to metadata search, becomes usable for campaign work, and titles that stopped airing years ago can re-enter live cultural conversations. Industry recognition – Social Department was showcased as a top startup at NAB 2026 and featured in the Entertainment Loft. “The hard part here was never the search. It was working out what to search for: how you take something a great marketer does on instinct, about which moment lands and why, and turn it into criteria a system can execute a thousand times. We spent a long time on that. Working with TrackIt on a compressed timeline, we went from spec to a live demo at NAB. TwelveLabs gives us the intelligence to understand what’s inside the footage and AWS gives us the infrastructure to do it at library scale. Our job is to know what to point it at.” — Jonathan Verk, Founder and CEO, Social Department “Everyone in media talks about adding metadata to their libraries but tagging content doesn’t pay for itself. What I like about this project is that it closes the loop. It takes shows sitting in an archive and makes them useful for real campaigns, at a cost that makes sense to run every day. That’s the part the industry has been missing.” — Chris Marchitelli, Vice President, TrackIt What we learned The following four lessons from the project stood out. The hard problem was expressing judgment, not retrieving content Retrieval is largely solved. What isn’t solved is specifying intent precisely enough for retrieval to be useful without an expert marketer in the loop. Most of the engineering effort went into defining what qualifies as a shareable scene or a memeable frame, tightly enough to execute repeatedly across libraries with different tones and audiences. That definition is the output of a standing evaluation program of numbered iterations, each measured against per-show ground-truth sets and a tracked baseline, that tunes the criteria against results rather than intuition. A multi-model architecture on Amazon Bedrock QuickSearch uses a multi-model architecture, with each model assigned to the work it fits best. Claude Haiku handles bounded, cost-sensitive output, and Claude Sonnet handles tasks where cultural recall changes the answer. TwelveLabs Marengo produces multimodal embeddings, TwelveLabs Pegasus provides scene-level video reasoning, and the Amazon Rerank model handles reranking. Amazon Bedrock made this architecture practical to build and run. Models from TwelveLabs, Anthropic, and Amazon are available through one API, so Social Department can change the model behind a task when measured quality moves, without building a new integration. Asynchronous invocation runs long video embedding jobs at library scale, and the Rerank API adds a relevance pass without separately hosted infrastructure. Amazon Bedrock Guardrails helps screen search input in the same service, and customer data stays within the AWS data boundary. Because Amazon Bedrock is fully managed, the team operates no model-serving infrastructure of its own. Signal that decays needs its own architecture Treating trend data as another input to the content pipeline would have failed, because the two have incompatible lifecycles. Recognizing early that library content and live culture are separate systems, with separate refresh rates, storage, and failure modes, was the decision the rest of the design depended on. Clear layer boundaries made a multi-party project work Social Department owned the pipeline architecture, retrieval system, and editorial criteria. TrackIt built and delivered against that architecture, and TwelveLabs supplied the video understanding models. AWS, with the AWS Generative AI Innovation Center, provided the infrastructure, model access, and solution architecture for the AWS components. Because each party had a clearly defined scope, the 8-week production timeline held. Conclusion We described how Social Department, working with TrackIt, TwelveLabs, and AWS, built culture-aware video discovery for entertainment marketing using its own pipeline and retrieval architecture. Several AWS components make up the layer this post describes. TwelveLabs Marengo handles multimodal embedding and TwelveLabs Pegasus handles scene-level video reasoning, both on Amazon Bedrock. Anthropic’s Claude models on Amazon Bedrock handle interpretation and editorial ranking, and Amazon OpenSearch Serverless handles hybrid retrieval. AWS Step Functions, AWS Lambda, AWS Elemental MediaConvert, Amazon Rekognition, Amazon DynamoDB, and Amazon SQS run the pipelines around them. The result is a set of single-action queries that return the clips audiences already care about, the lines they already quote, the frames that will work as memes, and the moments that connect to whatever is trending right now. To get started with multimodal video understanding on AWS, explore Amazon Bedrock and Amazon OpenSearch Service. To learn more about the TwelveLabs models available on Amazon Bedrock, visit the TwelveLabs page on AWS Marketplace. To explore how the AWS Generative AI Innovation Center can help your organization, visit the AWS Generative AI Innovation Center page. Acknowledgements: The authors would like to thank Katie Kenney, Pranay Duhan, Xiaoye Qian, Rinne Han, and Dahler Battle for their work on this solution. We would also like to thank the TrackIt and TwelveLabs teams for their support. About the authors Mariah Miller Mariah is a Senior Generative AI Strategist and industry leader for media, entertainment, games, and sports in AWS Forward Deployed Engineering, a role she also held at the AWS Generative AI Innovation Center. She leads customer missions that build production AI systems with enterprise teams across industries, including financial and legal services, and develops reusable solution assets such as multimodal video search, agentic commerce, and voice AI agents. She also built the Innovation Center’s partner program. Outside of work, she likes to spend time outdoors. Sirajus Salekin Sirajus Salekin is an Applied Scientist at AWS Forward Deployed Engineering (formerly the AWS Generative AI Innovation Center). He works with customers and AWS Partners to design and build scalable generative AI solutions, from early experiments through production. His work spans generative AI, multimodal ML, NLP, and video search, combining applied research with delivery to make sure every solution is tied to real business outcomes. Nicholas Chase Nicholas is a Lead Software Engineer at Social Department, the AI operating system for entertainment marketing, where he leads system architecture and the video search stack. Chase drove the retrieval requirements and search quality work described in this post.