From Idea to Prototype in 3 Months: A Case Study in AI Fact-Checking for BigFilter

BigFilter

Building a Toolbox for an Informed Society

BigFilter is a social impact startup dedicated to combating the global information crisis with a host of long-term and short-term solutions. The company is developing a suite of tools designed to help people stay well-informed while seamlessly integrating into their day-to-day and balancing public benefit with business viability.

Industry: AI/ML Big Data

Technologies: Python

Solutions: AI/ML Big Data

The Challenge

Validating Automated Fact-Checking as a Business Concept

"Bleak" and "a threat to democracy" – that’s how a 2016 Stanford University study characterized young people's ability to critically evaluate information. The reason? "Ordinary people," the authors noted, "once relied on (…) subject matter experts to vet the information they consumed. But on the unregulated Internet, all bets are off."

One way Big Filter wants to address this fact-checking gap is by developing a service that enables users to verify information in real time across the platforms where they consume it. The solution would enrich the information in real time by summarizing relevant and high-quality evidence using a combination of custom AI models and human oversight.

BigFilter approached Janea Systems to evaluate elements of the technical feasibility and financial viability of this concept.

Our objective was to develop a prototype - not as an early product, but as a simple testbed for quick and easy exploration of technical challenges and potential features.

Client impact

The Outcome: A Functional Prototype Developed in 3 Months

Janea Systems helped BigFilter gain a deeper understanding of the technical and financial challenges behind their project, allowing for data-driven decisions moving forward.

The Solution: A Pragmatic Approach to AI Prototyping

In order to achieve this result, we used:

LLMs as Stand-Ins for Custom Models: We chose to pursue LLMs as a cost-effective alternative to custom models while seeking to mitigate their limitations.
Segmented Architecture for Improved Observability: We designed the prototype around a structured, step-by-step architecture, which offered greater visibility into each stage of the process.

How the Prototype Works

The workflow was divided into discrete stages, as outlined below:

Stage 1: The user selects a model and enters a claim (Fig. 1): Fig. 1: Selecting a model and entering a claim.

Stage 2: The LLM rephrases the claim for greater specificity and easier evaluation (Fig. 2): Fig. 2: Rephrasing the claim.

Stage 3: The system determines the type of expert best suited to evaluate the claim (Fig. 3): Fig. 3: Matching with expertise.

Stage 4: Relevant topics are suggested from Wikipedia (“Encyclopedic topics”) and Semantic Scholar (“Scientific topics”) (Fig. 4): Fig. 4: Adding relevant topics.

Stage 5: The topics are followed by relevant sources (Fig. 5): Fig. 5: Listing relevant sources.

The sources are indexed using vector search to improve retrieval efficiency for similar future queries.

Stage 6: Evidence is gathered and assessed (Fig. 6): Fig. 6: Gathered evidence (continues beyond screenshot).

Stage 7: The LLM summarizes the findings and provides a reasoned verdict (Fig. 7): Fig. 7: Summary and verdict.

Stage 8: The report includes expandable lists of citations for further reference and verification (Fig. 8): Fig. 8: Expandable citations for and against the given claim.

This structured approach enabled BigFilter to gather insights at every stage, facilitating prototype refinement, data sourcing optimization, and accuracy improvements.

With over two decades of experience, Janea Systems excels at solving complex software engineering challenges across diverse industries. Our team of world-class engineers is dedicated to building mission-critical products and services that drive innovation and deliver lasting impact.

If your organization is looking to bring an AI-driven idea to life or requires expert support in software development, we invite you to connect with us. Let Janea Systems be your trusted partner in navigating the complexities of technological advancement.

Notes

A February 2025 study by the BBC revealed that “51% of all AI answers to questions about the news (…) have significant issues of some form”.

Furthermore, even the most advanced LLMs, including GPT-4.5, fail in 40 to 60% of cases in SimpleQA, a collection of straightforward questions designed to gauge model accuracy (via AAAI 2025 Presidential Panel on the Future of AI Research and Measuring short-form factuality in large language models).

Finally, a new study from March 2025 found that AI search engines cite wrong news sources 60% of the time (via Ars Technica).