How Do You Extract Structured Data From a Legacy System With No Bulk Export?
Insurance core systems and utility billing platforms are two of the most data-rich, API-poor legacy systems in production today. They hold exactly the information modern products need, policy details, coverage limits, billing amounts, usage history, but almost none expose a reliable API. Deck exists because legacy system integration for data extraction has to work around that gap, not wait for it to close. This guide covers the realistic approaches to pulling structured data out of legacy systems that were never built to share it, using insurance and utility portals as the sharpest example of the broader problem.
What Makes Legacy Systems Like Insurance and Utility Portals So Hard to Extract From?
A login-gated legacy system is any application, mainframe, or portal that holds valuable data behind an authentication flow designed for a human, with no programmatic export path. Insurance and utility platforms sit at the extreme end of this category, alongside ERP modules and government portals, for four consistent reasons. Access is login-gated and session-based, with username and password, MFA, CAPTCHA, and SSO sessions that expire after inactivity. Data belongs to individual users rather than an organization, so extracting it means acting inside that specific user's authenticated session, not a shared organizational instance. Presentation varies wildly between vendors: one carrier renders coverage limits in an HTML table, another buries them in a downloadable PDF. And bulk export rarely exists at all. You get a rendered page, a PDF, or nothing. Every one of these traits also describes SAP modules, government benefits portals, and internal ERP tools, which is why legacy system integration keeps surfacing as one problem instead of many separate ones.
What Use Cases Actually Need This Kind of Legacy System Integration?
Four use cases show up repeatedly:
- Cross-account tracking: giving users a complete view of insurance premiums and utility bills together, in a place that exists nowhere else by default.
- AI agent bill pay and claims management: agents paying bills, filing claims, and tracking status, each requiring navigation through the target portal with the user's own credentials.
- New customer onboarding: reading existing policy or account data during signup instead of asking a user to transcribe it manually.
- Lending and underwriting: using utility payment history or insurance coverage data as a risk input at a scale manual review cannot match. Each one depends on the same underlying capability: reliable, structured access to a legacy system that has no API.
Does Screen Scraping Work for Legacy System Integration?
Screen scraping, an HTTP client requesting pages and parsing the returned HTML, is the oldest approach to this problem. It works for simple portals with stable markup, data rendered directly in the page rather than loaded asynchronously, and use cases that can tolerate occasional failures. It breaks down almost everywhere else in this category. JavaScript-heavy portals render only a shell to a plain HTTP client, so the actual data never arrives. Anti-scraping defenses, bot detection, session fingerprinting, and CAPTCHA are common on insurance and utility platforms specifically. Data delivered as a PDF or image defeats HTML parsing outright, and any UI update changes the element structure the scraper depends on. Screen scraping is a proof of concept, not a production path for legacy system integration at this level of difficulty.
What About Custom Browser Automation?
Browser automation with Playwright or Puppeteer controls a real Chromium instance instead of raw HTTP requests. It renders JavaScript, handles cookie-based sessions, and interacts with dropdowns, date pickers, and modals the way a person would, which makes it a real step up from scraping. The tradeoff is maintenance. Building navigation logic per portal is real engineering work, and that logic breaks every time a portal changes its layout. Running headless browsers at scale needs meaningful infrastructure for concurrency and memory, MFA and consent dialogs require special handling, and some legacy portals now detect and block headless browser traffic specifically. For one or two portals, this is manageable. For legacy system integration across dozens of carriers or utilities, it becomes a full-time maintenance function.
What Does User-Permissioned Access Through a Platform Look Like?
Deck is a computer use agent platform that automates workflows by operating any web interface directly, so the target legacy system never has to expose an API for it to work. Instead of building browser automation per portal, an application sends a structured request to Deck's API specifying what data it needs and from where, and Deck handles authentication, navigation, extraction, and error handling, returning structured data rather than a scraped HTML blob or an unparsed PDF. This matters most at scale. A product supporting insurance in the United States might need hundreds of carriers and dozens of utilities, and building custom extraction logic for each is not a realistic engineering budget. It also matters because user-permissioned access is the correct model for this data in the first place: insurance and utility account data belongs to the individual user, and accessing it inside their own authenticated session, with their consent, is how the system was designed to be used. Deck's approach to legacy system integration keeps that model intact instead of routing around it with a shared credential.
How Do You Handle MFA, PDFs, and Session Timeouts?
A few problems show up across nearly every legacy system integration project in this category. MFA: a user-initiated session where the person completes the MFA step directly is the most reliable path. TOTP support works when a secret is securely stored. SMS interception is technically possible but increasingly blocked by carriers and should be treated as a fallback, not a plan. Data delivered as PDFs: policy documents, EOB statements, and billing summaries often exist only as PDFs, which requires a separate parsing or OCR step. Budget for this explicitly if the target portals lean on documents instead of rendered data tables. Session timeouts and pagination: insurance portals often expire sessions after a few minutes, and utility portals paginate billing history across multiple pages. Treat extraction as an explicit state machine, with each step verifying its expected state before moving forward, rather than a linear script that assumes every page loads the same way twice. Deck handles MFA, session continuity, and document extraction as part of the platform, which removes these as separate engineering problems for every new legacy system integration.
How Do the Three Approaches Compare?
| Factor | Screen Scraping | Custom Browser Automation | Deck |
|---|---|---|---|
| Target portal count | 1 to 2, very stable | 1 to 5, with engineering resources | Many, or limited engineering capacity |
| JavaScript-rendered content | No | Yes | Yes |
| MFA handling | No | With significant effort | Yes |
| Per-portal maintenance | High | High | Managed by the platform |
| Reliability in production | Low | Medium | High |
What Does This Look Like in Practice?
A property and casualty insurance aggregator needed to pull active coverage status from over 40 regional carrier portals, none of which offered an API, to power a real-time quoting tool. An initial attempt with custom Playwright scripts covered 12 carriers before the maintenance load outpaced the team's capacity, with roughly two portals breaking every month after layout changes. The team rebuilt the integration on Deck, connecting each carrier through Deck's API using the end user's own login. Coverage checks that previously required a 30-minute manual call to a carrier's support line now return in under two minutes, and the aggregator now supports all 40-plus carriers without a dedicated maintenance engineer for the integration layer.
FAQs
Does Deck work for legacy systems outside insurance and utilities?
Yes. Deck's approach to legacy system integration applies to any login-gated system with no API, including ERP modules like SAP and Oracle, government portals, and payroll platforms, using the same user-permissioned access model.
Is Deck a replacement for custom browser automation?
For a single stable portal, custom automation with Playwright can work. For legacy system integration across many portals, Deck removes the per-portal maintenance burden that makes custom automation expensive to scale.
How does Deck extract data that only exists as a PDF?
Deck downloads the document as part of the session and returns it alongside structured data, so a PDF-only policy document or billing statement does not require a separate manual retrieval step.
Does Deck handle MFA during legacy system integration?
Yes. Deck manages MFA and CAPTCHA challenges as part of connecting a user's account, which is one of the most common failure points for automated access to insurance and utility portals.
What is the difference between screen scraping and what Deck does?
Screen scraping parses static HTML and fails against JavaScript-rendered content or anti-bot defenses. Deck operates a real browser session the way a user would, handling dynamic content, authentication, and structured extraction as a managed platform rather than a one-off script.
Ready to get started?
See how Deck can connect your product to any system — no APIs needed.
Build my Agent →