At Josys, processing each organization’s data as a separate job made daily analytics increasingly expensive and time-consuming. IDAC 2.0 changed how we scheduled, grouped, and processed that work.
The system behind the reports
IDAC—our Ingestion, Data, Analytics, and Compute framework—powers analytics from multiple integrations. IDAC 2.0 moved the affected workloads to scheduled Kafka reads, shared processing across organizations, and parallel source transformations. The goal was lower cost and shorter processing windows while preserving the intended report calculations.
Our earlier engineering post introduced the distributed foundation: Kafka, Apache Spark, and the medallion architecture, with Bronze, Silver, and Gold layers. Apache Airflow coordinates the processing steps. Customers connect different integrations, so data volumes and source combinations vary by organization.
Why separate organization jobs became a scaling bottleneck
IDAC 1.0 processed Bronze data every 30 seconds, while the reports consuming that output refreshed only once or twice a day for each customer. Continuous writes, small daily increments per organization, and fine-grained partitioning produced many small files and growing Delta transaction logs.
Each organization’s data was processed through a separate Airflow job. Jobs ran concurrently, but available compute and Airflow concurrency limits capped how many could run at once. Each job still took time to start, discover and read its many small files, and complete its transformations—even when it contained little new data. Organizations waiting for a slot extended the overall processing window.
IDAC 1.0 took 18–21 hours to complete daily processing. Because each organization required a separate job, processing time grew with the number of organizations, rather than primarily with the amount of new data. Further growth could push processing beyond 24 hours, leaving one day’s work unfinished when the next day’s cycle began. This made the organization-by-organization execution model a scaling bottleneck.
What we tried first
We first checked the report delivery timelines required by our service-level agreements (SLAs), then examined the daily incremental data volume, small-file counts, and time taken by each organization’s job.
That led to concrete questions: were small files dominating processing time? Did reports refreshed once or twice a day need continuously running streaming jobs? Did the partition layout fit the way we read data? Were the Spark configurations justified by such small increments?
Our first attempt to address the small files was to run OPTIMIZE to compact them into larger files. It did not make a substantial difference to the overall bottleneck: streaming jobs continued generating small files, and the partition layout, Spark configurations, and separate organization-level jobs remained unchanged.
Compaction treated the accumulated files, but not the processing model that kept creating the problem. We needed to change how the work was grouped and scheduled, alongside the file layout and compute configuration. That shaped the move to shared batches.
What changed in IDAC 2.0
Read Kafka in scheduled batches
We retained Kafka as the ingestion boundary and read accumulated events during scheduled windows. For reports refreshed once or twice a day per customer, continuously processing those events added no reporting value. Scheduled batches aligned the work with when the reports actually needed it.
Group data for the way we process it
For the affected Bronze and Silver paths, we shifted from organization/date/hour partitioning toward event-date-oriented partitions, with a job-type dimension where needed. Processing multiple organizations together avoids dividing small daily increments into so many tiny writes and repeated scans, while the organization-aware transformations provide built-in tenant isolation so data does not overlap or mix across different tenants.
Run independent source transformations in parallel
Source transformations run independently and in parallel, preserving Bronze → Silver → Gold dependencies within each path. If one source fails, the daily report data from other sources can still refresh. Recovering the failed source does not require blocking updates from the healthy sources.
Use source-specific compute configurations
Sources and stages use separate Spark resource profiles rather than one configuration sized for the largest workload. Driver and executor resources are configured for the workload: smaller data volumes use less compute, while larger volumes receive more processing capacity.
The trade-offs we accepted
- A larger blast radius at the shared rollup. A failure in the shared Gold rollup can delay the final daily refresh across organizations. We accepted this trade-off because the shorter processing window leaves substantial headroom within our reporting SLA, and because we do not expect transformation logic to change between IDAC 1.0 and IDAC 2.0; all business transformations remain stable. Performance, scalability, and reliability testing validated resource sizing for the tested workloads. Checkpoints and idempotent writes support safe reruns if needed. A source-specific failure still does not block refreshes from other sources.
- More concentrated downstream load. Instead of emitting smaller updates to the core database throughout the day, batch completion produces a larger burst of updates in a shorter window. We handled this using the core service’s existing parallel processing threads and increasing its CPU capacity.
Validate the intended business calculations
The objective was to preserve the intended business calculations, not redesign transformation semantics. We compared organization-level outputs and report metrics, accounting for collection cutoffs and known baseline issues. The validation record distinguished timing differences, earlier processing defects, and cases needing investigation. Correctness required explaining those categories, rather than optimizing for a single match percentage.
What the production observations showed
Over 20 days of monitoring, average reported daily cost was 61.8% lower after IDAC 2.0. This is the reduction reported in the rollout cost update. The comparison covers the same production environment after IDAC 1.0 was retired.
Reported daily processing time fell from 18–21 hours to 2–2.5 hours. After 20+ days, the pipeline duration ranges between 1 hour 30 minutes and 2 hours. These durations describe pipeline completion, not event-to-report freshness.
What we learned
- Choose cadence from the consumer backward. Continuous upstream processing needs a freshness requirement that justifies its cost.
- Measure repeated work as well as useful computation. Small increments can cost more to schedule and discover than their size suggests. File layout and job boundaries deserve attention alongside compute sizing.
- Scale processing with data volume, not organization count. Adding organizations with little new data should not add a proportional amount of processing time. Group work so that the volume of data drives compute demand, rather than creating a separate job for every organization.
Acknowledgements
Thanks to the entire IDAC team and leadership for their work and support across every aspect of IDAC 2.0.
References
Data Engineering at Josys · Apache Kafka · Apache Spark · Apache Airflow · Delta Lake





