TL;DR
We had a subtle but critical payroll bug: deductions were getting silently dropped due to timezone mismatches between us (Rippling) and a partner system (Employee Navigator). What looked like a simple effective_date == pst_now() check turned into a case study in why real-world systems need to be resilient to time, contracts, and tacit assumptions. We solved it with a ScheduledDeductions model and a batch job system. Here's the story.
Background
At Rippling, our 3P Ben Admin team integrates with external benefit platforms like Employee Navigator to sync deduction data with our internal payroll engine. The catch? Rippling Payroll doesn't support future-dated deductions.
So we had an agreement with our partner: they'll send us deduction updates only on the day those deductions are meant to take effect. Our logic was simple:
if deduction.effective_date == pst_now().date():
sync_to_payroll(deduction)
And for a while — it worked. Until it didn’t.
The Bug: When "Today" Isn’t the Same Day
We started noticing missing deductions. No crashes, no retries, no alerts — just... missing deductions in payroll.
After digging through logs, we found a recurring pattern:
The partner sent the data at 1:55 AM CST on 2025-04-14.
But pst_now() on our end still read 2025-04-13, 11:55 PM PST.
So our equality check failed, and we skipped syncing the deduction.
Yup — we were off by one hour, but it cost us payroll data.
The partner was technically sending data on the right date. But in their timezone, not ours.
The Fix: ScheduledDeductions + Batch Processing
We realized that our implementation was too brittle. Timezone edge cases shouldn’t cause silent data loss. So we moved to an architecture built for this.
🔧 Enter: ScheduledDeductions
Instead of syncing deductions immediately, we now:
Store every incoming deduction update in a ScheduledDeductions model
Add metadata like effective_time, companyId, employeeId, etc.
Set the status to "Scheduled"
🕒 Batch Job FTW
Every 6 hours, a cron job runs:
Fetches deductions where effective_time <= pst_now() - timedelta(hours=6) and status = 'Scheduled'
Applies them in batch to EmployeeDeductionType
Marks success as COMPLETED, and collects failures into an errors bucket
🧠 Why This Works
Handles future-dated deductions reliably
Decouples from timezone-sensitive assumptions
Adds observability: we now know how many deductions succeeded or failed
Scales: 1,000+ deductions processed in seconds
Architecture Diagram
Lessons Learned
Equality checks on time are dangerous. Always normalize or buffer.
Tacit contracts break easily. Make assumptions explicit.
Timezones are where bugs go to hide.
Batch systems are more resilient than real-time syncs when dealing with external inputs.
Final Thoughts
This wasn’t the flashiest bug I’ve fixed, but it was the most quietly devastating. And solving it felt like a turning point — not just technically, but in how I thought about system design.
We didn't just fix a bug. We turned a fragile sync system into a robust, scalable pipeline.
Hope this helped someone think a bit deeper about time, contracts, and systems. If you've ever chased a timezone bug at 2 AM, you’re not alone.
