From Batch Windows to Streaming Tables: A Career Spent Rebuilding the Plumbing of Enterprise Data

22 August,2026 04:28 PM IST |  Mumbai  | 

Naveen Bagam.


The people who keep large institutions running rarely appear in the coverage of the technologies they use. A bank that settles a card transaction the moment it is presented, a pharmacy chain that knows whether a drug sits on a shelf in one of more than eight thousand pharmacies, a travel company that can produce years of tax records for a jurisdiction on request: each of those capabilities rests on data pipelines that somebody designed, tuned, and then nursed through the nights when they broke. Naveen Bagam has spent more than eighteen years doing that work, and the arc of his career tracks almost exactly the shift that reorganized the discipline around him.

He began in 2006 in Chennai, writing Pro'C and PL/SQL for Target's distribution system, where the questions were about scheduling merchandise into warehouses and shipping it out to stores. The tools of that era were unforgiving. Performance problems were diagnosed with EXPLAIN PLAN and TKPROF, and a poorly written cursor could stall a nightly batch long enough to matter by morning. Bagam moved from there to Citibank's International Personal Banking division, where he worked on ORBIT, an online real-time banking platform serving customers in London, Jersey, and Switzerland across more than ten modules covering account openings, funds transfers, deposits, mutual funds, and brokerage. The end-of-day process on that system was batch accounting in its classical form: accruals calculated on customer and branch accounts, a handoff file generated, ledgers updated downstream, reports produced for internal audit.

That grounding turned out to matter more than the specific technology. When Citibank consolidated online banking operations for more than seventy countries under a single system called Rainbow, Bagam worked as a subject matter expert on the data conversion, mapping interim files across three legal vehicles and validating that years of accumulated customer records survived the move intact. Conversion projects of that kind are where careless assumptions become visible. A field that was optional in the old schema and mandatory in the new one will surface as thousands of rejected records at three in the morning, and the person who wrote the mapping is the one who has to explain what happened.

The middle years of his career moved him through American industries that each stress data systems differently. At Express Scripts he built extraction and transformation logic for a pharmacy benefits manager whose clients included health insurers and government health programs. At Walgreens he worked on the inventory feed behind Intercom Plus, the proprietary pharmacy system linking retail, mail service, and specialty locations, where store-level stock updates arrived hourly through automated processes. At Eversource he helped design a customer reference database built so that critical self-service transactions would still function when the primary data center or one of its components was unreachable, a requirement that shapes architecture more than any performance target does. At Priceline he developed a tax data mart that centralized transactional history from inception, because tax jurisdictions require it and legal requests arrive without warning. At USG he replaced a twenty-year-old reporting database with a warehouse consolidating Hyperion forecast data, Oracle project actuals, and figures from operations in Mexico and Latin America.

Since 2019 he has worked as a data engineer at Discover Financial Services on card data operations, and the character of the work changed again. Oracle application data has been migrated to Snowflake. Avro files are loaded through a utility written in Python. Analytical models are deployed with Docker on the OpenShift container platform through Jenkins pipelines, with artifacts managed in Nexus and JFrog. He contributed to Kintsugi, an internal framework built to move model deployment out of ad hoc practice and into an MLOps discipline, and he has worked with Kafka and Spark on the streaming side of the same estate. The vocabulary is new. The underlying obligation, that the numbers arriving at the other end are the right numbers, is not.

Alongside the industry work, Bagam has built a scholarly record. He holds a Master of Science in Data Science from Bellevue University and a Master of Science in Information Technology from Anna University. He has authored and co-authored peer-reviewed journal publications and technical books on data engineering, SQL, machine learning, and enterprise analytics, and he holds multiple UK Registered Designs covering data visualization, quality assurance, financial analytics, and data security. He serves on the editorial board of the International Journal of Novel Trends and Innovation, as an associate editor at the Sarcouncil Journal of Applied Sciences, and as a reviewer for international journals in artificial intelligence, cloud computing, and software engineering. He is a Fellow of the Institution of Electronics and Telecommunication Engineers.

His own account of that research runs in terms any of his former colleagues would recognize: practical frameworks for enterprise data integration, SQL optimization, real time analytics, and collaborative data engineering. The industry half of the work has not loosened its grip in the meantime. Among the duties still listed against his current role, alongside the Snowflake migrations and the container deployments, is covering production support on weekends and public holidays when the offshore team is unavailable.

Author: Faizan Farooqui

"Exciting news! Mid-day is now on WhatsApp Channels Subscribe today by clicking the link and stay updated with the latest news!" Click here!
Buzzfeed business Service Technology entrepreneur
Related Stories