Batch-Based Data Processing vs. Real-Time Event-Driven Data Processing

This article is co-authored by Kristoffer Fürst, CEO of Limina, and Doctor Ian Hunt, independent consultant and IBOR expert. Our goal with this article is to summarise what problems batch-based systems cause for Asset Managers, especially in the Front Office. We also want to show what alternatives exist. This article is one in a series about Investment Book of Record Technology.

 

What is an overnight batch process?

Batch processing is the processing of transactions in a group or batch without the need for human interaction. Whilst batch processing can be carried out at any time, it's often executed during the overnight hours outside of regular trading hours, in preparation for the upcoming trading day. 

Examples of overnight batch processes within the buy-side industry are:

• Calculation of various risk measures and scenarios across a set of portfolios.

• Production of an end-of-day position view and cash, to form a closing portfolio.

• Calculation of the NAV for that end-of-day portfolio.

While these processes are generally performed overnight, and deliver results on T+1, they may be run intra-day. In any case, they are still “batch” processes.

Overnight batch processes are usually well suited for applications where a result, once approved, is final. Great examples are NAV calculations and period-end ledgers. Once a NAV has been set and approved, it should never change. If a mistake is found months later, the NAV isn’t updated (unless, in exceptional circumstances, it’s decided that history should be restated).

A batch process does this job very well. The NAV is calculated, adjustments are made and then it’s stored away. The calculation is performed once, as a single ‘bulk’ process.

Another example of a common batch process is the loading of portfolios into an Order Management Software (OMS) or Portfolio Management Software (PMS), to create a start-of-day position view. This is illustrated below:

Front Office systems based on batch processes 

Accounting systems do not generally present a real-time, or even a timely view of portfolios; it’s not what they are for. Transactions are invisible to accounting until they are posted, which generally happens only when they are due for settlement. 

These transactions, impacting both cash and holdings, are processed only in periodic batches, and typically once per day. The key attribute (and strength) of an accounting system is not its timeliness: it is its completeness. Accounting systems must process all transactions in one way or another, and at some point in time; if they don't, then the accounts will be incorrect.

Early initiatives in Front Office and compliance automation exploited this convenient feature of completeness in accounting systems. Essentially, the Front Office cash and position management software was based on end-of-day or start-of-day accounting positions. This removed the obligation of completeness from the new Front Office platforms, allowing them to focus only on trades.

This also took away the need for Front Office software to process awkward non-trade transactions, for example, injections, income, corporate actions, cash fees, liquidations and the rest. The Front Office system could just let such transactions wash through into the accounting positions in the fullness of time, knowing that their impacts would eventually appear in the loaded start-of-day data. In the meantime, the Front Office platform could avoid worrying about them.

Portfolio and Order Management Systems’ approach to processing

Portfolio and Order Management Systems developed 'hybrid' position-keeping approaches, whereby orders and executions were added intraday to the overnight accounting positions, which produced an 'intra-day' position view. This delivered an incomplete and inconsistent mixture of positions based on yesterday's account postings and today's trades.

Even for reconciliation purposes, this slew of data was of little use and would be flushed at the end of the day to be replaced by the latest accounting position snapshot. This process then cycled daily. It’s a low-value, low-integrity approach, but it kept the new Front Office systems away from the mucky detail of incidental transactions.

Early Order Management and compliance software, including Charles River Development, LatentZero and Longview were based on flush/refresh position management. Over time they added other transaction types to their intra-day updates, but completeness was never a coherent objective.

NAV portfolio snapshots vs. batch-based systems: what is the difference?

A position snapshot of any kind is the result of a batch process of some form; the snapshot is produced at one time, and stored away for subsequent use. Such an approach is usually used by accounting systems and accounting books of record (ABOR).

The main limitations to this approach, which makes it suboptimal for use in the Front Office, are:

• The inability to update positions and cash fully and in real-time, leading to difficulties optimising investment decisions.

• The uncertainty of what transactions are included in the start-of-day snapshot. Trade corrections, corporate actions, cash contributions, withdrawals etc. might not be included in the overnight snapshot when it’s loaded into the Portfolio Management or Order Management System. This makes the cash and positions inaccurate when used when, for example, rebalancing a mutual fund portfolio.

• The exclusion of known future events, including corporate actions, income and agreed injections and withdrawals. As a result, either the Order Management Software needs to keep its own record of these and apply them “on top of” the start of day snapshot, or the manager needs to keep records of them outside the core system. These corrections are cumbersome to maintain daily, and hence only the major type of events are usually maintained. Completeness isn’t achieved as a result. 

Event-driven data processing enables live position data extract 

Event-driven data processing means that, instead of processing all daily transactions and postings at end-of-day, each is processed in real-time as it happens. This means live portfolio views are updated immediately - e.g. a preliminary deposit which arrives intraday.

In fact, one of the benefits of a truly event-driven Investment Book of Record is that positions aren’t maintained at all. Past transactions including any amendments, and future expected events, including projected income, corporate actions, injections. withdrawals etc. are what is stored. Positions are calculated from a definitive time series of transactions and other stock and cash events - either live or on request.

This is exactly what is meant by a live extract Investment Book of Record (IBOR). It extracts (or derives) positions live from events (transactions etc.), only when requested. It can also provide live views for any position state, that updates continuously as transactions happen.

Building a system with this kind of capability is technically demanding compared to a batch process approach. The extract process must be highly efficient. However, the benefit to users is well worth it in terms of accurate, timely and complete portfolio data.

How real-time event-driven data processing empowers Asset Management Systems

With the ability to have live portfolios that are complete, accurate and timely, thanks to event-driven data processing carried out in real-time, the Front Office’s capabilities are elevated. 

Data coming from custodians or other service providers can be captured and reflected immediately, and new data sources can be added quickly.

Any system supporting asset management decision-making, such as an Order Management System or Portfolio Management System, becomes more powerful if it is directly integrated, in real-time, to a live extract IBOR. The OMS / PMS doesn’t need more functionality than other systems to be superior, it just needs to exploit the complete, accurate and timely data that it has access to from the IBOR. Managers can rely on this - and have no reason to maintain their own offline records in Excel or elsewhere.

Summary

Any OMS / PMS which relies on overnight batch-based data processing is inevitably limited both in integrity and in the range of use cases that it can support. For effective Front Office capabilities, improved portfolio views and consistent transaction data accuracy, it’s worth leveraging real-time, event-driven data processing and a fully functional, live extract IBOR solution.

Find out more about Limina’s IBOR-based Investment Management Platform and how it helps Asset Managers make the best use of their data, or get in touch with us to see the platform in action.