Even banks don't know their own positions (IT systems)

[Ace Pace]

A recent article by The Economist has a rather scary excerpt.

The demands of financial markets make matters worse. Hedging positions, trading derivatives and modelling financial products all require highly sophisticated programs that are only really suited to specific asset classes. The code for new financial products has to be developed quickly. Innovation often takes place on Excel spreadsheets on traders’ desktops. “The big task of management is to manage down the number of spreadsheets,” says one risk chief, whose bank creates 1,000 product variations a year.


As a result, many banks have huge problems with data quality. The same types of asset are often defined differently in different programs. Numbers do not always add up. Managers from different departments do not trust each other’s figures. Finding one’s way through all these systems is detective work, says a former IT manager at a big British bank. “And sometimes the trail would go cold.”

This fragmented IT landscape made it exceedingly difficult to track a bank’s overall risk exposure before and during the crisis. Mainly as a result of the Basel 2 capital accords, many banks had put in new systems to calculate their aggregate exposure. Royal Bank of Scotland (RBS) spent more than $100m to comply with Basel 2. But in most cases the aggregate risk was only calculated once a day and some figures were not worth the pixels they were made of.

During the turmoil many banks had to carry out big fact-finding missions to see where they stood. “Answering such questions as ‘What is my exposure to this counterparty?’ should take minutes. But it often took hours, if not days,” says Peyman Mestchian, managing partner at Chartis Research, an advisory firm. Insiders at Lehman Brothers say its European arm lacked an integrated picture of its risk position in the days running up to its demise.

So lets hype the picture a bit, even banks are not even sure of their own numbers because systems are fragmented to a point we're running things based on some bankers Excel spreadsheet....

[ArmorPierce]

As a result, many banks have huge problems with data quality. The same types of asset are often defined differently in different programs. Numbers do not always add up. Managers from different departments do not trust each other’s figures. Finding one’s way through all these systems is detective work, says a former IT manager at a big British bank. “And sometimes the trail would go cold.”

I am currently working at a pretty major financial services firm and I can personally attest to this. The major problems here are empasis of speed over quality, over-competitiveness in order to look good to higher ups, and shifting blame downwards.

[J]

Even the Federal Reserve itself has misplaced $9 trillion, that's trillion with a capital T. With the proprietary high frequency trading algorithms which the financial firms employ, I can't say I'm shocked that they've lost track of god knows how much money in their transactions. 100 million trades a day done with some "black box" program, yup, that's asking for all kinds of wonky results.

[Alyeska]

I work in Banking IT and I can tell you this is very much true.

[Grif]

So, where does all the misplaced money go to? It has to end up somewhere.

[Academia Nut]

So, where does all the misplaced money go to? It has to end up somewhere.

You're assuming it was anywhere to begin with in the first place. Such large sums going missing would pretty much have to all be credit of some form or another, but I'm doubtful that all of it, or even the majority of it, actually has assets attached to that value, and if they do, I doubt the majority of said assets are actually worth what they are being valued at.

[Dahak]

From my experience, this is close to reality. I only work in risk management environments for consumer credits, but Excel is the tool of choice for most bankers, and sometimes the business users don't have defined ideas what they calculate (it's usually one person who has or had that knowledge. Talk about single points of failure...).

[Steel]

So, where does all the misplaced money go to? It has to end up somewhere.

You're assuming it was anywhere to begin with in the first place. Such large sums going missing would pretty much have to all be credit of some form or another, but I'm doubtful that all of it, or even the majority of it, actually has assets attached to that value, and if they do, I doubt the majority of said assets are actually worth what they are being valued at.

Lets think about that: Credit is not free money that pops out of nowhere. Credit is someone else's money that they have given you. When you receive credit you also get a debt to the person lending you the money so your net position is no change, but you have a wad of cash + some debt. If you suddenly lose your money it doesn't matter that you've got it in the form of credit; you still have the debt on the books and your creditor is still missing their money.

[Spyder]

Seen this before with NZ Inland Revenue. They have centralised managed systems for doing this sort of thing but some department is always going to suddenly decide that putting a spreadsheet together is a good idea.

Those and Access databases too. At one point they had more Access databases then employees.

[Mad]

You're assuming it was anywhere to begin with in the first place. Such large sums going missing would pretty much have to all be credit of some form or another, but I'm doubtful that all of it, or even the majority of it, actually has assets attached to that value, and if they do, I doubt the majority of said assets are actually worth what they are being valued at.

Lets think about that: Credit is not free money that pops out of nowhere. Credit is someone else's money that they have given you. When you receive credit you also get a debt to the person lending you the money so your net position is no change, but you have a wad of cash + some debt. If you suddenly lose your money it doesn't matter that you've got it in the form of credit; you still have the debt on the books and your creditor is still missing their money.

I think he meant credit given, since that can be listed as an asset in some form or another.

Some of these spreadsheets may have entered loans as something along the lines of an asset of loan amount $x (and maybe even expected interest earned $y). So in those calculations, the bank appeared to have a lot of money. Unless the something happened and the loan wasn't paid and the property lost its inflated price when the bank got it back. But, hey, how likely is that?

[Alyeska]

Money isn't lost so much as misreported by certain systems. We handle our core banking using an AS400. Everyone's account is handled using the AS400. Checking, savings, whatever. But when it comes to real-estate, loans, and other systems, things get a little weird. Different programs that run a little differently and they don't necessarily talk with each other very well. The decisions made by the company are based on data people collect with these programs and how we work with our customers. We can tell you exactly how much money you have in your checking account, but when it comes to just how your loan works, well its completely dependent on the people who entered it into the system and maintain its information.

It gets even more interesting when the AS400 is down for various reasons. We have fall back systems, but none of them networked. And if god forbid the computers as a whole go down, people are still allowed to bank while the bank conducts everything on pen and paper for that period of time. Meaning you can walk into a bank with its lights on and make a withdrawal from your savings account and not realize they handled the entire transaction by paper because the system was down. If you over drew, they won't know until the system is back up and they input the transaction. Yes, its possible to cheat the system. But you would have to know the systems are down in the first place.

Real-estate and other systems are a little more complicated and they aren't run when the systems are down. But core banking is always active short of a power outage.

[rapidsquirrel]

Money isn't lost so much as misreported by certain systems. We handle our core banking using an AS400. Everyone's account is handled using the AS400. Checking, savings, whatever. But when it comes to real-estate, loans, and other systems, things get a little weird. Different programs that run a little differently and they don't necessarily talk with each other very well. The decisions made by the company are based on data people collect with these programs and how we work with our customers. We can tell you exactly how much money you have in your checking account, but when it comes to just how your loan works, well its completely dependent on the people who entered it into the system and maintain its information.

It gets even more interesting when the AS400 is down for various reasons. We have fall back systems, but none of them networked. And if god forbid the computers as a whole go down, people are still allowed to bank while the bank conducts everything on pen and paper for that period of time. Meaning you can walk into a bank with its lights on and make a withdrawal from your savings account and not realize they handled the entire transaction by paper because the system was down. If you over drew, they won't know until the system is back up and they input the transaction. Yes, its possible to cheat the system. But you would have to know the systems are down in the first place.

Real-estate and other systems are a little more complicated and they aren't run when the systems are down. But core banking is always active short of a power outage.

The core server (a creaky old IBM eServer run AIX) at the bank that I work at doesn't even go down for power outages. In fact, the teller computers are on UPS's and are part of the bank that can be switched over to generator. That being said, we still have a rather long paper trail for anything gets entered into the core system. Everything the tellers do is put on paper and later fed though a scanner and uploaded to the core software, so we could function if we had a catastrophic failure, at the very least until our 4 hour service plans kicked in and the things were fixed or replaced.

As for the stuff that's not part of the banking software that we use, I hate to think of the crazy number of spread sheets or horrible attempts at access databases that people have setup and we are a small bank.

[Solauren]

RapidSquirrel...

Someone in charge of finance institutions is letting people build Access Databases for that. People that don't know what they are doing in Access?

Ye Olde Gods.....

[Elessar]

RapidSquirrel...

Someone in charge of finance institutions is letting people build Access Databases for that. People that don't know what they are doing in Access?

Ye Olde Gods.....

Actually, there are numerous investment firms that still use primarily a paper-based solution for all their workflows. Access is rather advanced, but the firms aren't 'letting' their Financial Advisors (or more likely, their assistants) juggle their numbers in Excel or Access. It's just that the firm does not strictly control the tools that their FAs are using.

Keep in mind that in a variety of these institutions, Firm-level control is basically only at the bottom line. In many ways, Advisors are tiny little companies themselves, they have their own book of clients, they have their own running prospect lists, and they have their own business strategy. Firms typically provide competitive advantages, such as financial analysis, cutting edge tools to track all sorts of data, and a brand name.

It also depends on the type of financial institution that would result in different levels of compliance being followed. Audit trails are typically quite extensive, just time-consuming to backtrack and follow. As Alyeska pointed out, the various systems are rarely integrated, or integrated poorly, and so this kind of data just becomes costly to retrieve.

So it's not that they don't know their own positions. Just like when someone in IT tells you that something is impossible, it's not actually true. It's just really, really annoyingly difficult to do and would require far more time than it's deemed valuable.

[The Kernel]

I design these systems for a living and I can tell you that all of these shitty systems are the direct result of poor technology management at financial institutions.

My company for example maintains a stored value accounting system, a debit network switch and a prepaid card platform all tied into a shared series of web/mobile applications and one (count them ONE) backoffice admin tools to power the whole thing. And this isn't just a CSR admin tool either, it offers full velocities control, tiered fees, manual account adjustments (dual approval), and a fraud detection package that we built from scratch. This is on top of all the third party integrations for things like ID checks, ACH, Credit/Debit processing, etc. And we built this system largely with 1/10th of the resources that a Tier 1 bank has access to for software development.

I know so called technology people at several tier 1 banks and can tell you without any exaggeration that this people have no vision and no desire to try to bring order to the massive amount of fragmentation out there. It's gotten so bad that the US GOVERNMENT had to be the one to force banks to move away from utterly ARCANE check processing using manual methods and to an all electronic platform (Check 21). That's right people, the US Government managed to innovate faster than the banks.

[Tellos]

Hello everybody new to the fourms so look forward to getting to know you all. Well as mentioned by ArmorPierce alot of focus now a days in banking and I find in many work places is speed or quanity over quality. This leads often to errors that later come back to bite and then everybody wants to know why their not informed ahead of time of these errors. In my experiance (note this isint saying it happends in all places or is even the norm just a set of my own experiances) my bosses often seem to have been told by me or others or already knew but ignored these problems until they become problems and prefer manage by crisis forms of running a buisness. I beleive we really need to move to something more like a hybrid of computer and human checked systems with standardized methods of calculating the value of assets and that managment should do all it can to discourage others from "making their own" calulation methods. This I think might lead to some of that money not being lost however i as yet have no good model for how this would work it's just a general idea reguarding a better maybe more stable method figuring out our money and moving away from the "Get it done faster" mentality which while it has merits if fallowed to the exlusion of all else casues us as we see in the 9trillion figure provided by J.

[phongn]

I design these systems for a living and I can tell you that all of these shitty systems are the direct result of poor technology management at financial institutions.

Most big institutions don't seem to have any desire to properly use technology. They want a product they can buy off the shelf and magically solve all their problems or their CIO doesn't want to rock the boat.

[Tellos]

I am sorry if I bothered others with my last post which had alot of errors there are reasons for this but here is the post rewritten and fixed of all the errors I could locate.

Hello everybody new to the forums so look forward to getting to know you all. Well as mentioned by ArmorPierce a lot of focus now a days in banking and I find in many work places is speed or quantity over quality. This leads often to errors that later come back to bite and then everybody wants to know why their not informed ahead of time of these errors. In my experience (note this isn’t saying it happens in all places or is even the norm just a set of my own experiences) my bosses often seem to have been told by me or others or already knew but ignored these problems until they become problems and prefer manage by crisis forms of running a business. I believe we really need to move to something more like a hybrid of computer and human checked systems with standardized methods of calculating the value of assets and that management should do all it can to discourage others from "making their own" calculation methods. This I think might lead to some of that money not being lost however I as yet have no good model for how this would work it's just a general idea regarding a better maybe more stable method figuring out our money and moving away from the "Get it done faster" mentality which while it has merits if fallowed to the exclusion of all else the result is often things like the 9 trillion figure in lost money given to us by J.