Screening alert search engine: a better way to access alerts

If your business deals with money, sooner or later you will start looking for an AML software solution to manage your everyday needs. However, it can be a pain to normalise all customer and transaction data fields just for your AML software provider. It’s a painstakingly long and frustrating process that can slow you down rather than speed you up. Early on, we at Salv decided to keep everything “loose” and easy to customise, and save our customers headaches in the long run.

We fit to your needs as much as possible, not the other way around. Usually, we agree on what your customer and transaction data looks like in the integration phase, then you send it to us, and we make the magic happen. But what exactly happens once we’ve received your data and generated the alerts? How do we navigate through the rough seas of the screening alerts?

The answer is simple: screening alert search engine. It’s like a pirate map that marks the location of a hidden treasure, in our case, screening alerts, making them easy to locate and access. But the screening alert search engine didn’t magically materialise overnight. In fact, it all started with a screening alert backlog.

Screening alert backlog: how it worked

In the summer of 2021, some of our customers were growing really fast, which meant the amount of customer and transaction data we had to process was increasing as well. At that time, we had a simple backlog to navigate through screening alerts. Customers could send us any of the following entities:

1. Their own customers who they wanted to “watch”.
2. Transactions between their customers and internal/external counterparties.

Did you know that Salv can monitor your customers’ counterparties?

In the above cases, we are talking about an unlimited number of fields with any kind of keys and datatypes. You can think of them as arbitrary JSON BLOBs with 5-6 agreed fields, as well as over 50 fields of arbitrary metadata. The number of alerts we created at the time didn’t require a sophisticated solution. A simple screening alert backlog with two filters was enough. The filters included:

• Transaction and customer alerts
• Alerts with a specific status (“NEW” for example)

Screening alert backlog: issues

With time, things changed. Some of our customers grew out of a minimalistic screening alert backlog, which, at that stage, hindered rapid growth rather than improved it. Our customers wanted to:

1. Divide alerts based on certain properties, divide alerts into different backlogs, assign alerts to multiple compliance officers.
2. Inspect already solved alerts that match some specific criteria.

To help our customers overcome their woes, we decided to build a powerful screening alert search engine. Before we move any further, you may want to know how screening works at Salv.

Building the screening alert search engine

An obvious first thought was to leverage an existing database engine. Luckily, we had two databases already on hand: PostgreSQL and Elasticsearch. The best place to start was to examine their pros and cons.

PostgreSQL
✅ JSON blobs were already there
✅ Good JSON querying possibilities
❌ Our use cases were way too complex to index them to perform efficiently. Some of them were so complex that we couldn’t build working proof of concept queries at all.

Elasticsearch
✅ Meant to search data in large datasets with unfixed schemas
❌ We’d have to migrate the data just for the search
❌ Elasticsearch is the core of screening at Salv. Synchronising all the data to Elasticsearch and at same time performing searches on this data could affect screening performance substantially.

This quick breakdown helped us realise that Elasticsearch wasn’t our best option, so we went along with PostgreSQL. It was a good time to challenge our misconceptions about screening alert search engines and rethink our whole approach. For example, we realised searching for screening alerts wouldn’t always produce quick results.

You probably want to ask why. When a search is set up for alert resolving then it’s expected that the new alerts emerge seconds after an object has been screened. It’s a totally different story when you want to pull someone’s historical alerts for an audit. In that case, you can expect the query to take up to an hour.

It was time to move to the next step. 👇

Screening alert search engine logic

Based on the information above, we came up with the following logic behind the screening alert search engine:

1. Search criteria. A search criteria is created and saved to the database.
2. Background search. An asynchronous background worker goes through a wide set of alerts by pulling them to memory in batches and checking them against the saved criteria.
3. Search match. Each time a match is found we save a pointer to this match to a separate table called alert_search_results.
4. Search results. If the user checks for the results we just query the table alert_search_results without any realtime search and show the results based on this.
5. New alerts. Every time a new alert is created we check it against all the pre-existing search criteria and see if it matches any. If it does, then again we add the pointer to alert_search_results.
6. Handling changes. Every time an alert or its related transaction/customer entities are changed we check if it fits any of the criteria and update the pointers in alert_search_results accordingly.

So that’s what we did. We already had a very nice event streaming mechanism in place so it could easily be implemented as a separate component. This also meant that it didn’t affect the system’s performance on the global scale.

Since the start of 2022, we’ve implemented a very powerful screening alert search engine. It’s already used by most of our customers, and we are slowly enabling it for the rest of them. From the users’ perspective, it satisfies all their initial requirements, and even more. The screening alert search engine works almost like a real-time search. We made it possible without adding anything to our stack, while keeping the burden to the core system as light as possible.

We have more exciting product features in stock. Read about this data upload case study.

Wrapping up

AML compliance can be tricky. You want to be compliant and catch criminals, but at some point you realise that you don’t have in-house resources to deal with a growing number of customers and transactions. That’s when you start looking for an AML software solution to manage your everyday compliance needs and protect your customers. Your business evolves and so do the criminal schemes. Book a demo today to strengthen your AML defences and fight crime.

And lastly, as requested by our product engineers: we are always looking for great engineers! Read about my experience at Salv and pay a visit to our Careers page. 🤗