Thread: API poorly suited for fully-automated position management

This is more than just a rant, this is an honest effort to provide the API team feedback from a real world, fully-automated trading application perspective.

My experiences are based on porting a working, fully-automated Forex trading java application from Gain Capital's API (forex.com) to FXCM's API. The reason I had to switch was because Gain Capital's API is currently not in compliance with NFA reporting requirements (according to Gain), I figured the FIX protocol is "standards" based which would give me more options in the future regarding brokers, and my current partners are working with FXCM on the money management side. I'm writing this post to share some of the encountered pitfalls and solutions, and to vent a bit as it has been extremely frustrating dealing with the multitude of complexities induced by the FXCM architecture in a fully-automated app.

My overview:

• 20+ years of programming
• Intimately familiar with Forex trading concepts and best practices
• Extensive track record in building highly available, data-based web applications
• All things considered, I may be missing something and don't claim to have all the answers as I've only been working with the FXCM API for a couple of weeks (with a good portion of that being demo account testing time)
• That said, I think I have a fair understanding of how the FXCM API was architected by converting my app to use it

Application overview:

• Typical java-based web application
• Deploys and runs in JBoss
• Back-end DB is MySQL
• Leverages the Spring framework extensively (database abstraction, service configurations, makes it easier to test the code, etc.)
  
• Uses the FXCM API to:
• Collect and analyze rates historically and in real-time... works great, straightforward
• Execute trades and place stop orders automatically... works good, straightforward
• Manage the trade life-cycle (close trades, confirm stops, execute reversals, etc.)... works, but there are gaps and it is difficult to guarantee position state for all use-cases

First, the positives:
The FXCM API seems to be stable, relatively fast, and works as advertised. It seems to be based purely on a publish/subscribe (pub/sub) architecture, meaning that once you initiate a request, you must wait (listen) for the FXCM API to publish its response by subscribing a listener to that response. As demonstrated in the JavaFixTrader example, this is fine for sequential operations where there are no requirements to do anything other than wait for the appropriate response before proceeding with the next operation. The advantages of a pub/sub architecture are in reducing resource overhead when addressing frequent requests for the same data. This type of approach is suitable for a User Interface application such as FXCM's Trading Station (just guessing that their own app is leveraging their own API based on how it behaves). This type of architecture is also good for things like rate updates and streaming data requests. It falls short, however, when dealing with transactional requests, in my opinion.

The rant:
Here is a pseudo formula attempting to express why the FXCM API architecture falls short for transactional requests:

( [(opening / closing trades and stops) + (ensuring that the trade or stop executed properly)] * (multiple currencies) * (potentially multiple accounts) * (probability of multiple events occurring simultaneously, i.e. event collisions) ) ^(ensuring continuity if something goes wrong) ==
(extreme frustration with a pub/sub architecture)

Maintaining the state of pending versus open versus stopped out trades is kind of a pain. To make it a bit easier to debug / step through the code, I decided to maintain separate maps of pending versus active trades. When the app generates a trade event, the event is queued in pending trades by currency pair and a UUID (unique identifier). Why the UUID? Well unlike the JavaFixTrader example which only facilitates one operation at a time, my app must support the possibility that multiple trade events for multiple currencies could occur simultaneously. The UUID allows me to tag the orders' (market and stop) customId with a token that my app knows about beforehand. This allows me to definitively determine if a position request was executed properly when the listener receives the response. This also provides a solution in the unlikely, yet potential, scenario that multiple events for the same currency "stack up" due to network latency / short network outages. If everything goes ok, then the pending queue is cleared for the currency and the event moves to the active queue. The active queue makes it easier for me to deal with closed position reports and to avoid collisions for new pending trades. This was driven by my design to have the "transactional" part of the application inside the listeners to ensure that the system didn't assume that a trade occurred until it was absolutely confirmed. Since the transactional behavior was built into the listener chain (instead of the request operation) it induced additional complexity in managing multiple trade events across multiple currencies, and especially in managing reversal trades.

Another reason I chose the UUID approach was in the case of reversal trades. Because of the asynchronous non-guaranteed ordering of responses from the API, there is a chance that I could get an execution report for the reversal before the existing position was closed. For now, I've built in some overly-complicated logic where the reversal trade is executed by maintaining a workflow state through each step of the reversal trade event (check to see if existing position is already closed -> if not, then close existing trade (otherwise this will open a new position) -> make sure it and its stop closed -> initiate a new trade (the reversal) with a new stop at the desired lot amount -> make sure the trade and stop were created). While this seems to be functional, I still have yet to determine the amount of "slippage" from my expected entry for the newly reversed position as everything is being executed discretely with request/wait/request/wait/etc. With Gain, a reversing a position was as simple as closing the existing position for twice the lot amount with a new stop. The problem with FXCM's API is that there is no guarantee that you've received the stop notification for a given position that you want to reverse at the time you want to reverse it, and you could wind up with either a) no position, or b) a position with twice the lot volume that you wanted. I may not be describing it well, but it is kinda a chicken versus egg scenario with the pub/sub architecture.

At the end of the day, I wound up implementing a pessimistic design, and punted on trying to automatically cover all the use cases (at this time)... eventually I will. For now, I created a messaging decorator in my DB event log which I can "scrape" with monitoring software (i.e. Zabbix, Nagios) to send an alert to a cell phone, email, etc. so that we can manually go in and rectify trades if needed. This sucks because I believe there are significant benefits in having a 100% human-less Forex trading system, but at this point I can't see any other way with the current FXCM API and it's asynchronous, non-guaranteed (from a timing standpoint) messaging architecture.

Even though Gain's API was a very old, poorly documented, request/response, mostly-stable, SOAP-based API... it was fairly simple to address the factorial of cases expressed above by simply making a request, receiving the response, and retrying for a reasonable number of intervals if something went wrong (i.e. network connectivity, etc.). If the operation failed, it was simple to ensure the application state was consistent and to rollback any associated events. Everything worked great and required no human intervention. Ironically, Gain used a pub/sub architecture for their rate feed which worked fine. Their transactional API was a separate URL.

The proposal:
Add a request/response based architecture option for all transactional requests (opening / closing trades, limits, and stops; position update reports). The amount of additional overhead induced by supporting these requests individually is significantly lower than the overhead associated with serving up rates to all your users. This wheel has been invented already... pick a current protocol and run with it - checkout the Spring Framework for some simple RPC examples and implementations. Don't forget to make sure to keep the NFA happy at the same time.

Here are some specific use cases for transactional requests:

• Allow a simple position request / report on-demand per currency, per account
• Execute a new BUY or SELL for a single currency on a single account, allow relative stop (and/or limit) order, return a unique ID which is associated with the event
• Execute a reversal on an existing position for a single currency on a single account, auto-cancel existing orders, allow relative stop (and/or limit) order, return a unique ID which is associated with the event
• Execute a close event on an existing position, auto-cancel existing orders
• All of the above for multiple accounts
• For all of the above, the return object should contain all confirmations, codes, and messages associated with the transaction

In a request / response scenario, all of the above requests could be wrapped in a try..catch block and the application could easily rollback any dependent transactions to maintain state and ensure data integrity.

I feel a little better now... back to coding.

tman-az

Thank you for your feedback and the detail. This information was reported to managers. Please let us know if you have any additional questions or comments on our APIs.

Regards,

Andrew Bruce