getHistory occationally returning null - DailyFX Forex Forum

hotdogz919 · #1 (**permalink**) 11-07-2009, 05:14 PM

I'm having an issue when using the getHistory function, and pulling a lot of tick data, 300 ticks at time, occasionally a call will return null. In that case I just try the call again, but it usually happens around 15 times if I'm mining a months worth of tick data. Its not really a big issue, because my work around of just trying again seems to work, but i'm just curious if I'm doing something wrong or if this is normal. here's my code, pretty much straight from the example code...

Code:

 List<Tick> mList;
            List<Tick> lastLoaded = new List<Tick>();
            int ticksLoaded = 0;
            while (true)
            {
                mList = new List<Tick>();
                FXCore.MarketRateEnumAut rates = null;

                try
                {
                    rates = (FXCore.MarketRateEnumAut)core.Desk.GetPriceHistory(instrument, "t1", from, to, 300, true, true);
                }
                catch (Exception)
                {
                    rates = null;
                }

                if (rates == null)
                {
                    //Server returned null, trying again
                }
                else
                {
                    for (int i = rates.Count; i >= 1; i--)
                    {
                        FXCore.MarketRateAut rate = (FXCore.MarketRateAut)rates.Item(i);

                        //check whether rate is in the range
                        if (_to != _NOW && rate.StartDate > to)
                            continue;
                        if (rate.StartDate < from)
                            continue;

                        bool exists = false;

                        for (int j = lastLoaded.Count - 1; j >= 0; j--)
                        {
                            if (rate.StartDate > lastLoaded[j].DateTime)
                            {
                                exists = true;
                                break;
                            }
                            else if (rate.StartDate < lastLoaded[j].DateTime)
                            {
                                break;
                            }
                            if (lastLoaded[j].Equals(rate))
                            {
                                exists = true;
                                break;
                            }
                        }

                        //add a new tick if it does not exists
                        if (!exists)
                        {
                            mList.Add(new Tick(rate));
                            ticksLoaded++;
                        }

                        to = rate.StartDate;
                    }

                    loadTicksIntoDataBase(mList);

                    if (from.CompareTo(to) >= 0 || rates.Count == 0)
                    {
                        break;
                    }

                    to = to.AddSeconds(1);

                    lastLoaded = mList;

                }      
        
                Thread.Sleep(1);
                
            }

fexxxer · #2 (**permalink**) 11-09-2009, 05:01 PM

I've been trying to reproduce this without asking FXCM's developers anything, but so far i haven't received null from .GetPriceHistory() methods.

One thing that immediately catches the eye is this:

Code:

try
{
    rates = (FXCore.MarketRateEnumAut)core.Desk.GetPriceHistory(instrument, "t1", from, to, 300, true, true);
}
catch (Exception)
{
    rates = null;
}

In case of exception you yourself assign "rates" object a null pointer. So in case of exceptions your rates will be null, and that will not be a server response but your own assignment.

I think in case of exceptions you should display, or if not display then atleast log and see which type of exception and message that method throws.

Did i understand you correctly?

hotdogz919 · #3 (**permalink**) 11-09-2009, 07:05 PM

yeah, that makes sense. I guess I'll try again and log the exception. Like I said it works the way I had it, I was just looking for an explanation, but I didn't even think twice that it could be me assigning it to null... the value having a second person look at your own, "you'll never see that missing semi-colon on the 3rd line that's causing a compile issue on the 15th in YOUR own code", hahaha. I'll let you know what the exception is that I was getting.

hotdogz919 · #4 (**permalink**) 12-14-2009, 08:30 AM

I finally got around to looking at this issue again and found that the exception that it's throwing is a timeout exception, which would explain why just trying it again would work. Is this common and or should I be concerned that something else might be going on?

Luke Quinn · #5 (**permalink**) 12-14-2009, 08:40 AM

I've seen this. It does not really come up when displaying a chart, but if you are doing tick data for extended periods, your request may time out occasionally. I also see timeout's occur much more often from a home pc as opposed to my workstation here in the NY office. I think as long as you manage them appropriately, I would not be concerned.

fexxxer · #6 (**permalink**) 12-14-2009, 01:19 PM

Quote:

Originally Posted by hotdogz919

I finally got around to looking at this issue again and found that the exception that it's throwing is a timeout exception, which would explain why just trying it again would work. Is this common and or should I be concerned that something else might be going on?

So .GetPriceHistory() method is the one throwing Timeout Exception? You code shows that you request history continuously within the infinite loop.

What you can try is setting a bigger timeout. You can just take your current one and make it 2 times longer.

Code:

long milliseconds = tradeDesk.GetTimeout(tradeDesk.TIMEOUT_PRICEHISTORY);
Console.WriteLine("My current .GetPriceHistory() time out: {0}", milliseconds);
tradeDesk.SetTimeout(tradeDesk.TIMEOUT_PRICEHISTORY, milliseconds * 2);

Try to handle the exception type by making your application wait little bit and try getting rates again.

What you think?

Fexxxer

hotdogz919 · #7 (**permalink**) 12-14-2009, 03:19 PM

Yes, getPriceHistory is the one throwing the com exception with the time out message. I'll try increasing my timeout and sleep length tomorrow probably. I'm in the middle of a relatively long data pull right now. Its been a lot slower than normal today, and a lot of timeouts, 63 so far today. I've been running for about 6 hours now and I've only gotten about 2 months worth of tick data. It'll go through periods of relatively quick response times, about a half a second a call, and then some periods of lengthy response times, 5 to 10 seconds, or even a timeout. Its pretty inconsistent right now, usually its not too bad. I definitely have had faster pulls in times past, but then again I usually do it at night where there is probably less traffic on the FXCM servers.

hotdogz919 · #8 (**permalink**) 12-17-2009, 08:09 AM

How far back does FXCM provide tick data for? I tried to do a pull tick data last night for EURUSD over the entire year of 2007, and it it stopped at March 27th, and I've stopped and started it several times now, and I can pull data from any later time period, but anything before March 27th, 2007 does not return any data.

Luke Quinn · #9 (**permalink**) 12-17-2009, 09:09 AM

Quote:

Originally Posted by hotdogz919

How far back does FXCM provide tick data for? I tried to do a pull tick data last night for EURUSD over the entire year of 2007, and it it stopped at March 27th, and I've stopped and started it several times now, and I can pull data from any later time period, but anything before March 27th, 2007 does not return any data.

Hotdogz919,

I've recently pulled data going back almost 10 years for EURUSD. Are you getting an error message? whats the last data you can pull?

hotdogz919 · #10 (**permalink**) 12-17-2009, 12:57 PM

It broke last night around 3am est(left it running while I was sleeping), then I tried it for an hour this morning, and then again about 5 minutes ago, getting the same result every time. I was in the middle of a pull from January 1st 2008 at 5pm, to January 1st 2007 at 5pm. It got to March 27th, at 7:20.38 am, and then jumped to February 7th at 10:23.00 am, and kept returning the same 1 tick result result over and over using .GetPriceHistory() between february 7th, 2007 and January 1st 2007 (jumped from march to february). See the attached screen shots. Issues with price server or something on my end do you think? You can look back at earlier in the post for the complete code I used to loop through and pull the data. Its basically straight from the o2g example code.

Luke Quinn · #11 (**permalink**) 12-17-2009, 01:56 PM

Hotdogz919,

I was checking my previous projects. I had pulled 1min data going back almost 10 years, not tick. I just tried to pull tick data for 2006 myself and was not able to. It looks like the issue is not your code. That said, right around the beginning of 2007 is when we transitioned to No Dealing Desk for the majority of our clients, so tick data before that may not be appropriate when comparing to today's price action and execution.

fexxxer · #12 (**permalink**) 12-18-2009, 01:54 PM

Quote:

Originally Posted by hotdogz919

It broke last night around 3am est(left it running while I was sleeping), then I tried it for an hour this morning, and then again about 5 minutes ago, getting the same result every time. I was in the middle of a pull from January 1st 2008 at 5pm, to January 1st 2007 at 5pm. It got to March 27th, at 7:20.38 am, and then jumped to February 7th at 10:23.00 am, and kept returning the same 1 tick result result over and over using .GetPriceHistory() between february 7th, 2007 and January 1st 2007 (jumped from march to february). See the attached screen shots. Issues with price server or something on my end do you think? You can look back at earlier in the post for the complete code I used to loop through and pull the data. Its basically straight from the o2g example code.

I had that problem, dateTimeFrom variable didn't do that i asked it to do when i was getting all 300 rates, it was returning me only 1 rate object in the array. So i decided to use only 1 date/time, is the one it needs to count backwards from, just bookmark it basically.

I have a small benchmarking app that i wrote for no apparent reason =)

Here is the loop when it gets ticks and keeps going backwards. Pass 0 to date time FROM which is not needed at this point 'cause you getting everything from certain date/time and backwards.

To pass a 0 to the method variable you do this -> DateTime.FromOADate(0)

Here is the part from my app, but code a Thread.Sleep([millis]) somewhere if you can, or it will put serious stress on the server. I deleted all unnecessary parts and left essential ones:

Code:

// or DateTime.Now but for UTC variant of the method, you'll need to calculate the difference between timezone you're in and UTC one.
DateTime dateBookmark = dateTimeTo;

// For me i needed certain amount of cycles but you can use while(true) loop
for( int i=1; i <= cycles ; i++)
            {
                 FXCore.MarketRateEnumAut rates = (FXCore.MarketRateEnumAut)tradeDesk.GetPriceHistoryUTC("EUR/USD", period, DateTime.FromOADate(0), dateBookmark, 300, true, true);
                
                // BOOKMARK "LAST" DATETIME ON THE LIST - IT IS FIRST IN THE ARRAY
                dateBookmark = ((FXCore.MarketRateAut)rates.Item(1)).StartDate;

                Console.WriteLine("{0,2}. Got price history array #{0,2}. Prices Contained: {3,3}. From: {2}. Execution seconds: {1}.", i, execSingle.TotalSeconds.ToString(), dateBookmark.ToString(), rates.Count);
                 
                // put a thead sleep and rates.Count check
                // cause if you use while(true) loop you might get stuck at this part
                // with 1 rate getting itself over and over again
                Thread.Sleep(2000); // 2 seconds
                 if( rates.Count <= 1 ) // if 1 rate then break the loop
                              break;

      }

Try it out, let me know if it helps any. My app worked fine when taking breaks between loop cycles with Thread.Sleep() otherwise it got really slow.

EDIT: Of course don't forget to add Exception handling, this is just to give an idea of what i'm talking about.

Fexxxer