Thread: Development Example: Yesterday's Close Value

Posted by Nikolay Gekht

Note: This article uses the latest API version, so the indicator developed will work only in the August 27 2010 release version. Please read notes at the end of the article to know how to make the indicator working in the current production release. I will remove this note as soon as the current beta version comes to the production.

Note: This point is, probably, the most complex in the indicator development. So, if you can handle this, there is nothing you cannot do.

Sometimes an indicator requires data of a bigger time frame, for example, we could use for close of the previous day for calculations. However, it’s possible to try and find this data in the price history the indicator is applied to, but… Just imagine. To get yesterday’s high/low, the whole yesterday’s data must be loaded. If the current history is a minute history, in the worst case we have to have at least 2880 bars to be able find that data (1440 minute bars for today and then 1440 minute bars for the whole yesterday). This is a huge amount to load and then to calculate. But the system has access to the day data, so why don’t we use it? Let’s do it.

However, there are a number of points we should handle carefully. Ok. Let’s develop an indicator which shows yesterday’s close price on the chart and discuss each point in detail.

Point 1. What is “yesterday”?

The handling of the date in indicators is pretty simple. A date value is a number, which is just a number of days past midnight of Dec, 30 1899. The format is also known as Windows Ole Automation date format. The integer part of this number is a number of days and the fractional part is a port of the day. So, to find a calendar midnight date and time, we must just floor the value (cut away the fractional part). To shift the date to N days ago, we must just subtract N from the date. So, if FXCM used the “calendar clock” for the trading days (i.e. the trading day starts at midnight), the finding of yesterday for bar x of the source would be pretty simple: math.floor(source:date(x)) - 1.

Unfortunately, FXCM’s trading day does not match the calendar trading day. Currently, the trading day starts at 17:00EST yesterday and lasts until 17:00EST today. And for different configurations this date can differ. Hopefully, the indicator core helps us to handle calculation of the candle borders for a particular date. There is a core.getcandle function which returns the date and time of the candle of the specified time frame to which the specified date and time belong. As you can see, this function requires two parameters which let us “tune” this function for a certain configuration: an offset of the trading day and an offset of the trading week against calendar. You must use host:execute("getTradingDayOffset") and
host:execute("getTradingWeekOffset") host’s calls to get the settings of the trading day and trading week for particular host application and connection.

So, the function to get yesterday’s date properly can look as follows:

Code:

Code: Select all
function getYesterday(date)
    local offset;
    local weekoffset;

    offset = core.host:execute("getTradingDayOffset");
    weekoffset = core.host:execute("getTradingWeekOffset");

    local today, yesterday;
    today = core.getcandle("D1", date, offset, weekoffset);
    yesterday = today - 1;
    return yesterday;
end

Good? Not yet. There is another point we should handle. There are “non-trading” periods when neither trading is possible nor prices are received. So, in fact, during a week only Monday trough Friday candles exist. There are no “Saturday” candles at all. So, when our minute belongs to a Monday day candle, the function will return us a Sunday candle which just does not exist! So, for Monday’s data we must take previous Friday’s close price.

To handle that correctly, there is another useful function: core.isnontrading which checks whether the date belongs to a non-trading period (Friday 17:00EST-Sunday 17:00EST) and returns the date and time when this non-trading period is finished. So, to find the proper “yesterday” candle we have to check whether the calendar yesterday falls into non-trading period and shift the “yesterday” date into past to skip this non-trading period.

Let’s modify our function a bit:

Code:

Code: Select all
function getYesterday(date)
    local offset;
    local weekoffset;

    offset = core.host:execute("getTradingDayOffset");
    weekoffset = core.host:execute("getTradingWeekOffset");

    local today, yesterday;
    today = core.getcandle("D1", date, offset, weekoffset);
    yesterday = today - 1;

    local nontrading, nontradingend;
    nontrading, nontradingend = core.isnontrading(yesterday, offset);
    if nontrading then
        -- if the calendar yesterday date is non-trading (Friday 17:00 EST - Sunday 17:00 EST)
        -- shift three days back against the non-trading period end (Sunday 17:00 EST).
        yesterday = nontradingend - 3;
    end

    return yesterday;
end

So, now, having the getYesterday() function, we can easily find how many days we need. The oldest day is getYesterday(source:date(source:first())), i.e. the “yesterday” bar which corresponds to the oldest available bar. And what about the newest one? There are two cases. The source collection could be either a historical (i.e. a collection loaded until the specified date, which will be never changed for the chart window) or “alive” (i.e. loaded till “now” and continuously growing as new bars come). According to these two cases, the “to” date could be either a “yesterday” day of the latest available bar (getYesterday(source:date(source:size() - 1)) or … now as well. To specify “now”, we use 0 value.

Point 2. How to get the day price data?

Ok, now we know which data we need. The next step is to get this data. Marketscope provides a host interface which lets you execute an application-specific functions. One of these functions is “getHistory”. By this call Marketscope loads the specified instrument in the specified time frame and in the specified time range. See host:execute(“getHistory”, …) for details.

Code:

Code: Select all
local days;     -- a variable to keep days

-- the function loads the proper day history
function loadData()
    if source:size() < source:first() then
        return ;
    end

    local from, to;
    -- load the data from the date of proper yesterday for
    -- the first available bar of the source
    from = getYesterday(source:date(source:first()));
    -- load to "now" is the source is up to "now" or
    -- to the day which corresponds to the latest
    -- day of the collection.
    if source:isAlive() then
        to = 0;
    else
        to = getYesterday(source:date(source:size() – 1));
    end
    days = host:execute("getHistory", 1, source:instrument(), "D1", from, to, source:isBid());
end

It is important that after the “getHistory” call, Marketscope only starts to load the data. It takes some time, but Marketscope does not wait until the data is completely loaded to avoid hanging of the indicator. When data loading is finished, Marketscope will notify the indicator by calling the AsyncOperationFinished function. Moreover, Marketscope does not care about the indicator status and will continue calling indicator to be updated.

So, there are two points to do.

First, we must protect the indicator from updating while the data is loaded.

To protect the indicator from being recalculated, we can just return from the update function while data is being loaded. Just add the “loading” global variable so that all functions could “know” that data is being loaded.

Code:

Code: Select all
local days;         -- day source data
local loading;      -- the loading day data flag

function Prepare()
    ...
    days = nil;
    loading = false;
    ...
end

function Update(period, mode)
    if loading then 
        -- do nothing if data is being loaded
        return ;
    end

    if days == nil then
        loadData();
        return ;
    end
    ...
end        

function AsyncOperationFinished(cookie, success, error)
    if cookie == 1 then
        assert(success, error);
        loading = false;
    end
    return 0;
end

function loadData()
    if loading then
        return ;
    end

    local from, to;

    -- load the data from the date of proper yesterday for
    -- the first available bar of the source
    from = getYesterday(source:date[source:first()]);
    -- load to "now" is the source is up to "now" or
    -- to the day which corresponds to the latest
    -- day of the collection.
    if source:isAlive() then
        to = 0;
    else
        to = getYesterday(source:date[source:size() - 1]);
    end
    loading = true;
    days = host:execute("getHistory", 1, source:instrument(), "D1", from, to, source:isBid());
end

Well, now we need to force the indicator update when data is loaded. There is function instance:updateFrom() which forces the indicator to be recalculated from the specified period. Also, when AsyncOperationFinished returns the core.ASYNC_REDRAW value, Marketscope redraws the indicator in the chart window.

So, since we haven’t calculated anything yet, we have to update our indicator right from the oldest data. Just fix AsyncOperationFinished a bit.

Code:

Code: Select all
function AsyncOperationFinished(cookie, success, error)
    if cookie == 1 then
        assert(success, error);
        loading = false;
        instance:updateFrom(source:first());
        return core.ASYNC_REDRAW;
    end
    return 0;
end

Almost finished. The last question is how to get the data. We can calculate the day date. Of course, we can just loop through the day history and find the date, but… It’s a bit long, isn’t it? Please, take into consideraion this article, there is a good recipe there.

So, we can build the first version of our indicator. Let’s do so and try it.

Code:

Code: Select all
function Init()
    indicator:name("Show Yesterday Close");
    indicator:description("The indicator shows yesterday's close value on the chart");
    indicator:requiredSource(core.Tick);
    indicator:type(core.Indicator);

    indicator.parameters:addColor("clr", "Indicator Line Color", "", core.rgb(255, 255, 0));
end

local source;       -- indicator source data
local first;        -- first available source data
local days;         -- day source data
local loading;      -- the loading day data flag
local loadedFrom;   -- the date from which the days was loaded
local SYC;          -- yesterday close output

local host;
local offset;
local weekoffset;


function Prepare()
    local name;

    name = profile:id() .. "(" .. instance.source:name() .. ")";
    instance:name(name);

    source = instance.source;
    first = source:first();
    loading = false;
    days = nil;

    host = core.host;
    offset = host:execute("getTradingDayOffset");
    weekoffset = host:execute("getTradingWeekOffset");

    SYC = instance:addStream("SYC", core.Line, name, "SYC", instance.parameters.clr, first);
end

function Update(period, mode)
    if period < first then
        return ;
    end
    if loading then
        return ;
    end

    -- load the data if there is no day
    if days == nil then
        SYC:setBookmark(1, source:first());
        loadData();
        return ;
    end

    -- try to find the value in the days collection
    local yesterday;
    yesterday = getYesterday(source:date(period));
    if yesterday < loadedFrom then
        SYC:setBookmark(1, period);
        loadData();
        return ;
    end

    local i;
    i = findDateFast(days, yesterday, false);
    if i >= 0 then
        SYC[period] = days.close[i];
    end
end

function AsyncOperationFinished(cookie, success, error)
    if cookie == 1 then
        assert(success, error);
        loading = false;
        local i = SYC:getBookmark(1);
        if i < source:first() then
            i = source:first();
        end
        instance:updateFrom(i);
        return core.ASYNC_REDRAW;
    end
    return 0;
end

-- the function loads the proper day history
function loadData()
    if source:size() < source:first() then
        return ;
    end

    if loading then
        return ;
    end

    local from, to;

    if days == nil then
        -- initial data loading

        -- load the data from the date of proper yesterday for
        -- the first available bar of the source
        from = getYesterday(source:date(source:first()));
        -- load to "now" is the source is up to "now" or
        -- to the day which corresponds to the latest
        -- day of the collection.
        if source:isAlive() then
            to = 0;
        else
            to = getYesterday(source:date(source:size() - 1));
        end
        loading = true;
        loadedFrom = from;
        days = host:execute("getHistory", 1, source:instrument(), "D1", from, to, source:isBid());
    else
        from = getYesterday(source[source:first()]);
        to = days:date(0);
        loading = true;
        loadedFrom = from;
        host:execute("extendHistory", 1, days, from, to);
    end
end

-- returns the beginning of the "yesterday" date for
-- the specified data
function getYesterday(date)
    local today, yesterday;

    -- get beginning of the today's candle and round it up to the second
    today = core.getcandle("D1", date, offset, weekoffset);
    today = math.floor(today * 86400 + 0.5) / 86400;

    -- get calendar yesterday
    yesterday = today - 1;

    local nontrading, nontradingend;
    nontrading, nontradingend = core.isnontrading(yesterday, offset);
    if nontrading then
        -- if the calendar yesterday date is non-trading (Friday 17:00 EST - Sunday 17:00 EST)
        -- shift three days back against the non-trading period end (Sunday 17:00 EST).
        yesterday = nontradingend - 3;
    end
    return yesterday;
end

-- ------------------------------------------------------------------------------
-- Find the specified date in the specified stream
--
-- The function uses the binary search algorithm, so it requires only O(n) = log2(n) operations
-- to find (or to do not find) the value.
--
-- The function compares the date and time rounded to the whole seconds.
--
-- Parameters:
-- stream       The price stream to find the date in
-- date         Date and time value to be found
-- precise      The search mode
--              In case the value of the parameter is true, the function
--              Searches for the the exact date and returns not found in the
--              date is not found.
--              In case the value of the parameter is false, the function
--              returns the period with the biggest time value which is smaller
--              than the value of the date parameter.
-- Returns:
-- < 0          The value is not found
-- >= 0         The index of the the period in the stream
-- ----------------------------------------------------------------------------------
function findDateFast(stream, date, precise)
    local datesec = nil;
    local periodsec = nil;
    local min, max, mid;

    datesec = math.floor(date * 86400 + 0.5)

    min = 0;
    max = stream:size() - 1;

    while true do
        mid = math.floor((min + max) / 2);
        periodsec = math.floor(stream:date(mid) * 86400 + 0.5);
        if datesec == periodsec then
            return mid;
        elseif datesec > periodsec then
            min = mid + 1;
        else
            max = mid - 1;
        end
        if min > max then
            if precise then
                return -1;
            else
                return min - 1;
            end
        end
    end
end

Now apply the indicator to the chart.


It works!!!! But stop, stop, stop… When we try to load more data into past it apparently stopped to work!



Point 3. How to get the day price data?

What happened?! Ok. Remember, we loaded the data starting from yesterday for the oldest of the existing candles. When we started to scroll our chart back, more data was loaded. But who updated our collection of day bars? Of course, nobody did it. So, we have to force Marketscope to load new data when the source collection is extended into past. Good news is that there is another cool call of the host interface: host:execute(“extendHistory”) which helps us just load more data into already loaded history instead of loading the whole history again. Because the original source history can be extended only into past, it is too easy to check whether new data is loaded. When it happens, the “yesterday” value we are trying to find is earlier than the date we previously loaded the day history from.

So, do the following modification in our code:

a) Keep the “from” date we loaded the day history earlier.
b) In case the day collection already exists, extend it instead of loading older data.

So, add new loadedFrom global variable. And change the Update and loadData functions.

There is new indicator version:

Code:

Code: Select all
function Init()
    indicator:name("Show Yesterday Close");
    indicator:description("The indicator shows yesterday's close value on the chart");
    indicator:requiredSource(core.Tick);
    indicator:type(core.Indicator);

    indicator.parameters:addColor("clr", "Indicator Line Color", "", core.rgb(255, 255, 0));
end

local source;       -- indicator source data
local first;        -- first available source data
local days;         -- day source data
local loading;      -- the loading day data flag
local loadedFrom;   -- the oldest yesterday bar we already loaded
local SYC;          -- yesterday close output

local host;
local offset;
local weekoffset;

function Prepare()
    local name;

    name = profile:id() .. "(" .. instance.source:name() .. ")";
    instance:name(name);

    source = instance.source;
    first = source:first();
    loading = false;
    days = nil;

    host = core.host;
    offset = host:execute("getTradingDayOffset");
    weekoffset = host:execute("getTradingWeekOffset");
    SYC = instance:addStream("SYC", core.Line, name, "SYC", instance.parameters.clr, first);
end

function Update(period, mode)
    if period < first then
        return ;
    end
    if loading then
        return ;
    end

    -- load the data if there is no day
    if days == nil then
        loadData();
        return ;
    end

    -- try to find the value in the days collection
    local yesterday;
    yesterday = getYesterday(source:date(period));

    -- check whether such day bar is not loaded yet
    if yesterday < loadedFrom then
        loadData();
        return ;
    end

    -- find the day data
    local i;
    i = findDateFast(days, yesterday, false);
    if i >= 0 then
        SYC[period] = days.close[i];
    end
end

function AsyncOperationFinished(cookie, success, error)
    if cookie == 1 then
        assert(success, error);
        loading = false;
        instance:updateFrom(source:first());
        return core.ASYNC_REDRAW;
    end
    return 0;
end

-- the function loads the proper day history
function loadData()
    if source:size() < source:first() then
        return ;
    end

    if loading then
        return ;
    end

    local from, to;

    if days == nil then
        -- initial data loading

        -- load the data from the date of proper yesterday for
        -- the first available bar of the source
        from = getYesterday(source:date(source:first()));
        -- load to "now" is the source is up to "now" or
        -- to the day which corresponds to the latest
        -- day of the collection.
        if source:isAlive() then
            to = 0;
        else
            to = getYesterday(source:date(source:size() - 1));
        end
        loading = true;
        loadedFrom = from;
        days = host:execute("getHistory", 1, source:instrument(), "D1", from, to, source:isBid());
    else
        from = getYesterday(source[source:first()]);
        to = days:date(0);
        loading = true;
        loadedFrom = from;
        host:execute("extendHistory", 1, days, from, to);
    end
end

-- returns the beginning of the "yesterday" date for
-- the specified data
function getYesterday(date)
    local today, yesterday;

    -- get beginning of the today's candle and round it up to the second
    today = core.getcandle("D1", date, offset, weekoffset);
    today = math.floor(today * 86400 + 0.5) / 86400;

    -- get calendar yesterday
    yesterday = today - 1;

    local nontrading, nontradingend;
    nontrading, nontradingend = core.isnontrading(yesterday, offset);
    if nontrading then
        -- if the calendar yesterday date is non-trading (Friday 17:00 EST - Sunday 17:00 EST)
        -- shift three days back against the non-trading period end (Sunday 17:00 EST).
        yesterday = nontradingend - 3;
    end
    return yesterday;
end

-- ------------------------------------------------------------------------------
-- Find the specified date in the specified stream
--
-- The function uses the binary search algorithm, so it requires only O(n) = log2(n) operations
-- to find (or to do not find) the value.
--
-- The function compares the date and time rounded to the whole seconds.
--
-- Parameters:
-- stream       The price stream to find the date in
-- date         Date and time value to be found
-- precise      The search mode
--              In case the value of the parameter is true, the function
--              Searches for the the exact date and returns not found in the
--              date is not found.
--              In case the value of the parameter is false, the function
--              returns the period with the biggest time value which is smaller
--              than the value of the date parameter.
-- Returns:
-- < 0          The value is not found
-- >= 0         The index of the the period in the stream
-- ----------------------------------------------------------------------------------
function findDateFast(stream, date, precise)
    local datesec = nil;
    local periodsec = nil;
    local min, max, mid;

    datesec = math.floor(date * 86400 + 0.5)

    min = 0;
    max = stream:size() - 1;

    while true do
        mid = math.floor((min + max) / 2);
        periodsec = math.floor(stream:date(mid) * 86400 + 0.5);
        if datesec == periodsec then
            return mid;
        elseif datesec > periodsec then
            min = mid + 1;
        else
            max = mid - 1;
        end
        if min > max then
            if precise then
                return -1;
            else
                return min - 1;
            end
        end
    end
end

Install the indicator and try it. Wow. Now, when the previous version stopped working during the scroll, the new version works anyway! Congratulation!

http://www.fxcodebase.com/code/downl...le.php?id=1297

Notes

Note 1. How to make this example working on pre August 27, 2010 releases?

The new feature used in this example is advanced AsyncOperationFinished parameters and return values. In earlier releases the function has only one parameter (cookie), and never returns a value, so, Marketscope redraws the indicator always after the call of this function. So, just remove additional parameters and their processing, as well as the return value, and the indicator will work on the older Marketscope versions:

Code:

Code: Select all
function AsyncOperationFinished(cookie)
    if cookie == 1 then
        loading = false;
        instance:updateFrom(source:first());
    end
end

Note 2. How to debug such indicators using SDK debugger?

First, you must prepare the data. SDK debugger does not load the data automatically, you must prepare the price data. Just open some instrument in both resolutions (for example 30 minutes and 1 day) and then save both histories in the Indicore SDK data format (File->Export to Indicore…).

When you start the indicator, choose the smaller timeframe history in the indicator parameters. Please, also check the “Predeliver Data” checkbox in the indicator parameters to let the debugger show the whole available history to the indicator, like Marketscope does. Without this option, the debugger will show the history only up to the currently simulated bar. When the indicator executes “getHistory” call, the prompt for data appears. Choose the day data previously saved from Marketscope in that prompt.

Note 3. How to handle timeframes other than 1 day?

To make the example indicator simpler, I used a 1-day time frame. However, you would want to do the same for other time frame data, for example, use 1-hour close price instead of a day close price. To do that, you can:

a) Let the user choose the time frame. Add a string parameter to the indicators parameter list and then set the time frame selector flag to that parameter. As a result, in the Prepare function, the corresponding parameter will contain the code of the time frame chosen by the user.

Code:

Code: Select all
function Init()
    ...
    indicator.parameters:addString("TF", "Timeframe", "", "m1");
    indicator.parameters:setFlag("TF", core.FLAG_PERIODS);
    ...
end

b) Calculate the length of a candle of the specified time frame in day. Use the core.getcandle() function to get the start date of any candle and the start date of the next candle. The difference between these two values is the length of the candle in days. I also strongly recommend calculating all dates in seconds rather than in days to avoid rounding errors.

Code:

Code: Select all
local candleLength;  -- length of candle in seconds

function Prepare()
    ...
    offset = host:execute("getTradingDayOffset");
    weekoffset = host:execute("getTradingWeekOffset");
    ...
    local thisCandle, nextCandle;
    thisCandle, nextChandle = core.getcandle(instance.parameters.TF, core.now(), offset, weekoffset);
    candleLength = math.floor((nextCandle - thisCandle) * 86400 + 0.5);
    ...
end

c) Use previously calculated candle length instead of 1 day value in your “getPreviousBar” function. Please note that while the candle length could be different, the non-trading period is still in days. So, handle it carefully.

Code:

Code: Select all
function getPreviousBar(date)
    local curr, prev;

    -- get beginning of the today's candle and round it up to the second
    curr = core.getcandle(instance.parameters.TF, date, offset, weekoffset);
    today = math.floor(today * 86400 + 0.5);

    -- get calendar yesterday
    prev = (curr – candleLength) / 86400;

    local nontrading, nontradingend;
    nontrading, nontradingend = core.isnontrading(prev, offset);
    if nontrading then
        prev = prev - 2;
        prev = math.floor(prev * 86400 + 0.5);

        -- get calendar yesterday
        prev = (prev – candleLength) / 86400;
    end
    return prev;
end