Any C++ programmers willing to help modify RTDMan?

jagankris

Well-Known Member
#11
This code is working well. I have hardcoded the refresh period as 1 second.
It creates 1sec. and 1 min. bars optionally as per bar period in settings.ini set as 1000 or 60000.

I tallied the results with Hourly statistics data from NOW. It is tallies to a large extent.
However, it is missing some ticks. (In every 1 crore nifty futures, 6000 to 20000 shares are missed). This happens because thread A is maintaining OHLC from data received in array. It creates one current and one previous bar for each scrip. If it receives 1 tic of one period and 1 tick of next period in the same array, it creates new current bar and data for first tick is lost as it is pushed to previous bar. This also creates slight 5, 10 paise deviation in OHLC and volume of bars.

Though this is acceptable in time frames of 3 min and above, I am not satisfied with it. Therefore I decided to go for option 2 of Tracerbullet.
Dear Josh,

Please find the comments from Zerodha to a similar question asked.

Are you trying to capture the data for several scripts parallel ?
The issue may be due to Internet connection ? and not with the code.

http://www.traderji.com/brokers-trading-platforms/89121-zerodha-part-3-a-1031.html#post1088126
 

josh1

Well-Known Member
#12
Last edited:

TracerBullet

Well-Known Member
#13
In my first attempt, I made changes to the way in which thread A (worker::processRTDdata) was maintaining OHLC of scrips. Entire processing is done in that thread and thread B (AmibrokerPoller) accesses the bars every second and pushes them to Amibroker.

The new code is here-
New header file-
......
This code is working well. I have hardcoded the refresh period as 1 second.
It creates 1sec. and 1 min. bars optionally as per bar period in settings.ini set as 1000 or 60000.

I tallied the results with Hourly statistics data from NOW. It is tallies to a large extent.
However, it is missing some ticks. (In every 1 crore nifty futures, 6000 to 20000 shares are missed). This happens because thread A is maintaining OHLC from data received in array. It creates one current and one previous bar for each scrip. If it receives 1 tic of one period and 1 tick of next period in the same array, it creates new current bar and data for first tick is lost as it is pushed to previous bar. This also creates slight 5, 10 paise deviation in OHLC and volume of bars.

Though this is acceptable in time frames of 3 min and above, I am not satisfied with it. Therefore I decided to go for option 2 of Tracerbullet.
joshbhai, Assuming its same code as the one you pmed before - this is same problem as i mentioned in original post(which you quoted) and here. This is not solution 1 as thread 1 will have to call and block thread 2 when its changing minute.
Just discard option 1 as its not nice. Option 2 with arrays is the most clean and preferable way to do it.
Maybe you didnt see, but i also gave workaround option 3 which should hopefully be the easiest to do. In this the first thread (processRTDData) will keep two bars instead of 1 for current and switch between them when minute changes. All this is theory as i didnt read code for long time.

I created new structure RTDData to hold tic data and two forward_lists, array1 and array2 containing the structure.

Thread Process_RTDdata receives real time data from RTD server and puts it into array1 with push.front.

Thread Amibroker_Poller copies array1 into array2 -- clears array1 -- reverses array2 and thereafter processes data from array 2 for maintaining OHLC of current Bar. OHLC is maintained as per the same logic as my earlier code. The current bar is sent to Amibroker every second.

This code is also working properly. Though I have not tallied it with Hourly statistics data yet.
However, there is one stupid bug in the code. It is not creating bars for the last scrip given in settings.ini. This problem can be solved easily by including one extra scrip in settings.ini. However, that is bad work around. I need immediate help to solve this.

Here is the code (worker.cpp) for second option.
....

There are some other issues which can be looked at later.
1. Whether list can be used instead of forward_list.
2. RTD server sometimes sends previous days' last tic in the morning. Tracerbullet's code ignores first tic of the day as a workaround. However, tic is missed in case of scrips for which RTD server sends current day's data.
3. compact the code.
Didnt read code, but you dont need to use linked list. A simple vector ( which is like ArrayList in java) will do and will be faster. You just need Thread 1 to keep list of quotes in a Vector[] ( So its Double array - 1 vector for each scrip ). Thread 2 will read this Vector and empty it after reading. Read data will be sent to AB.
The read vector part of thread 2 has to be within the EnterCriticalSection and LeaveCriticalSection - This blocks thread 1 from accessing the data when Thread 2 is reading it. No need to create copies, just appened them the new bars in the loop. For each scrip you will need to add another loop for the vector.

Dear Josh,

Please find the comments from Zerodha to a similar question asked.

Are you trying to capture the data for several scripts parallel ?
The issue may be due to Internet connection ? and not with the code.

http://www.traderji.com/brokers-trading-platforms/89121-zerodha-part-3-a-1031.html#post1088126
This is not relevant as its for Nest to handle. We only talk with Nest client and not outside. Latency ( roundtrip ping + client time + server time) impacts on how many ticks Nest can recieve from its server.
 

mastermind007

Well-Known Member
#14
If I attempt to download the source, I get error message that reads

Sorry, this file is infected with a virus

Only the owner is allowed to download infected files.
 

josh1

Well-Known Member
#15
1. joshbhai, Assuming its same code as the one you pmed before - this is same problem as i mentioned in original post(which you quoted) and here. This is not solution 1 as thread 1 will have to call and block thread 2 when its changing minute.
Just discard option 1 as its not nice. Option 2 with arrays is the most clean and preferable way to do it.

2. Maybe you didnt see, but i also gave workaround option 3 which should hopefully be the easiest to do. In this the first thread (processRTDData) will keep two bars instead of 1 for current and switch between them when minute changes. All this is theory as i didnt read code for long time.

3.
Didnt read code, but you dont need to use linked list. A simple vector ( which is like ArrayList in java) will do and will be faster. You just need Thread 1 to keep list of quotes in a Vector[] ( So its Double array - 1 vector for each scrip ). Thread 2 will read this Vector and empty it after reading. Read data will be sent to AB.
The read vector part of thread 2 has to be within the EnterCriticalSection and LeaveCriticalSection - This blocks thread 1 from accessing the data when Thread 2 is reading it. No need to create copies, just appened them the new bars in the loop. For each scrip you will need to add another loop for the vector.
1. No this is not the same code I pmed you before.

2. Not tried that yet. I did not want to send duplicate quotes to Amibroker.

3. I am using forward_list since it is fastest and simplest form of array compared to list and vector (as I understand from Cplusplus.com ). However, it does not have pushback function since it does not keep pointer to last element.
Thread 1 pushes bars to the front of list1
Thread 2 merges list1 into list2. That empties list1. That part is in EnterCriticalSection and LeaveCriticalSection.
List2 is thereafter required to be reversed to get it in cronological order.
Thereafter I add another loop for maintaining and current and previous bar.
 

mastermind007

Well-Known Member
#16
......
However, there is one stupid bug in the code. It is not creating bars for the last scrip given in settings.ini. This problem can be solved easily by including one extra scrip in settings.ini. However, that is bad work around. I need immediate help to solve this.
Josh1

I did manage to get the Source Code from another link and have a solution for problem of code missing the last scrip in the INI file.
You will not need to use any work-around anymore.

Error is in the Settings::loadSettings() {} function

The current structure of the function in pseudo code (omitting details) is
Code:
while (1)  {
    readData();

   if (dataIsEmpty)
        break;

  processData();

  incrementCountersAsNeeded();
}
AND is wrong. It should be

Code:
readData();
while (!dataIsEmpty)  {

  processData();

  incrementCountersAsNeeded();

  readData();
}
Notice that calls to the readData() are repeated twice and also loop is no longer an infinite loop with a break embedded inside.

There are two other problem areas in the code that I can identify that may need correction.
First (a) On line 37, there is a throw statement which is bit too hard. I believe that if the data string is not split validly, it is not a catastrophic failure to throw exception. Code can either assume some defaults or ignore the error and proceed with the rest.
Second the throw statement that says ("At least one scrip is needed") needs to be based off scrips_array.size and not our local variable no_of_scrips.

Entire Corrected source code is

Code:
#include "settings.h"
#include "misc_util.h"

void Settings::loadSettings(){

    rtd_server_prog_id    = MiscUtil::getINIString("RTDServerProgID");
    bar_period            = MiscUtil::getINIInt   ("BarPeriod");
    csv_path              = MiscUtil::getINIString("CSVFolderPath");
    time_format           = MiscUtil::getINIString("TimeFormat");
    ab_db_path            = MiscUtil::getINIString("AbDbPath");
    no_of_scrips          = 0 ;

    std::string scrip_value;
    std::string scrip_key;

    if( bar_period < 1000  ){                                                // check $TICKMODE 1
        throw "Minimum Bar Interval is 1000ms";
    }

    MiscUtil::createDirectory( csv_path );                                   // If folder does not exist, create it.
    csv_path.append("quotes.rtd");

    // >>>>>Code fixing by mastermind007 (Traderji.com): moved data reading from inside the while loop to above the loop and then at end
    scrip_key    = "Scrip";  scrip_key.append( std::to_string( (long long)no_of_scrips+1 ) );
    scrip_value  = MiscUtil::getINIString( scrip_key.c_str() ) ;

    // -------- Code fixing by mastermind007 (Traderji.com): moved the loop's exit condition into the while
    while(!scrip_value.empty()){

    // <<<<<<Code fixing by mastermind007 (Traderji.com)

        //  ScripID(mandatory);Alias(mandatory);LTP(mandatory);LTT;Todays Volume;OI
        std::vector<std::string>  split_strings;
        MiscUtil::splitString( scrip_value , ';', split_strings ) ;
        if(split_strings.size() < 3 ){                                       // 3 mandatory field at start
            throw( scrip_key + " Invalid" );
        }

        Scrip    scrip_topics;

        scrip_topics.topic_name   =  split_strings[0];
        scrip_topics.ticker       =  split_strings[1];
        scrip_topics.topic_LTP    =  split_strings[2];

        if(split_strings.size() >=4 ){
            scrip_topics.topic_LTT       =  split_strings[3];
        }
        if(split_strings.size() >=5 ){
            scrip_topics.topic_vol_today =  split_strings[4];
        }
        if(split_strings.size() >=6 ){
            scrip_topics.topic_OI        =  split_strings[5];
        }

        scrips_array.push_back(  scrip_topics ) ;
        no_of_scrips++;

        // >>>>>Code fixing by mastermind007 (Traderji.com): Repeated the reading of data here
        scrip_key    = "Scrip";  scrip_key.append( std::to_string( (long long)no_of_scrips+1 ) );
        scrip_value  = MiscUtil::getINIString( scrip_key.c_str() ) ;
        // <<<<<<Code fixing by mastermind007 (Traderji.com)
    }

    // >>>>>Code fixing by mastermind007 (Traderji.com): Moved the minimum requirement checking from inside the while loop to below the loop;
    // ------Code fixing by mastermind007 (Traderji.com): Recommended to use scrips_array.size instead of no_of_scrips
    if( no_of_scrips == 0 ){
    // No more Scrips left
        throw( "At least one scrip is needed in the settings.ini file" );
    // <<<<<<Code fixing by mastermind007 (Traderji.com)
    }
}
 

josh1

Well-Known Member
#17
Josh1

I did manage to get the Source Code from another link and have a solution for problem of code missing the last scrip in the INI file.
You will not need to use any work-around anymore.
Thanks. I will check that.
There is no virus in files. Google complains because it thinks Autoit is a virus.
 

josh1

Well-Known Member
#18
@mastermind007

No luck. It is not working. :mad:

Edit- Data is received from all scrips. However, something wrong is happening when it is transferred to second array.
 
Last edited:

TracerBullet

Well-Known Member
#19
1. No this is not the same code I pmed you before.

2. Not tried that yet. I did not want to send duplicate quotes to Amibroker.

3. I am using forward_list since it is fastest and simplest form of array compared to list and vector (as I understand from Cplusplus.com ). However, it does not have pushback function since it does not keep pointer to last element.
Thread 1 pushes bars to the front of list1
Thread 2 merges list1 into list2. That empties list1. That part is in EnterCriticalSection and LeaveCriticalSection.
List2 is thereafter required to be reversed to get it in cronological order.
Thereafter I add another loop for maintaining and current and previous bar.
2) Duplicate quote will only get overwritten. To end user it should not make any difference and this will be easiest way to do it, esp if new to c++.

3) ok i forgot about creating bars.

About the bugs in array code - best is to debug in Visual Studio and check whats the issue. Hard to read code otherwise.

I skimmed over it, not related to bug but - make sure to put all code that accesses shared data within EnterCriticalSection and LeaveCriticalSection. It seems to have been
removed in processRTDData(). processRTDData() and amibrokerPoller() are called by two different threads and they should never be able to access same data at same time.

Finally "code ignores first tic of the day as a workaround.". Are you sure of this, i was previously always reading data on application start. Only that was changed and now it should start taking data whenever RTD Server starts sending.

Josh1

I did manage to get the Source Code from another link and have a solution for problem of code missing the last scrip in the INI file.
You will not need to use any work-around anymore.

Error is in the Settings::loadSettings() {} function

The current structure of the function in pseudo code (omitting details) is
Code:
while (1)  {
    readData();

   if (dataIsEmpty)
        break;

  processData();

  incrementCountersAsNeeded();
}
AND is wrong. It should be

Code:
readData();
while (!dataIsEmpty)  {

  processData();

  incrementCountersAsNeeded();

  readData();
}
Notice that calls to the readData() are repeated twice and also loop is no longer an infinite loop with a break embedded inside.
Could you explain what exactly is wrong with the original code apart from how you prefer it. I can see that it 'looks' better but it doesnt cause any bug.

There are two other problem areas in the code that I can identify that may need correction.
First (a) On line 37, there is a throw statement which is bit too hard. I believe that if the data string is not split validly, it is not a catastrophic failure to throw exception. Code can either assume some defaults or ignore the error and proceed with the rest.
Second the throw statement that says ("At least one scrip is needed") needs to be based off scrips_array.size and not our local variable no_of_scrips.

Entire Corrected source code is
...
1) Again, this has nothing to do with his problem. For me, incorrect settings.ini is bad and hiding it will make it harder for the user to discover - but anyway i doesnt matter much.

2) They both do same thing, do you seen any bug here?
 
Last edited:

josh1

Well-Known Member
#20
@Tracerbullet,

There is nothing wrong with your code. All data from RTD server is received.
It comes sequentially in pairs like this.
topic_id - ltp, topic_id - ltt, topic_id - volume_traded_today, topic_id - oi
These are accumulated into mydata structure.
I created a condition to push mydata into array at the change of topic_id. Something is going wrong there.

This is the loop.
Code:
void Worker::processRTDData( const std::map<long,CComVariant>* data ){
	
	RTData mydata;												//inserted by Josh1
	int  prev_script_id = 999999;								// set prev_script_id as a large integer which can never be reached 

	for( auto i=data->begin(), end=data->end() ;  i!=end ; ++i  ){
		
        const long   topic_id     = i->first;
        CComVariant  topic_value  = i->second;
                
        std::pair<int,int> &ids = topic_id_to_scrip_field_map.at( topic_id ) ; 

		int script_id   =  ids.first;                                      // Resolve Topic id to Scrip id and field
        int field_id    =  ids.second;
        ScripState *_current = & current[script_id];
//		Changes made by Josh1 - fill all fields of mydata from RTDdata.
//****************************************************************************

		if(mydata.sid != script_id && prev_script_id != 999999) {	//if data for different scrip and it is not 
			array1.push_front(mydata);									// first scrip then push it to array1

/**************************************************************************/
//Uncomment these lines to view data filled in mydata structure
		if(mydata.sid == 6 ){
		std::cout << mydata.ticker << " - " <<
					script_id <<" - " << 
					mydata.sid <<" - " << 
					mydata.ltt <<" - " << 
					mydata.ltp <<" - "<<
					mydata.vol_today <<" - "<<
					mydata.oi  <<"Pushed"<< std::endl;
		}	
/**/

		mydata.reset();
		};

		mydata.ticker = settings.scrips_array[script_id].ticker;		// fill scrip name from settings
		mydata.sid = script_id;											// store script id also
		prev_script_id = script_id;										// and push to previous script id for comparing next time

		switch( field_id ){											//Start filling data in my data.
            case LTP :{
                double      ltp      = MiscUtil::getDouble( topic_value );
                mydata.ltp = ltp;
                break ;    
            }
            case VOLUME_TODAY :{  
                long long vol_today          = MiscUtil::getLong  ( topic_value );
                mydata.vol_today = vol_today;
                 break ;
            }
            case LTT  : {
				mydata.ltt  = MiscUtil::getString( topic_value );  
				break ;
			}

            case OI   :  mydata.oi   = MiscUtil::getLong  ( topic_value ); break ;
        }	

/**************************************************************************/
//Uncomment these lines to view data filled in mydata structure
		if(script_id == 6 ){
		std::cout << mydata.ticker << " - " <<
					script_id <<" - " << 
					mydata.sid <<" - " << 
					mydata.ltt <<" - " << 
					mydata.ltp <<" - "<<
					mydata.vol_today <<" - "<<
					mydata.oi  << "received" <<std::endl;
		}	
/**/

//****************************************************************************/
//		End Changes made by Josh1 

	}    

}
This is settings.ini
Code:
[RTDMan]
RTDServerProgID=Nest.ScripRTD
CSVFolderPath=R:\RT\
; Time format used when LTT empty - std::strftime
TimeFormat=%H:%M:%S
BarPeriod=1000
Scrip1=CNX Nifty;NIFTY;Index Value;;
Scrip2=MCXENERGY;MCXENERGY;Index Value;;
Scrip3=MCXMETAL;MCXMETAL;Index Value;;
Scrip4=MCXCOMDEX;MCXCOMDEX;Index Value;;
Scrip5=mcx_fo|COPPER15AUGFUT;COPPER15AUGFUT;LTP;LTT;Volume Traded Today;Open Interest
Scrip6=mcx_fo|CRUDEOILM15JULFUT;CRUDEOILM15JULFUT;LTP;LTT;Volume Traded Today;Open Interest
Scrip7=mcx_fo|CRUDEOIL15JULFUT;CRUDEOIL15JULFUT;LTP;LTT;Volume Traded Today;Open Interest
 

Similar threads