Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts MS

Status
Not open for further replies.

xsis

Active Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

thks josh! finally got the difference!

btw - i just saw that ur signature has changed. so got curious and read few last pages of this wonderful thread. is there a change in the process/method of getting RT data from various terminals to amibroker. i mean is this a different method? since ur working on sumthing new, it has to be better. but cn u explain how this new method is better than the old one?


Nest plus is irrelevant. It is additional utilities for Which separate login is required. It is useful for multiple days backfill.
NSENOW is given by NSE. ZT aka Nest trader is its sister given by Zerodha and others.
 

TracerBullet

Well-Known Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Edit - Can you please change your signature link from the test build to https://github.com/SpiffSpaceman/AmibrokerFeeder/releases/latest
This one will always have the latest build so you wont have to change it. I also forgot to put license and source link in the test build.
Done
The link redirects to 0.1.2. So what you copied in sig goes to 0.1.2 always. Copy text inside quotes - "https://github.com/SpiffSpaceman/AmibrokerFeeder/releases/latest" exactly

Configured NestRtd for NSENOW. It is updating everything including options. No issues. May be I did not configure options properly last time.
How can I keep VC++2010 working after 30 days? I may not be able to understand your code in that time
Great. I think you only need to register once (free). You may need a ms email id, dont remember.
http://msdn.microsoft.com/en-us/library/ms246592(v=vs.90).aspx
 

josh1

Well-Known Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Lets hope quotations is the bottleneck. Otherwise we need a way to profile this. ( profile tool shows how much time each function takes cumulatively)

I could not find any tool for autoit using google. Instead we can use timers
http://www.autoitscript.com/wiki/Tutorial_Optimizing

If the bottleneck is in remove of AB then we are out of luck ( Maybe alternative is try to run AFL using a COM call if available )
Working Script for removing quotes. You will have to install Autoit to run

Code:
#cs ........Intro
;// //                                                                   //
;// // File:                DelRT.au3                          			//
;// // Author:              Josh1		     	                        //
;// // Purpose:             Remove all quotations between                //
;// //                      two given Date 				               //
;// // Language:            Autoitscript					             //
;// // ENJOY :-)                                                         //
;// //                                                                   //
;   // ////////////////////////////////////////////////////////////////////
#ce

Global $DataDir, $DayDeleteFrom, $DayDeleteUpto

#include <ButtonConstants.au3>
#include <DateTimeConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include <GuiDateTimePicker.au3>
#include "DTC.au3"
#include <Date.au3>
#include <Array.au3>
#include <GuiEdit.au3>


#Region ### START Koda GUI section ### Form=
Opt("GUIOnEventMode", 1) ; Change to OnEvent mode
;$Scrip = "NIFTY"

$Form1 = GUICreate("Select Days to delete", 460, 322, 202, 120)
GUISetFont(10, 400, 0, "MS Sans Serif")
GUISetOnEvent($GUI_EVENT_CLOSE, "CLOSEButton")
$Label1 = GUICtrlCreateLabel("From Date", 40, 20, 67, 20)
$Date1 = GUICtrlCreateDate(_NowCalcDate(), 128, 12, 233, 24)
$Label2 = GUICtrlCreateLabel("To Date", 40, 52, 53, 20)
$Date2 = GUICtrlCreateDate(_NowCalcDate(), 128, 52, 233, 24)
$Label3 = GUICtrlCreateLabel("Database Path", 24, 92, 94, 24)
$Input1 = GUICtrlCreateInput(IniRead(@WorkingDir & "\DelRT.ini", "AB", "Datadir", "C:\AmiBroker\"), 128, 92, 297, 24)
$Label4 = GUICtrlCreateLabel("Delete data for", 24, 122, 94, 24)
$Input2 = GUICtrlCreateInput(IniRead(@WorkingDir & "\DelRT.ini", "AB", "Scrip", "NIFTY"), 128, 122, 297, 24)
$Button1 = GUICtrlCreateButton("OK", 168, 162, 105, 25)

$Edit1 = GUICtrlCreateEdit("",128,192,300,120)

GUICtrlSetOnEvent($Date1, "Date1Change")
GUICtrlSetOnEvent($Date2, "Date2Change")
GUICtrlSetOnEvent($Input1, "Input1Change")
GUICtrlSetOnEvent($Input2, "Input2Change")
GUICtrlSetOnEvent($Button1, "DeleteTics")

$DTM_SETFORMAT_ = 0x1032 ; $DTM_SETFORMATW
	$style = "yyyy/MM/dd"
	GUICtrlSendMsg($Date1, $DTM_SETFORMAT_, 0, $style)
	GUICtrlSendMsg($Date2, $DTM_SETFORMAT_, 0, $style)

GUISetState(@SW_SHOW,$Form1)
#EndRegion ### END Koda GUI section ###


While 1

	Sleep (100)
WEnd

Func Date1Change()
	If GUICtrlRead($Date1) <= GUICtrlRead($Date2) Then
;		MsgBox(64,"",$DayDeleteFrom & " - " & GUICtrlRead($Date1))
	Else
		MsgBox(64,"","From Date cannot be greater than To Date")
		GUICtrlSetData($Date1,_NowCalc())
	EndIf
EndFunc

Func Date2Change()
	If GUICtrlRead($Date2) < GUICtrlRead($Date1) Then
		MsgBox(64,"","To date cannot be before From Date")
		GUICtrlSetData($Date2,_NowCalc())
	ElseIf 	GUICtrlRead($Date2) > _NowCalc() Then
		MsgBox(64,"","To date cannot be future Date")
		GUICtrlSetData($Date2,_NowCalc())
	EndIf
EndFunc

Func Input1Change()
	$DataDir = GUICtrlRead($Input1)
EndFunc

Func Input2Change()
	$Scrip = GUICtrlRead($Input2)
EndFunc

Func CLOSEButton()
    Exit
EndFunc   ;==>CLOSEButton

Func DeleteTics()
	$DayDeleteFrom = (_Date_Time_Convert(GUICtrlRead($Date1), "yyyy/MM/dd", "yyyyMMdd")) & "000001"
	$DayDeleteUpto = (_Date_Time_Convert(GUICtrlRead($Date2), "yyyy/MM/dd", "yyyyMMdd")) & "235959"
	$DataDir = GUICtrlRead($Input1)
	$Scrip = GUICtrlRead($Input2)
	IniWrite(@WorkingDir & "\DelRT.ini", "AB", "Datadir", $DataDir)
	IniWrite(@WorkingDir & "\DelRT.ini", "AB", "Scrip", $Scrip)

	$oAB = ObjCreate("Broker.Application")
	$oAB.Visible = 1
;	$oAB.LoadDatabase($DataDir)
	$oStocks = $oAB.Stocks
	$Qty = $oStocks.Count
	$i = 0
    $j = 0
    $k = 0

	For $i = 0 To ($Qty-1)
		$Stock = $oStocks($i)
		$k = 0
		$ScripName = $Stock.Ticker
		$QQ = $Stock.Quotations.Count
		$Quotes = $Stock.Quotations

		if $ScripName = $Scrip Then
			_GUICtrlEdit_AppendText($Edit1,"Total Quotes - "&$ScripName&" -"& $QQ & @CRLF)
			$ts = TimerInit()
			For $j = ($QQ-1) to 0 Step -1
					$Quote = $Quotes($j)
					$tmpDateNum = $Quote.Date

				if (($tmpDateNum >= $DayDeleteFrom) and ($tmpDateNum <= $DayDeleteUpto)) Then
					$k+=1
					$Quotes.Remove($j)
				ElseIf ($tmpDateNum < $DayDeleteFrom) Then
					ExitLoop
				EndIf
			Next
			$oAB.RefreshAll
			$td = TimerDiff($ts)
			_GUICtrlEdit_AppendText($Edit1,"Total" & $k &"- Quotations removed in -" & int($td) & " miliSecs")
		EndIf
   Next
EndFunc
Edit -Actually Database path is redundant. I am not loading it. Taking default/running database.
 
Last edited:

josh1

Well-Known Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Working Script for removing quotes. You will have to install Autoit to run
thanks Sirji....

DTC.au3 bhi de dijiye......sabhi ko....
Search DTC.au3 on the net. I am giving executable now.
 

josh1

Well-Known Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

NestVwapBackfill_0.2 (much more elegant than 0.1) along with DeleteRT

Download from here - http://ubuntuone.com/6K3Id280AxbseUIZZpfpIm

It will create .ini file in same folder
 

josh1

Well-Known Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

thks josh! finally got the difference!

btw - i just saw that ur signature has changed. so got curious and read few last pages of this wonderful thread. is there a change in the process/method of getting RT data from various terminals to amibroker. i mean is this a different method?
since ur working on sumthing new, it has to be better. but cn u explain how this new method is better than the old one?
Yes
Advantages are given by TracerBullet on his site.
 

TracerBullet

Well-Known Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

Working Script for removing quotes. You will have to install Autoit to run

Edit -Actually Database path is redundant. I am not loading it. Taking default/running database.
There are still extra com calls :) but its irrelevant. Quotations is not bottleneck.

I have about 11000 quotes for today.

1. I commented $Quotes.Remove($j) . Time went from 77190 to 51625. i dont think there are extra removes. So even if we magically cut down everything else, it will still take 25 seconds which is too much.

- You will need to have less number of bars.
- Another way to improve here will be to backfill only missing data. Then delete will only remove overlapping bars. But we will still need to query all bars which seems to take 50 seconds for 1 day data per scrip.

2. The most common use case for me is to backfill at start of day. In this case, there is no remove - so backfill should be fast enough in any case.
I need to check if i can do it faster manually for 2 scrips - i only take 5-10 seconds, so should be close :)

3. When it runs, AB is at full cpu usage. So assuming we are calling correctly, the bottleneck is in AB calls - $Quotes($j) / $Quote.Date / Quotes.Remove($j). So, even a c++ port will still be slow.

- You can ask AB (forum?) if there is a better way to do bulk delete. Maybe we can write an AFL scrip to delete data and call it from exe ( if this is possible)
- Gambler said that a yusi application does backfill. Maybe he can review code?
 

josh1

Well-Known Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

- You can ask AB (forum?) if there is a better way to do bulk delete. Maybe we can write an AFL scrip to delete data and call it from exe ( if this is possible)
- Gambler said that a yusi application does backfill. Maybe he can review code?
There is no command in AFL to remove quotations. It is possible only thru COM calls and that too by iteration one at a time. That is bottleneck.

Yusi application does backfill. However, his data feed application compiles and sends 1 min. bars to AB. If I backfill at 1 PM, there are approx.225 bars to remove and 375 bars at 3:30 PM. Therefore it is fast.
 

TracerBullet

Well-Known Member
Re: Free RealTime Data NOW Nest ODIN Trade Tiger Google Yahoo to AmiBroker, Fcharts M

There is no command in AFL to remove quotations. It is possible only thru COM calls and that too by iteration one at a time. That is bottleneck.

Yusi application does backfill. However, his data feed application compiles and sends 1 min. bars to AB. If I backfill at 1 PM, there are approx.225 bars to remove and 375 bars at 3:30 PM. Therefore it is fast.
75/3 = 25. Removing remove function reduced time exactly by 1/3.
We make 3 COM calls per scrip >11000 times.
I think there is a possibility that COM itself is using up time. I tried to profile the exe once, and there was significant time spent on some marshalling stuff. I ignored it, but now i think maybe its trying to convert datatypes.

This gives me hope that we can reduce time with decent c++ implementation.
Ill try to test it later ( not soon ). If there is difference, we can simply call the exe from your autoit script.
 
Status
Not open for further replies.

Similar threads