Hello
When I first started trading, the person who guided me (or misguided me) got me to sign up for a EOD data feed that was costing 5000 Rs a year. I received approx 10 years EOD data in MWD file format for NSE, BSE, MCX, Forex as well as US scrips.
US Scrip provided in data feed is actually waste of disk space because (a) I am not going to trade in 'em anyway. (b) Most scrip names are cryptic
Its a neat trick though. By showing so much variety of data, it is easier to fool a novice.
Forex data was rounded off at 2 digits so that waste too. It contains so many many currency pairs that I actually doubt its accuracy. Even EODData.com does not provide all those currencies.
Coming to the most important part, NSE, BSE and F&O. Data was segregated
haphazardly. NIFTY folder had 180 scrips and some of them were dead scrips.
Wrote code to list out files and scrips and soon realized that few scrips were identical twins. Few scrips had huge gaps in data. All of them were NSE/BSE scrips but none of them had codes that NSE/BSE used. Codes were unique to
data provider only.
Anyways, one year later, I was determined not to renew his services and set out to reorganize data and to generally improve how it was maintained.
When subscription expired, I wrote whatever C++ was necessary
to transfer all of the data (minus their unique codes) into different
file/folder layout and more importantly, contains all codes that
are compatible with NSE & BSE and NFO.
Discovered something knew that NSE publishes Xls file that lists
renamed scrips. Not surprisingly, the old datafeed had never
bothered with it, so renamed companies would show up were
showing up as 2 separate scrips in best case or old one
un-updated and new one missing. Discovered many new scrips.
What good is clean data without daily feed?
First place I looked was dear old Bhavcopy software from Hemen Kapadia's. It is clean, free software and files that it downloads contain bare essentials and it handles all 3 biggies: NSE, BSE and F&O.
So, now I had clean data with 3 exchanges getting updated correctly.
All was well until I discovered that Data from Hemen Kapadia's software often jumbles up prices of F&O prices.
So, abandoned Hemen Kapadia for NSE and NFO and downloaded all of data from NSE archive and started all over again to create fresh-new-well-organized data.
Unlike Hemen. Bhav copy from the NSE website comes in a zip format and date is written differently. Modified whatever code that was necessary
so that it can now handle NSE as well as file format from hemen kapadia.
While at it, also experimented with data from Volume Digger.
Data from this downloads is generally same for prices but volumes seem to be different sometimes. They are miles apart when it comes to downloading
prices of Indices. NSE data contains one set of Indices. Hemen's site contains
slight variation. Volume digger contains 3rd set.
Most Unfortunately, data from NSE website does not contain EOD data
of Nifty itself whereas Hemen's does. NSE F&O contains DJIA and FTSE but not Nifty Spot. I can get Nifty's future but not the nifty spot itself.
NSE website also contains data in different format (PR files). I am guessing
that PR stands for Press Release but it could be "Paresh Raval" just as well.
as NSE has not made it clear. Unline single file bhavcopy, PR*.zip contains lots of small files and zips within zips.
Not surprisingly, it is in a format that needs a different parser and significant
energy to understand what is what.
Now, in my quest to obtain free data, I end up downloading data from Hemen Kapadia, volume digger, NSE as well BSE.
My ISP loves me for it and I have no choice but to love him back.
:clap:
Is all this round-about worth it. Should I not just pay 5000 and put up
with old format. Its not only about money, it is actually freedom from his
cheap tricks and knowing that when I see news on ACC anywhere
and try to look it up on my screen, I am 100% confident that I am
looking at same thing. Earlier, I could only be 99% sure. That 1% can
make a big difference if you actually go ahead and execute a trade.
In all of this process, also discovered yet another most valuable info.
World of Nifty is like Sambhar from Udupis. There is CNX. there is GS.
There is S&P. I cannot tell A apart from B. Shall we look ahead
for Congress NiFTy, BJP Nifty, Filmstar nIFty and Aam Aadmi NiFtY?
From a programming perspective, all of this was absolute fun . From trader perspective, this is a headache.
Ruminating my experience on this post for two reasons (a) learning about a better alternative. (b) Better Understanding of paid services and their
cheap-tricks. (c) just wasting time till dinner arrives .... :rofl:
I hope Lokmanya Tilakji will not get offended when I say "Free access to data is birthright of every trader and wannabe trader".
When I first started trading, the person who guided me (or misguided me) got me to sign up for a EOD data feed that was costing 5000 Rs a year. I received approx 10 years EOD data in MWD file format for NSE, BSE, MCX, Forex as well as US scrips.
US Scrip provided in data feed is actually waste of disk space because (a) I am not going to trade in 'em anyway. (b) Most scrip names are cryptic
Its a neat trick though. By showing so much variety of data, it is easier to fool a novice.
Forex data was rounded off at 2 digits so that waste too. It contains so many many currency pairs that I actually doubt its accuracy. Even EODData.com does not provide all those currencies.
Coming to the most important part, NSE, BSE and F&O. Data was segregated
haphazardly. NIFTY folder had 180 scrips and some of them were dead scrips.
Wrote code to list out files and scrips and soon realized that few scrips were identical twins. Few scrips had huge gaps in data. All of them were NSE/BSE scrips but none of them had codes that NSE/BSE used. Codes were unique to
data provider only.
Anyways, one year later, I was determined not to renew his services and set out to reorganize data and to generally improve how it was maintained.
When subscription expired, I wrote whatever C++ was necessary
to transfer all of the data (minus their unique codes) into different
file/folder layout and more importantly, contains all codes that
are compatible with NSE & BSE and NFO.
Discovered something knew that NSE publishes Xls file that lists
renamed scrips. Not surprisingly, the old datafeed had never
bothered with it, so renamed companies would show up were
showing up as 2 separate scrips in best case or old one
un-updated and new one missing. Discovered many new scrips.
What good is clean data without daily feed?
First place I looked was dear old Bhavcopy software from Hemen Kapadia's. It is clean, free software and files that it downloads contain bare essentials and it handles all 3 biggies: NSE, BSE and F&O.
So, now I had clean data with 3 exchanges getting updated correctly.
All was well until I discovered that Data from Hemen Kapadia's software often jumbles up prices of F&O prices.
So, abandoned Hemen Kapadia for NSE and NFO and downloaded all of data from NSE archive and started all over again to create fresh-new-well-organized data.
Unlike Hemen. Bhav copy from the NSE website comes in a zip format and date is written differently. Modified whatever code that was necessary
so that it can now handle NSE as well as file format from hemen kapadia.
While at it, also experimented with data from Volume Digger.
Data from this downloads is generally same for prices but volumes seem to be different sometimes. They are miles apart when it comes to downloading
prices of Indices. NSE data contains one set of Indices. Hemen's site contains
slight variation. Volume digger contains 3rd set.
Most Unfortunately, data from NSE website does not contain EOD data
of Nifty itself whereas Hemen's does. NSE F&O contains DJIA and FTSE but not Nifty Spot. I can get Nifty's future but not the nifty spot itself.
NSE website also contains data in different format (PR files). I am guessing
that PR stands for Press Release but it could be "Paresh Raval" just as well.
as NSE has not made it clear. Unline single file bhavcopy, PR*.zip contains lots of small files and zips within zips.
Not surprisingly, it is in a format that needs a different parser and significant
energy to understand what is what.
Now, in my quest to obtain free data, I end up downloading data from Hemen Kapadia, volume digger, NSE as well BSE.
My ISP loves me for it and I have no choice but to love him back.
:clap:
Is all this round-about worth it. Should I not just pay 5000 and put up
with old format. Its not only about money, it is actually freedom from his
cheap tricks and knowing that when I see news on ACC anywhere
and try to look it up on my screen, I am 100% confident that I am
looking at same thing. Earlier, I could only be 99% sure. That 1% can
make a big difference if you actually go ahead and execute a trade.
In all of this process, also discovered yet another most valuable info.
World of Nifty is like Sambhar from Udupis. There is CNX. there is GS.
There is S&P. I cannot tell A apart from B. Shall we look ahead
for Congress NiFTy, BJP Nifty, Filmstar nIFty and Aam Aadmi NiFtY?
From a programming perspective, all of this was absolute fun . From trader perspective, this is a headache.
Ruminating my experience on this post for two reasons (a) learning about a better alternative. (b) Better Understanding of paid services and their
cheap-tricks. (c) just wasting time till dinner arrives .... :rofl:
I hope Lokmanya Tilakji will not get offended when I say "Free access to data is birthright of every trader and wannabe trader".