Pursuit of Free data and Rumination of Determination ...

mastermind007

Well-Known Member
#1
Hello

When I first started trading, the person who guided me (or misguided me) got me to sign up for a EOD data feed that was costing 5000 Rs a year. I received approx 10 years EOD data in MWD file format for NSE, BSE, MCX, Forex as well as US scrips.

US Scrip provided in data feed is actually waste of disk space because (a) I am not going to trade in 'em anyway. (b) Most scrip names are cryptic

Its a neat trick though. By showing so much variety of data, it is easier to fool a novice.

Forex data was rounded off at 2 digits so that waste too. It contains so many many currency pairs that I actually doubt its accuracy. Even EODData.com does not provide all those currencies.

Coming to the most important part, NSE, BSE and F&O. Data was segregated
haphazardly. NIFTY folder had 180 scrips and some of them were dead scrips.

Wrote code to list out files and scrips and soon realized that few scrips were identical twins. Few scrips had huge gaps in data. All of them were NSE/BSE scrips but none of them had codes that NSE/BSE used. Codes were unique to
data provider only.

Anyways, one year later, I was determined not to renew his services and set out to reorganize data and to generally improve how it was maintained.

When subscription expired, I wrote whatever C++ was necessary
to transfer all of the data (minus their unique codes) into different
file/folder layout and more importantly, contains all codes that
are compatible with NSE & BSE and NFO.

Discovered something knew that NSE publishes Xls file that lists
renamed scrips. Not surprisingly, the old datafeed had never
bothered with it, so renamed companies would show up were
showing up as 2 separate scrips in best case or old one
un-updated and new one missing. Discovered many new scrips.

What good is clean data without daily feed?

First place I looked was dear old Bhavcopy software from Hemen Kapadia's. It is clean, free software and files that it downloads contain bare essentials and it handles all 3 biggies: NSE, BSE and F&O.

So, now I had clean data with 3 exchanges getting updated correctly. :cool:

All was well until I discovered that Data from Hemen Kapadia's software often jumbles up prices of F&O prices.

So, abandoned Hemen Kapadia for NSE and NFO and downloaded all of data from NSE archive and started all over again to create fresh-new-well-organized data.

Unlike Hemen. Bhav copy from the NSE website comes in a zip format and date is written differently. Modified whatever code that was necessary
so that it can now handle NSE as well as file format from hemen kapadia.
While at it, also experimented with data from Volume Digger.

Data from this downloads is generally same for prices but volumes seem to be different sometimes. They are miles apart when it comes to downloading
prices of Indices. NSE data contains one set of Indices. Hemen's site contains
slight variation. Volume digger contains 3rd set.

Most Unfortunately, data from NSE website does not contain EOD data
of Nifty itself whereas Hemen's does. NSE F&O contains DJIA and FTSE but not Nifty Spot. I can get Nifty's future but not the nifty spot itself.

NSE website also contains data in different format (PR files). I am guessing
that PR stands for Press Release but it could be "Paresh Raval" just as well.
as NSE has not made it clear. Unline single file bhavcopy, PR*.zip contains lots of small files and zips within zips.

Not surprisingly, it is in a format that needs a different parser and significant
energy to understand what is what.

Now, in my quest to obtain free data, I end up downloading data from Hemen Kapadia, volume digger, NSE as well BSE.

My ISP loves me for it and I have no choice but to love him back.
:clap:

Is all this round-about worth it. Should I not just pay 5000 and put up
with old format. Its not only about money, it is actually freedom from his
cheap tricks and knowing that when I see news on ACC anywhere
and try to look it up on my screen, I am 100% confident that I am
looking at same thing. Earlier, I could only be 99% sure. That 1% can
make a big difference if you actually go ahead and execute a trade.

In all of this process, also discovered yet another most valuable info.

World of Nifty is like Sambhar from Udupis. There is CNX. there is GS.
There is S&P. I cannot tell A apart from B. Shall we look ahead
for Congress NiFTy, BJP Nifty, Filmstar nIFty and Aam Aadmi NiFtY?

From a programming perspective, all of this was absolute fun :cool:. From trader perspective, this is a headache.

Ruminating my experience on this post for two reasons (a) learning about a better alternative. (b) Better Understanding of paid services and their
cheap-tricks. (c) just wasting time till dinner arrives .... :rofl:



I hope Lokmanya Tilakji will not get offended when I say "Free access to data is birthright of every trader and wannabe trader".
 

mastermind007

Well-Known Member
#2
Understanding SAR and Candles

SAR is one of the most simple Technical indicator and easy enough to plot. It usually shows up as dots above or below the candle but you could convert it into lines (if you want to confuse others):clap:

Metaphor or cat and mouse can be used to understand SAR and Candle.

Candle is the cat and SAR is the mouse. Whenever cat (candle) catches the mouse, the mouse jumps on other side of the cat direction and runs off in other direction. Mouse never gets caught and Game never ends ... :p

Another thing is that mouse is bit naughty and if gap between them is too large, the mouse will come closer to the cat. :clapping:

(Yup, Disney's Tom and Jerry are a good visual aids )
 

DSM

Well-Known Member
#3
Re: Understanding SAR and Candles

Mastermind007, from your previous post on value investing (Benjamin Graham) you seemed to be following only value investing, and did not beleive in technicals. Is there something that I am missing?

SAR is one of the most simple Technical indicator and easy enough to plot. It usually shows up as dots above or below the candle but you could convert it into lines (if you want to confuse others):clap:

Metaphor or cat and mouse can be used to understand SAR and Candle.

Candle is the cat and SAR is the mouse. Whenever cat (candle) catches the mouse, the mouse jumps on other side of the cat direction and runs off in other direction. Mouse never gets caught and Game never ends ... :p

Another thing is that mouse is bit naughty and if gap between them is too large, the mouse will come closer to the cat. :clapping:

(Yup, Disney's Tom and Jerry are a good visual aids )
 

mastermind007

Well-Known Member
#4
Re: Understanding SAR and Candles

Mastermind007, from your previous post on value investing (Benjamin Graham) you seemed to be following only value investing, and did not beleive in technicals. Is there something that I am missing?
Thanks to you, I learnt a new term (a new paradigm) today Value Investing

And yes, I wouldn't mind having investment in a scrip that is currently @ low price but has tremendous opportunity for growth.

Between fundamentals and technicals, I find technicals a lot easier. That said,
I do not believe everything that gets thrown around under the banner of technicals.

Generally, aim is to learn from ppl who have been successful and to understand what they've done so that I may imitate. Keyword here is
understand. I will not follow anyone blindly.
 

DSM

Well-Known Member
#5
Re: Understanding SAR and Candles

Great attitude dude! Surely you will go a long way!

Am reading The greatest trade ever by Gregory Zukerman. It's about the 12 billion Paul Johnson made shorting the subprime market, bonds, etc. Besides the trade, the research, psychology, the players make an interesting reading. You will love it.

Also came across an astounding fact : Benjamin Graham, the guru of Warren Buffet lost 70% of his portfolio value over 2 years when the market crashed.

Basically, one crucial ingredient in trading is timing. Those betting against the subprime were in minority, and lost money over 2 years or underperformed the market. A few threw in the towel and closed their trade, when they were closed to the market turn. Had the held on for a little longer, they could have made astounding sums of money!


QUOTE=mastermind007;791525]Thanks to you, I learnt a new term (a new paradigm) today Value Investing

And yes, I wouldn't mind having investment in a scrip that is currently @ low price but has tremendous opportunity for growth.

Between fundamentals and technicals, I find technicals a lot easier. That said,
I do not believe everything that gets thrown around under the banner of technicals.

Generally, aim is to learn from ppl who have been successful and to understand what they've done so that I may imitate. Keyword here is
understand. I will not follow anyone blindly.[/QUOTE]
 

yusi

Well-Known Member
#6
Re: Understanding SAR and Candles

Mastermind007, from your previous post on value investing (Benjamin Graham) you seemed to be following only value investing, and did not beleive in technicals. Is there something that I am missing?
Perhaps you are missing the difference between Mastermind007 and Mastertrader007. :lol:
 

DSM

Well-Known Member
#7
Re: Understanding SAR and Candles

Yes, got that. One is with pic. the other without.

Perhaps you are missing the difference between Mastermind007 and Mastertrader007. :lol:
 
#8
Hi Mastermind

Brilliant description and effort!!

Just what do the traders have to do to get some clean and accurate data? It's a shame.

But can't the encrypted data be translated to other formats?
 

mastermind007

Well-Known Member
#9
Re: Understanding SAR and Candles

Yes, got that. One is with pic. the other without.
Yeah I did not realize that too. Master mind is a concept .... Master trader is an entity.
 

mastermind007

Well-Known Member
#10
Hi Mastermind

Brilliant description and effort!!

Just what do the traders have to do to get some clean and accurate data? It's a shame.

But can't the encrypted data be translated to other formats?
Yes, as long as you can figure out format of encrypted data.
 
Last edited:

Similar threads