Possibly the purest database (NSE)

rmike

Well-Known Member
#1
Dear Members,

In every forum which I have had the privilege to be a member of, the most common lament I have come across is the lack of a split, bonus adjusted database. The importance of a clean database cannot be overemphasized as it is the datum on which to build upon your trading plan and data interpretation tools. An impure or incorrect database will lead you to incorrect deductions leading to very real and major losses in the markets.

Therefore it gives me real pleasure to contribute something which I consider very important and worthwhile to the Traderji community. I am uploading the NSE database. This database is adjusted for splits and bonuses, has been corrected for symbol change and mergers and also has OI data. (Took some work, I can tell you :))

The best part about this database is that it's in manageable chunks of one year each for the past five years till 25th July 08, which will be more than sufficient for any self respecting backtester. It's been uploaded in the text format after being exported from ami, therefore the total size is a mere 90 mb! You can also elect to download data only for a couple of years as suiting your requirements, each year's worth of data would only be ~ 15 mb or so.

The links are as shown and are self explanatory

http://www.ziddu.com/download/1882289/01Jul03-30Jun04.txt.html
http://www.ziddu.com/download/1882270/01Jul04-30Jun05.txt.html
http://www.ziddu.com/download/1882250/01Jul05-30Jun06.txt.html
http://www.ziddu.com/download/1882181/01Jul06-30Jun07.txt.html
http://www.ziddu.com/download/1767120/01Jul07-25Jul08.txt.html

Alternate Links - (For the past 2 years as a start)
01 Jul 06 - 30 Jun 07 http://uploads.bizhat.com/file/252455
01 Jul 07 - 18 Aug 08 (Corrected for KNL full name and updated tilll 18 Aug 08) http://uploads.bizhat.com/file/265675

You can ideally take it from here for further database maintenance or I would be amenable to upload further updates every three months - Depends upon member's choice and responses.

There will be companies like Bajaj-Auto RPG Life and Sundaram Clayton which will still indicate splits when analyzed with database purify, however this is due to the fact of trading being ceased in the companies and a new share price having been decided due to restructuring. You can opt to delete the data prior to the new share price for these companies as the price dynamics have changed, however I have chosen to upload the previous data for reference.

The procedure for import into ami is to pick up the database with import wizard and choose headers as,

TICKER, FULL NAME, DATE(YMD), SKIP, O, H, L, C, V, OI and most important Do not skip the first line ie put value of skip line as 0

and save the format file as ****.format (your choice of name!)

The values of OI are only for recent years (2 years back), which are all I could find.

No work is accomplished in isolation. In this regard I would like to place on record my regards to Karthik and Murthy who kickstarted this quest with their gem of a thread 'Amibroker Database For NSE Stocks'. For this reason I would urge all who have not referred to the thread as yet, to peruse it to learn further database mangement tenets. The link is,

http://www.traderji.com/amibroker/8686-amibroker-database-nse-stocks.html

I would also urge you to refer another nice thread by TaNewbie

http://www.traderji.com/fundamental-analysis/11898-basic-fundas-all-cos-1-excel-sheet.html

This will enable you to import funda data into your database.

The further structuring of database into sector and industries can be done in two ways, one is by Karthik's thread wherein he is working on the update for sectors and industries (follow the procedure as explained). Or another way is to use this nice thread by Adheer,

http://www.traderji.com/amibroker/16229-nse-indices-industry-watchlists-amibroker.html

And structure sectors and industries as decided by NSE on the watchlists, using the Add to Composite function for generation of synthetic sector/industry charts (Will explain it if anybody desires so).

The usage of OI data is nicely explained in this thread by TaTrader

http://www.traderji.com/technical-analysis/227-understanding-volume-open-interest.html

So, the convergence of all these threads will ensure that you have a pure database with Funda data and a sector/industry structure . And the means to build viable robust trading systems by using the priceless information right here in this forum.

So go and conquer the markets!!!!:)

Regards,
 
Last edited:
#5
I am getting an valid date format error.
used the last file with the amibroker unregistered version.

------
Log file output:
Logging started for 'C:\Downloads\01Jul07-25Jul08.txt' file, using format definition file 'Formats\wizard.format'
Error in line KNL,KNOLL, INC,20070702,00:00,19.50,19.68,18.51,18.85,324250.00,20746
Invalid date format/value
Error in line KNL,KNOLL, INC,20070703,00:00,18.60,19.80,18.03,18.96,1244310.00,76719
Invalid date format/value
-----

I used YMD as date format.

Any suggestions

Thanks
 
#6
Hi rmike,

I successfully downloaded 3 files (5-6,6-7,7-8). I tried to convert the data in metastock format with the help of downloader but I am getting the error message:-
"error #814 : invalid source file format."
I didnot understand this error message.

Presently I am downloading data using "Get bhavcopy" and very successfully converting the same into metastock format on daily basis. Here I am giving the format of both of the datas and request seniors and knowledgiable members to suggest remedial measures.

GET BHAVCOPY DATA:-
3IINFOTECH,20080812,114,114.95,112.1,113.4,86218
3MINDIA,20080812,1612,1614,1575,1583.3,353
AARTIDRUGS,20080812,56,57.5,55.8,56.5,11388
AARTIIND,20080812,47,47,45.95,46,52016
AARVEEDEN,20080812,73.95,77.95,73.95,75.15,7780
ABAN,20080812,2529,2535,2380,2387.4,274860
ABB,20080812,882,888,858.2,878.25,403289
ABGSHIP,20080812,381,381,362,373.05,289699
ABHISHEK,20080812,34.15,34.15,31.5,32.5,4222
=====================================

RMIKE'S DATA:-
3IINFOTECH,3IINFOTECH,20050701,00:00,44.50,45.00,44.03,44.35,156544.00,0
3IINFOTECH,3IINFOTECH,20050704,00:00,44.35,45.50,44.25,44.63,262384.00,0
3IINFOTECH,3IINFOTECH,20050705,00:00,44.70,45.50,44.35,44.53,187172.00,0
3IINFOTECH,3IINFOTECH,20050706,00:00,45.00,46.47,45.00,46.28,749732.00,0
3IINFOTECH,3IINFOTECH,20050707,00:00,46.50,46.78,45.05,45.25,284500.00,0
3IINFOTECH,3IINFOTECH,20050708,00:00,45.30,46.08,44.95,45.30,142070.00,0
3IINFOTECH,3IINFOTECH,20050711,00:00,45.83,46.30,45.15,45.40,183670.00,0
3IINFOTECH,3IINFOTECH,20050712,00:00,45.50,45.65,44.85,45.17,137376.00,0
===========================================

In "get bhavcopy" the bhavcopy is for the single date while in rmike's text file the data is for the whole year in single file. Each symbol is having the data for the whole year and then the next symbol comes and the process repeats.

BASUDEO
 

rmike

Well-Known Member
#7
I am getting an valid date format error.
used the last file with the amibroker unregistered version.

------
Log file output:
Logging started for 'C:\Downloads\01Jul07-25Jul08.txt' file, using format definition file 'Formats\wizard.format'
Error in line KNL,KNOLL, INC,20070702,00:00,19.50,19.68,18.51,18.85,324250.00,20746
Invalid date format/value
Error in line KNL,KNOLL, INC,20070703,00:00,18.60,19.80,18.03,18.96,1244310.00,76719
Invalid date format/value
-----

I used YMD as date format.

Any suggestions

Thanks
Dear nfd00,

None at the present, as per the log file output the date is ok and appears to be in the YMD format. Will have to check text file manually. Will get back to you.

Regards,
 
Last edited:

rmike

Well-Known Member
#8
Hi rmike,

I successfully downloaded 3 files (5-6,6-7,7-8). I tried to convert the data in metastock format with the help of downloader but I am getting the error message:-
"error #814 : invalid source file format."
I didnot understand this error message.

Presently I am downloading data using "Get bhavcopy" and very successfully converting the same into metastock format on daily basis. Here I am giving the format of both of the datas and request seniors and knowledgiable members to suggest remedial measures.

GET BHAVCOPY DATA:-
3IINFOTECH,20080812,114,114.95,112.1,113.4,86218
3MINDIA,20080812,1612,1614,1575,1583.3,353
AARTIDRUGS,20080812,56,57.5,55.8,56.5,11388
AARTIIND,20080812,47,47,45.95,46,52016
AARVEEDEN,20080812,73.95,77.95,73.95,75.15,7780
ABAN,20080812,2529,2535,2380,2387.4,274860
ABB,20080812,882,888,858.2,878.25,403289
ABGSHIP,20080812,381,381,362,373.05,289699
ABHISHEK,20080812,34.15,34.15,31.5,32.5,4222
=====================================

RMIKE'S DATA:-
3IINFOTECH,3IINFOTECH,20050701,00:00,44.50,45.00,44.03,44.35,156544.00,0
3IINFOTECH,3IINFOTECH,20050704,00:00,44.35,45.50,44.25,44.63,262384.00,0
3IINFOTECH,3IINFOTECH,20050705,00:00,44.70,45.50,44.35,44.53,187172.00,0
3IINFOTECH,3IINFOTECH,20050706,00:00,45.00,46.47,45.00,46.28,749732.00,0
3IINFOTECH,3IINFOTECH,20050707,00:00,46.50,46.78,45.05,45.25,284500.00,0
3IINFOTECH,3IINFOTECH,20050708,00:00,45.30,46.08,44.95,45.30,142070.00,0
3IINFOTECH,3IINFOTECH,20050711,00:00,45.83,46.30,45.15,45.40,183670.00,0
3IINFOTECH,3IINFOTECH,20050712,00:00,45.50,45.65,44.85,45.17,137376.00,0
===========================================

In "get bhavcopy" the bhavcopy is for the single date while in rmike's text file the data is for the whole year in single file. Each symbol is having the data for the whole year and then the next symbol comes and the process repeats.

BASUDEO
Dear Basudeo,

Though am not a meta user but looking at the data posted, if you delete 2nd and 4th column of my data then it will fit the format used for import into meta. The error you are getting is essentially a format incompatabilty ie your import is formatted differently. If meta is anything like ami, then the other option is to format the import according to the columns as in the data posted by me.

Regards,
 

rmike

Well-Known Member
#9
Dear nfd00,

Error in line KNL,KNOLL, INC,20070702,00:00,19.50,19.68,18.51,18.85,324250. 00,20746
Invalid date format/value

Thanks a lot for noticing this and giving feedback. Please delete the INC,
portion including comma and replace KNOLL with Karuturi Network Ltd.

Same goes for all members who have downloaded this file.

Really Sorry for the inconvenience.

Regards,
 
#10
Dear nfd00,

Error in line KNL,KNOLL, INC,20070702,00:00,19.50,19.68,18.51,18.85,324250. 00,20746
Invalid date format/value

Thanks a lot for noticing this and giving feedback. Please delete the INC,
portion including comma and replace KNOLL with Karuturi Network Ltd.

Same goes for all members who have downloaded this file.

Really Sorry for the inconvenience.

Regards,

do we have to remove "inc," and replace knoll by karuturi network ltd.from all lines or just from 20070702?
and if its only for 20070702 then it will make a lot of differance to other lines....
 
Thread starter Similar threads Forum Replies Date
P Equities 11

Similar threads