EOD Download/ Convert from NSE and BSE

EOD Download/ Convert from NSE , BSE and MCX (NCDEX in future)

Hi Traderji and all the Forum members

I am posting EODDwonloadConvert version 0.0.70 for forum members to

try and give their comments.Feel free to use it and give feedback. No warranties.Code is optimisation under progress.

It needs unzip32.dll which can be obtained from
http://www.mirrorservice.org/sites/ftp.info-zip.org/pub/infozip/WIN32/unz552dN.zip

It need to be placed in \windows\system ditrctory and register it.Rightclick on it will register it else use "regsvr32.exe unzip32.dll" in Run dialog of Start Button. Ignore any error. The version is unzip32.dll 5.5.2.0. . Only dll is available in zipped file in my briefcase.

The programme is compiled using vb6(sp6) and needs runtime files.Normally these are available on almost all systems now but on some older system it may not be available and give errors.You can download runtime files from here if required.

http://www.microsoft.com/downloads/...61-7A9C-43E7-9117-F673077FFB3C&displaylang=en

Its 1.0 MB download.

you need IE 5.5 or above installed.

Dependency Script is attached for your use to verify if required files are present.I have tested it on Win98SE and WINXP system . it should not give much trouble. The zipped file EODDownloadConvert.zip(142.5kb) may be downloaded from my briefcase.
http://briefcase.rediff.com/bn/br/pjhapondy

The programme should be extracted to a seperate folder of your choice. It will create folder structure and required files on first run . Settings related files are in BIN folder. My BIN folder is also placed in the Briefcase in case you may like to download some files such as Favourites.txt,holiday.txt etc. Programme icon is in BIN\images folder.

I have to thank http://www.planetsourcecode.com , www.freevbcode.com

and www.programmersheaven.org and www.info-zip.com for their code

snnipets and dll files. All thanks to respective programmers.My part was limited to certain areas and ideas specific to downloading files from nse and bse website and assemblling it to make a workable solution for members of Traderji.com

I have to thank satya whose dataextractor gave some inspiration for

this and his idea of thread to share software. My code will be freely available once it reaches version 1.0.0 and after your feed back is also incorporated.The programme has its own lifetime and utility.It may stop functioning if there are changes in URL structure of NSE and BSE websites.If newer operating system do not support older programmes. In such a situation it would need to be ported to newer versions.

What it seeks to do:-

Download:

For a given range of dates it will download bhavcopy for nse. For BSE it will download current Bhavcopy which is in zipped format. Itwill unzip the same without user intervention.You have to enter dates from and to in the field and click ADDUrl.

All URLs will be displayed in the box below with save path. As of now use default options it should be ok. For other options if there is any problem please tell me.

On the top there is option for providing download urls if you know it. If it is from BSE or NSE it will be stored in default folders. Other wise it should be available in Others folder.

Convert:

Initially use only default options.

If files are available in default folders then it will be visible in Input Files Box. Defaults are ok i.e. csv and nse or bse. Delimiter is always "," i.e. comma. leave it unchanged. You will see the first two lines of records in Select output options. If first line is header record you can check it . You can leave all other options as it is. On clicking Convert button it will convert all available file in NSE or BSE equities folder One folder will contain date wise EOD data for each date for all stocks, the other folder will contain
individual stock data for all dates available.

Datewise format is symbol,o,h,l,c,v,date
Stockwise format is o,h,l,c,v,date

First line is header line.

Perhaps these files can be used by members for importing into their
programmes. if any error is given I would like to know.

If you have downloaded many files you need to sort these records in the output files. Use sort button for this. It needs to be used for once only. thereafter if you are running your program after 5.30 or whatever time bse/nse post their latest data you dont need to sort.
I understand that bhavcopy is posted after 5.30 pm.

You can exclude any date by editing the holiday.txt file either directly. All dates are posted below respective month. Same pattern needs to be followed. the program will indicate if their is a failure of Download for whatever reason mostly URL is not available i.e. file is not found. It is trapped and failure.log is generated.

You can verify if URL exixts or not by using some other downloader.Error.log file wil keep records of two days errors and may be used for feed back.

Browser:

You can use this to browse your favourit websites wihtout overhead of IE. But you need IE 5.5 or above installed.

Some of the Links will give you direct access to nes/bse html files without going to home page. Loading will be faster. You can add favourites First input is for title and secon input is for URL. If you omit URL title can act as section header.

You can directly edit Favourites in your text editor. Odd lines will be titles and even lines should be URLs.

You can also check if your computer is connected to internet and if NSE website is up and running. Buttons are on Download page.

If this programme works it will be free and once it is optimised it will be given in source code form for maitainance by members if desired.

Future Wishlist

To optimise code.

To incorporate charts and other indicators based on the data collected on daily basis. Chart code and form is included but made invisible as further development is required.

To incorporate multi threaded downloading capability.

To take print( Presently you can print file in one go)

The name given to the executable is only tentative and members are invited to suggest better name. We may vote for name and whichever name gets highest vote will be made as programme name.


thanks are in order for Satya, joymitali,Saint, AmitBE,AMITKR murthymsr and otherboard members for inspiraton.

I dedicate this programme to the enterprising spirit of Traderji.com

and Moderator traderji.

With kind regards
Pankaj

Some Screen Shots

hi pkjha
CONGRATS! ON YOUR EXTRAORDINARY EFFORT.
i have been following u r posts for many months now.
and what a sound technical person u r are. guys like u make India a IT super power. (I am not and that is why all the more admiration)
of course may be in u r line these kind of softwares are dime a penny but for us non software guys these are great things.
i was thrilled to use Srisara's data extractor and am sure will have even better experience with this downloader.
Great work.
Keep it up

pkjha,

congrats, on the piece of excellent professional quality software. i also see the developer with a big heart behind this software!

i could succesfully download the eod files from nse as well as convert them stockwise, but with limitations. still trying to understand fully. will post/mail the observations in a few days. (My OS is WIN98 SE with IE 6.0)

i see some tabs presently without any function. i can visualise the fully blown up software as more features/options/services added to it.

i also congratulate traderji and the forum members for having a professional with a great heart like pkjha amongst us.

murthymsr

Hi ragh_ash
Thanks a lot.First sign of encouragement always makes an exciting impression. I need feedback to improve it and get rid of bugs which will be there.one thing I want to clarify is that I am not an IT guy or even trader.This idea came after I saw Satya's data exrtractor and his suggestion to have a thread where programmers knowing stock market requirement could contribute to development of a software which would be available to forum members.It is not an alternative to satya's DE2.0 which is an excellent software.
This will be free and members could contribute any modules they want.
As I promised , source code would be made available once I am reasonably certain that it meets basic requirement which ,by the time it is ver.0.1.0, it would be.

I am visiting this forum for a long time and gained a lot of insight and warmth here. This is my small contribution to the forum. hope it is worth everybody's attention.
Thanks a lot.

Hello Mr.Pankaj,

That was a nice utility you have gifted. Like you I am also not a programmer or a regular trader. But two years before I had requested a programmer friend of mine to create a utility which will download and convert Bhavcopy from NSE into CSV files. With in a few hours he created a utility in a crude form which I had been using ever since. The codes were written in Visual Basic. I use windows XP professional SP2 OS. But this utility may work with all windows OS - 98, 2000. This utility has some advantages and certain limitations which I will mention below. I am uploading the utility with its source code so that group members can compare both the utilities and make suggestions to pankaj for necessary changes in his EOD convertor.

Advantages:-

1. Will download NSE bhavcopy from NSE directly when the selected date is clicked. What the utility does is that when a Bhavcopy is downloaded it is immediately stored in the MS data access base as per the coding instructions.
2. Generating the required output can be in ascending or descending form. I included the descending form because certain TA softwares will require data in that form. For example – Best Charts. In the generate option buton you can select all the scrips or the selected ones. Just click on the scrips. A search facility is also included to find a scrip’s name or symbol easily. Currently you cannot generate output simultaneously in both ascending and descending forms. This problem can be solved easily since my programmer friend has already done it but in SQL Server 2000.
The button delete old generated file was included because to ensure that any corruption to outputdata generated earlier will not affect the new output.
3. The settings button is used to specify the output folder where you want data to be generated. Any number of folders in any directory can be specified. Example- E:\CSV data or D:\NSE Data etc or in any directory.
4. The file extension is csv. So simply type csv in the file extension blank space and press the apply button below
5. The projection data was a function I had requested for the following purpose. For example while analyzing scrip say in Amibroker you get a buy signal. But you feel the buy will be a good one only if the Moving averages cross over or say RSI has cut its MA. While analyzing the scrip the MAs or RSI may not have crossed its MA. So you would like to know at what price these cross over will happen. So in the projection data you can enter all the imaginary or projected price fields and volume to generate a possible scenario. A safety future has been included so that the existing down loaded data is not corrupted. So next day when you download the bhavcopy the projected data fields are automatically overwritten. I think the safety future will give a warning message box if you try to alter downloaded data. There is a delete facility for projected data entered. I am sorry to say now a days I do not use this function much.


Disadvantages

1. Since the utility uses MS Access for data storage there are certain limitations. In MS Access there is a limit to the capacity to store data. Currently my data storage in MS Access of NSE data from year 2000 to current date amounts to 220 MB. I have been told usually MS Access supports storage up to a maximum of 75-100 MB. Presently the utility is working bit slower and takes a little more time to generate the output may be because I generate data for all the scrips from year 2000. But if only a few scrips and dates are selected it runs very fast. I also feel it runs very fast on powerful computers. I use a laptop with Celeron chip and 256 MB memory.
2. But recently we tried the utility with Microsoft SQL server2000 instead of MS Access and it is really fast with unlimited data storage capacity. For example the utility will generate both the ascending and descending output data simultaneously in 2 or 3 seconds and the downloading of bhavcopy will take another 1-2 seconds. So in 5 seconds you will have the data output from year 2000 to the current date in both ascending and desending forms of all the scrips in NSE. SQL server when installed will always be quietly running in the background without using additional system resources. There is a free version of SQL server 2005 named express edition which can be downloaded from Microsoft site. This server supports data storage capacity up to 4 GB.

So I request you to try SQL server also. But there will be a problem while sharing with the group because the codes have to be written in SQL which requires the services of a programmer with good knowledge of SQL and database applications. And also the group members will have to use SQL server. Well it is only a suggestion. I suppose we will have to stick with MS Access. May be try to find out a way to increase MS Access storge capacity and to make it work faster. If you need data for 3-5 years this utility will more than suffice.

3. Another disadvantage of the utility is while it was being created we had included under the equity table in the MS Acess data base the list of scrips and its other fields including the NSE ID code number unique to a scrip. Go to the installation folder double click on MS Data access file named stock quotes. There you will find a table named equity. Double click on it and you will find price and other field details of scrips.
4. So say while output data of ACC is being generated the programme will cross check ACC symbol with its NSE ID code number in the symbol macro column of equity table to ensure that no wrong symbol data is generated. Although this is a good security check it has certain drawbacks.
5. When new scrip is being traded at NSC its ouput will not be generated until the details of the scrip is included in the MS Access equity table. The search facility also will not list the new scrip. By altering the code little bit the programme can be instructed to generate output on symbol basis. So it will generate all scrips in the bhavcopy downloaded. This can be easily done by any VB programmer. Or anybody can cut and paste the details of new scrip from bhavcopy or the list of scrips traded being provided by NSE to the MS Access equity table. Do not forget to save the changes. And also do not leave any blank lines in between data.
6. The utility has no merge facility. For example Tisco has changed name to Tatasteel. The utility will generate both Tisco and Tatasteel. This is OK if you are using Amibroker or any other TA software which has a merge facility. I feel this function ought to be included in the Data convertor itself since many of the Free TA programmes do not have merge facility. And this is certainly possible.
7. The output file generated is a single file for each scrip consisting of data for dates selected. Say for example for ACC the output is a single file which will consist of data from year 2000 to the current date. Probably the utility will work even faster if output is generated for a single day. But then what happens if older generated data gets corrupted. Taking backups regularly may not be easy for all members. May be you can work out a solution.

8. Another thing I agree with you is regarding the NSE and BSE links of Bhavcopy downloads. There is always a possibility of NSE or BSE redesigning their websites and URLs could change. I feel the best solution is to customize the utility by giving a provision to the members to enter the new URL. Separate space be provided for entering new URLs for equities and futures and BSE. The URLs may change but NSE or BSE may not change the structure of Bhavcopies.

9. Another problem is regarding the NSE index datas. I am not sure NSE bhavcopy includes index details. May be all the group members write a request to NSE they will do it. It will be very easy for NSE to include index details in bhavcopy. I think BSE bhavcopy has sensex details. Currently I think only the PR Zip of NSE has index details. PR is for press publications.

10. I also feel any data conversion utility should include NSE futures and options data generation facility. The only problem will be regarding the merger of data after the expiry date. But this can be done. May be any of the group members can suggest a method to merge future and option datas after expiry date. The merge is required because many of the TA softwares will require a minimum number of data to plot charts correctly. For example in ACC futures there is current month, near month and far month futures. How are we going to merge data of current month after its expiry date.

11. Auto generation facility immediately after download of Bhavcopy can be looked into. So it becomes a single click utility.
12. A single Exe. Installation setup file would be very easy and helpful. Those is just click and install the utility.

The above are only suggestions.


The download link is rapidshare and file size is around 5 MB.

http://rapidshare.de/files/15994041/NSE_Down_Loader.zip.html

Group members may try the utility by downloading and generating data for its speed and other functions. Any suggestions or ideas are very much welcome and please post your replies to forum only so that other members also can come forward with suggestions. If we all pool together our resources we can create a great utility. I would prefer that any new ideas or changes be made to EOD convertor since it is a new application. Mr. Pankaj I have not yet tried out your EOD convertor . But looking at the JPG. Pictures of it I feel it will be really good. I do not get much time to concentrate on trading due to my occupation in the profession of Chartered Accountancy. Any way I will let you know my feedback soon.

My e-mail ID is [email protected].

Regards,

Saji Oommen

Hi saji
Thanks a lot for encouraging words.
The EOD programme is just a seed to enable the forum members to contributr to development of an utility as per their requirement.

I will have a look at PR.zip and see if I could incorporate it.Should not be too difficult.
BSE now a days does not give EOD historical data. But daily bhav copy is given
If the format is ok then it should not be difficult to maitain daily eod data for bse. Just run and convert. All the field choices and their order will be made by user with few preset choices. this is being worked on.

I need suggestions on user interface also.

Currently I am sorting in ascending order of date. it should not be difficult to sort either on volume or on any price field. But what is the use I dont know. Mostly data is required in timeline only.

Some of the suggestions given by you are very good and would require considerable efforts by members.
Mostly data needed is in flat format ot ascii text files containing ohlcv and date fields. Text files are certainly cumbersome.MS Sql may need more programming expertise and time for this. It may not be freely available. Yes, mysql or postgre sql are another alternatives.
My initial target is to stabilise it and sortout some of the bugs which i am in the process and then speed up the general operations. It ran nicely under win98 pentium 266,128mb ram and winxp athalon2800 1GB ram.
presently memory required is around 4-9 mb but i will reduce it by removing dead codes and optimising variables etc.

This will be able to download derivatives from FO segment and the options are provided. I have not worked on conversion yet but will do. I am not much aware of technical requirement of derivatives data. But will work on it.

If the bin folder provided for download it should not give much trouble. But I am working on to remove any dependency on external files.All files can be generated on the first run and can be maintained thereafter.

I have stored symbols information in separate txtfile and will work on it to provide some functions such as search, retrieve, plotting. A seperate module for math calculation will be worked out where user can give his fomula and plot. If possible all indicators will be provided but first I have to get a class module going for that.

Being first time effort it might disappoint initially but hopefully it should hold its ground once it reaches ver0.1.0

Your ideas will be kept in mind and will be revisited again and again.
regards
Pankaj

Hi Murthy

Thanks a lot. If you could tell me the limitations i will attend to it.Lot of thing are made invisible as they are in the process of development. By the way I must clarify that I am not an IT professional. I am using computer since 386mhz days. I still remember when I requested for 1 GB hard disk I was informed that it may not be required. My present computer is having 1GB RAM as against 8 MB . What I essentially believe in is that the programme must be able to cater to the requirement of its user or it will fail.
I know you are yourself an acoomplished person. I am following your thread on daily stock list.

Thanks a lot
regards
Pankaj

Hi,
I tried to install the software in the workstation. I am on windows XP Professional SP2, but getting error message and it is not working. I followed the instruction provided .
I am attaching the error message file.
Please suggest where I have gone wrong.

Regards,

Sarit

Hi saritguha

It is an executable so nothing to install except vb6 run time files and unzip32.dll in \windows\system directory.I you click the programme second time it should run. This problem is because empty error.log file. This has been corrected on my system. Update will be posted by SUNDAY. However alternatively delete your BIN folder and copy the BIN folder provided by me.

The programme should run fine. Type mismatch was in error handler which is given in all subroutine to catch errors to avoid crash of programme.

On your system it will run fine. Do report back

I am also followed your post in another thread and will try to incorporate those requirement.

Pankaj