EOD Download/ Convert from NSE and BSE

Hi Murthy

Your feedback has kept me quite busy. I will post responses to them. I believe that nothing is impossible. It is all a matter of time and effort withinn one's capabilities to achieve anything in life.

Your most of the suggestions will come in the next revision.

However I want one suggestion from you. I am holding Download queue in a listview box which is limited to 256 items. It could also directly call the Download routine without going through ListView buffer. Foe this reason I provided the limit of one year gap in from and to dates.

If buffer is not used routine will work much faster and without limit as URLs are created on the fly.

Appreciate your feedback
Thanks in advance
Regards
Pankaj

hi pankaj,
i have downloaded ur software but am still reading the various posts .
in fact i print all the posts in the thread to read and understand.
actually most of my queries are usually answered by some member or other so it becomes kind of superflous to ask same thing again.
all i can give is words of encouragement or ask some minor thing that i donot understand.
Once again many congratulations!

hi pankaj,

quote:
If buffer is not used routine will work much faster and without limit as URLs are created on the fly.
:unquote

i feel, you can avoid using the buffer and make the program run faster. you may add a note to user to limit the download period to less than a year.

except in the first run and may be in the unfortunate event of data loss due to hard disk reformatting, most of the downloads will be for a period of less than a month.

to avoid data loss, i install only the OS and the other SW needing installation on one partition and all other SW/Data/downloads on the other partition. This keeps me tension free on a system crash. Rather, i periodically format the OS partition and reload windows. i know this para is NOT pertinent to this thread. apologies.

murthymsr

Hi Pankaj,

Thank you for creating this beautiful utility. I would like to give some feedback on it. I am not a technical person. So my feedback would entirely be from the end user's point of view.

1) First and most important, my Metastock Downloader (7.21) cannot read the converted text files to convert them into Metastock format. The needed format is:

<ticker>,<name>,<date>,<open>,<high>,<low>,<close> ,<volume>
526921,21ST CEN.MGM,20060324,40.5,41,39.25,39.9,53529
532628,3I INFOTECH ,20060324,187.55,188.95,181,182.15,106497

Alternatively

3IINFOTECH,20060324,186.7,187.4,181.1,182.2,232548
3MINDIA,20060324,1509,1511,1460.05,1471.55,31168
(This later one is the output format of Satya's DataExtractor(Version 2). Here the header line is dispensed with. Downloader can read and convert it. If the output data is in this format, it can probably be read by all the major stock softwares available.)

2) The converted BSE text file (datewise) does not contain BSE numerical ticker symbols. Only the names are given. Tickers are essential. It would be best to provide tickers in both datewise and stockwise text files in the format provided above. Nor does the datewise file contain date. The stockwise files contain dates, but the format is wrong (2024 for year and 06 for month):

OPEN,HIGH,LOW,CLOSE,NO_OF_SHRS,DATE
40.50,41.00,39.25,39.90,53529,03/06/2024

3) The "Skip Header Line" in the convert dialogue function does not work.

4) After a file download has been "COMPLETED" the "Stop Download" button does not change automatically. I have to click it to change it to "Download". Then a screen pops up and I need to click OK. Only then can I proceed with the next download.

5) In the Download page when I choose BSE or NSE and click the "Add URL" button, I get this message: "Error 62(Input past end of file) in procedure holiday of Module basGeneralFunctions". This, however, does not seem to matter as after clicking OK the program runs smoothly.

There are certain other issues but these are the main ones that I am facing as a user.

The main thing is the converted text file format. Once that issue is sorted out, this software would be really cool.

Thanks again for your selfless effort.

Debraj

hi pankaj,

quote:
Hi Murthy
Your feedback has kept me quite busy.
:unquote
But i can't but give my next set of observations.
i am sorry if i am putting you on an endurance test.

1) while checking connection, i get the coment, "Your connection is not properly configured." even after i configured my RAS, as suggested by the program. of course neither RAS not being configured, nor the the present comment are coming in the way of data download or conversion. These may unnecesarily raise alarm in the mind of user and needs lot of support from developer, subsequently. This checking routine may be dispensed with if it saves any resources or, may be shifted to insside one of the tabs, may be next to where you are presenting the system information.

2) I could successfully download the NSE equities in dbf format. Even a foxpro/dbf user like me does NOT download in dbf format as each file size is 3 times larger and prefer to use the command "APPEND FROM" csv file. i feel this can be dispensed with even if it does not save any resources. just think it over.

3) NSE derivatives data was succesfully downloaded and the file contents are ok.

4) Stockwise convertion of NSE equity data is OK.

5) user has to select very carefully the the source and destination folders independently, as any mistake may delete the files. anyway ultimately you may save environment and restore on next use. while doing so, yo may save the source and destination files for equities/derivatives of NSE/BSE as pairs so that once i select NSE equities or other options, the corresponding source and destination files are restored together.

6) and now i respond to debraj's post on the desired format.

of course metastock is a standard format and many software use that format. but neverthless, different software use different formats. sometimes data vendors give technical analisys software free. it uses a special format and data in that special format is sold by the data vendor. we can make provision for a user defined format where the user can select, apart from the standard formats, one of his choice, wherein he can choose the sequence of fields and also the date format in all possible combinations. even if this not done at this stage, provision can be made to add this feature later. Pankaj, if you go through the requests/complaints of users of satya's DataExtractor, after the initial period of stabilisation, it was only the data format. so it is worth the efforts to plan well and make such a provision.

for BSE data stock code is a must as stockcode is the unique ID for the stock. the source selected may contain this. company name can be included in the output only if the source data contains it. NSE bhav copy does not contain this field ( of course PR.zip contains this field, but it is a bit more involving and may thought of at a later stage.

7) i have not gone into full datails of the data download/convert sections of the utility. as i use my own data converter, for the purpose of my screener, i have not inputted the converted data into TA software, but will do it later.

all the best.

murthymsr

Hi DebRaj
Thanks for appreciation. It is in devlopment stage and
your feedback have been noted. Some are being implemented for sunday release others for next sunday.
My advice would be to just test it and not to use for regular work.
As for File format I am working on it and some ideas seems to work. User will have the option to select the desired output format in the Listview box once header fields are read. You will have the freedom to change the sequence to any method you like. It will be saved and you will be prompted to use the same format or create a new on. Working on it...
I do not know formats for available softwares but I recon they must work on the data provided by the Exchanges. Will need feedback on that. Proprietary software will keep their data in binary format or at least do not give the extensions on the format they use internally in the programme. Binary reading is possible and data format can be converted directly to desired software format such as metastock or FC Charts. but a bit more involved work is required. Flat files works best on most system as noted by satya.

Other things will also be sorted out.
thanks again.
Pankaj

Hi Murthy

you recommendations are going to the list of tasks to do and is being completed one by one.

For internet connection I also faced the same situation. I would like to point out that these two buttons for checking connections just report the status. If red circle is shown and still you try to dwonload the downloader will try and report file not found after time out. I did not want to incorporate with the downloader as it was not essential as long as your Internet explorer or any such software is working. So it is independent in its working. Perhaps at a later date might need to incorporate in to main functioning.

Regarding calender it is a standard module for Windows. I have not yet explored its full functionality. Will consider at a later stages when automation and companies announcement news board meetings etc are also included.

Sundays and saturdays are automatically leftout. Holidays are searched from the text file holiday.txt. If holidays are not entered it reports file not found after failed attempt.

Working weekends are few and far those urls can be typed in the text box and added to the list.

As a caution I have first used error module so as to avoid crash and allow the program to end normally. That is the reason why it still holds even if some erros are obviously there in the logic sequence.

your commnets on saving files is being analysed for imlementation. It has its own folder structure which is implemented by default and subsequently saved in application.ini. User choice is saved if default folder is unchecked. But EODDate and EODStock folders will still get created for storing seperate files. For derivatives same thing will happen. I might need specific recommendations for folder structure and saving options.

dbf files were just provided for the hack of it. It will not be converted. Somebody might use it without conversion. If not needed will drop at a later date, just need to delete the control

Reserve other remaks for later reply.
let me get back to fine tuning it.

regards
Pankaj

Hi debraj

Could you tell me what was the date enetered
6/3/2006 or 24/3/2006
Date need more work ....
I was thinking that first line should indicate header in case standard formats are not used.However I will remove header lines if skip header is checked( will be enforced strictly). However header record will be saved with stockfile name paired together if needed.(is it required at all????

Pankaj

hi pankaj,
quote:
Regarding calender ..... at the later stages when automation and companies announcement news board meetings etc are also included.
:unquote

i made a program in foxpro which which uses PR.zip for NSE announcements, board meetings, financial results schedule (all datewise and stockwise info) for study by you and taking the logic and integration into the current program at a later date, when you come out of the present stage of the project, only if you like it.

best wishes.
murthymsr

Hi Pankaj,

1) The date of the BSE Bhavcopy was 24th March 2006. After conversion the stockwise text files read thus:

OPEN,HIGH,LOW,CLOSE,NO_OF_SHRS,DATE
40.50,41.00,39.25,39.90,53529,03/06/2024

The same thing happened with converted bhavcopies of other dates as well. For example (Bhavcopy of 20th March 2006):

OPEN,HIGH,LOW,CLOSE,NO_OF_SHRS,DATE
177.80,189.00,177.50,187.45,503172,03/06/2020


2) Would it be possible to put the headerline in this format:

<ticker>, <name>, <date>, <open>, <high>, <low>, <close>, <volume>

Without this particular headerline Equis Downloader cannot read and convert BSE Bhavcopy into Metastock readable format. This is because BSE tickers are numerical. MS Downloader cannot recognise numerical tickers as tickers without the headerline. If this headerline is not there, Downloader reports error.

Satya's DataExtractor dispenses with headers altogether. This works for NSE Bhavcopy as NSE tickers are alphabetical. But for BSE Bhavcopy the headerline is a must for the above reason.

The format that works for BSE Bhavcopy (This contains the headerline):

<ticker>,<name>,<date>,<open>,<high>,<low>,<close> ,<volume>
526921,21ST CEN.MGM,20060324,40.5,41,39.25,39.9,53529
532628,3I INFOTECH ,20060324,187.55,188.95,181,182.15,106497
523395,3M INDIA LTD,20060324,1512.95,1513,1460,1460.15,2632

The format that does not work for BSE bhavcopy(This does not contain the headerline):

526921,21ST CEN.MGM,20060324,40.5,41,39.25,39.9,53529
532628,3I INFOTECH ,20060324,187.55,188.95,181,182.15,106497
523395,3M INDIA LTD,20060324,1512.95,1513,1460,1460.15,2632


3) After conversion is over I get the message: "Error 13 (type mismatch) in procedure cmdConvert_Click of Form frmConvert". After clicking OK the software does not shut down or anything.

I look forward to your Sunday update.

Debraj