PyDownloader

rmike

Well-Known Member
#1
Well, the (un)expected termination of URL for historical intraday data by google on 13 Sep appears to have thrown a major 'wrench in the works', causing a fair bit of consternation amongst retail traders (to put it mildly :rolleyes:). Even though the generous offerings of TB & Josh have somewhat mitigated the extent of setback, the slipshod formulation of the nest plus api coupled with ad-hoc tampering by Thompson Reuters has led to the backfill process becoming a significantly capricious affair, especially for the spot Indices!!!

Despite other sources of data being available, google finance has always been the staple convenient mainstay for the source of good quality, reliable (and not to mention, free!!! :)) intraday historical data for backfilling and backtesting by retail traders

Came across the news that google is planning to revamp their finance offerings, come November. However, the clock is ticking on the fact that post 15 trading days after 13 Sep, the window to fill the data gap may be irretrievably lost!!!

Had heard of some python scripts on the net which (allegedly) might do the job, so decided to take a shot at rectifying the current situation!

Hence sat down last night with a nebulous idea of searching for a suitable script and tinkering by (maybe) rectifying the changed URL. All of which, ideally, ought not to have taken more than an hour or two, at the most. Turns out that I was sadly mistaken! Big time!! The couple of hours ended up in pulling an 'all nighter' ...... and then some ...... extending well into today!! Phew!!!

During the course of which I learnt how to install the python module (used 2.7), python environment, fix the environment variables, download and install the various dependencies/ libraries/ packages and rudimentary manipulation of the python IDE

All of the scrips on the net were not functional (obviously!!!). But what came as a major downer was the fact that all of them (which I could locate) were also not suitable even if I could manage to replace the URL because they pulled data only for a single ticker, and that too only to the env dataframe and not to file!!! This meant that the nebulous idea of 'minor tinkering' was misguided 'wishful thinking' to begin with!!! o_O

However, I did come across a script from 2011 which had the 'definitions' for query, parser and
write functions. I now had to look at making requisite alterations and adding a module to interpret and download a list of tickers to file. Well... the next obvious stop was online python tutorials reference to make some sense of the definitions' code and to understand what exactly was happening under the hood!!!

To cut a long tedious story short ....... after countless false starts, (I have no formal, or even informal, coding background), finally attained the Eureka Moment!!!

The resultant outcome works flawlessly on my rig. Am uploading the past 15 days of one minute IEOD for spot Nifty & BankNifty for people who may have need of it, and also as proof of concept

For sharing the utility, will now have to look at and learn how make a distributable with user input & error handling built in, as it'd be unreasonable to expect all users to have python installed on their rigs. Plus I'm seriously not upto 'hand holding' due constraints of time for people who will decidedly need guidance for installing python/ modules/ Packages/ Libraries and environment to use this 'as it is' in its present form!

Let me know if any of you guys (n gals?) here would be interested in using/ testing this out, so that I know that the effort to make a distro is useful, at least!

Will likely work on and share this by tomorrow. Presently too spaced out to tackle the job :greyalien:

Cheerio!!!

P.S - BTW, did I happen to mention it earlier? The distro's free!!! ;)

Edit - Forgot to attach the format file for import of uploaded spot indices IEOD to amibroker! Done now!!
 

Attachments

Last edited:
#2
Thanks bro, for the indices 15 days data
and I did it this afternoon, after all sort of cut, paste, add, deletion, changing format etc etc :D

Will be waiting for your work on Python.
 

rmike

Well-Known Member
#5
Usage Instructions - PyDownloader_nse_v1.0

1. This is a Python distributable utility to download 1 min historical data for NSE from Google finance. This is compiled to work on Microsoft Windows OS. Will not work for Mac and Linux OS.

2. Important – Follow all instructions regarding package and file placement. Don’t modify the name of any file or folder!!!

3. Download the zip package from the provided link, unzip the package to the root directory of your D: drive. This will place the ‘PyDownloader’ Folder in D: drive, like so “D:\PyDownloader”.

4. Inside the PyDownloader Folder, you’ll find the following :-

(a) ‘TickerList’ Text File – Enter Ticker symbols for which data is required. The sample Tickers viz. Nifty & BANKNIFTY are placed in the list as reference example. Input to list each desired Ticker in a separate line. The maximum limit of the TickerList is 100 Tickers.

(b) ‘Data’ Folder – The downloaded 1 min historical data by the utility will be written to this folder as a csv file for each individual Ticker in the TickerList. Historical 1 min IEOD can be downloaded for a maximum of 15 trading days.

(c) ‘PyD.format’ File – If you wish to manually import the downloaded IEOD into Amibroker, place this file in the ‘Formats’ Folder of your Amibroker installation. You’ll also need to edit the ‘import.types’ file in the Formats folder by inserting the following instruction.

PyDownloader (*.*)|*.*|PyD.format

(d) ‘PyDn_v1.0_Lic’ – The License file for the utility.

(e) ‘PyDn_v1.0 Help’ Pdf File – Help file.

(f) ‘PyDownloader_nse_v1.0.exe’ – The executable distro package of the utility.

pyd_1.png
 

rmike

Well-Known Member
#6
Usage Instructions - PyDownloader_nse_v1.0 (Contd.)

1. After inputting desired Tickers in TickerList, double click on PyDownloader_nse_v1.0.exe. A console window will pop up. This window will remain inactive for some while (approx. 40 to 65 seconds, depending upon your rig’s CPU and RAM configuration) during which the Python virtual environment is loaded for the utility to function.

pyd_2.png


2. After the virtual environment loading is complete, the console window becomes active. As the utility executes, it will ask the user to input the desired no. of days for which IEOD is required to be downloaded.

pyd_3.png


3. Input the no. of days as a NUMBER. For e.g Three days = 3 and NOT 03 OR Three. And Fourteen days = 14 and NOT 014 OR Fourteen. After providing input, press Enter/ Return. The utility will then execute as per the user input for no. of days and no. of Tickers in TickerList. The image below illustrates the example of downloading IEOD for 1 day for NIFTY & BANKNIFTY.

pyd_5.png


4. After the process is complete, the utility and the console window will auto exit. The IEOD can then be imported from the ‘Data’ Folder

5. This utility is distributed ‘as is’ without any cost to user for providing retail traders with means to utilize publically available data from Google Finance for analytical purposes. No user is authorized to distribute this utility without credit to the copyright holder(s). No user is authorized to commercialize this utility in any form.
 

rmike

Well-Known Member
#7
Posting the PyDownloader_nse_v1.0

This is just a very modest personal endeavour for 'Paying It Forward', in gratitude for the generosity of spirit of people like PK Jha, cooltera, Hemen, Joy, TB, Josh - over the years to current times (And many in between, whose nicks aren't mentioned for apprehensions of creating too lengthy a list which may cause a crash of forum servers ;). But they are silently acknowledged as I've been around a fair while to have known all of them) - who gave the undercapitalized retail traders the means to access publically available data for free!!!

Do Read the 'Help' before using the utility!!!

The distro works fine on the rigs which I tested, I hope it does the same on your rig too and proves to be of help

Cheerio!!!

P.S - If you wish to be kept apprised of version updates, you can choose to submit your mail id by using the 'Feedback' option while downloading the utility from my G drive
 

Attachments

#10
hi rmike ..
problem with the drive .. unfortunately i dont have D:\ drive ... only C,E,F,G ..
posting screenshot



How to rectify this?
Either change the code or change the drive letter.
I assume D will be defined for CD/DVD drive.

Edit: Go to disk management then change the drive letter to D. For doing this you must first free the D letter if it is assigned to CD/DVD drive.
 
Thread starter Similar threads Forum Replies Date
R Intraday 140

Similar threads