Well, the (un)expected termination of URL for historical intraday data by google on 13 Sep appears to have thrown a major 'wrench in the works', causing a fair bit of consternation amongst retail traders (to put it mildly 
). Even though the generous offerings of TB & Josh have somewhat mitigated the extent of setback, the slipshod formulation of the nest plus api coupled with ad-hoc tampering by Thompson Reuters has led to the backfill process becoming a significantly capricious affair, especially for the spot Indices!!!
Despite other sources of data being available, google finance has always been the staple convenient mainstay for the source of good quality, reliable (and not to mention, free!!!
intraday historical data for backfilling and backtesting by retail traders
Came across the news that google is planning to revamp their finance offerings, come November. However, the clock is ticking on the fact that post 15 trading days after 13 Sep, the window to fill the data gap may be irretrievably lost!!!
Had heard of some python scripts on the net which (allegedly) might do the job, so decided to take a shot at rectifying the current situation!
Hence sat down last night with a nebulous idea of searching for a suitable script and tinkering by (maybe) rectifying the changed URL. All of which, ideally, ought not to have taken more than an hour or two, at the most. Turns out that I was sadly mistaken! Big time!! The couple of hours ended up in pulling an 'all nighter' ...... and then some ...... extending well into today!! Phew!!!
During the course of which I learnt how to install the python module (used 2.7), python environment, fix the environment variables, download and install the various dependencies/ libraries/ packages and rudimentary manipulation of the python IDE
All of the scrips on the net were not functional (obviously!!!). But what came as a major downer was the fact that all of them (which I could locate) were also not suitable even if I could manage to replace the URL because they pulled data only for a single ticker, and that too only to the env dataframe and not to file!!! This meant that the nebulous idea of 'minor tinkering' was misguided 'wishful thinking' to begin with!!!
However, I did come across a script from 2011 which had the 'definitions' for query, parser and
write functions. I now had to look at making requisite alterations and adding a module to interpret and download a list of tickers to file. Well... the next obvious stop was online python tutorials reference to make some sense of the definitions' code and to understand what exactly was happening under the hood!!!
To cut a long tedious story short ....... after countless false starts, (I have no formal, or even informal, coding background), finally attained the Eureka Moment!!!
The resultant outcome works flawlessly on my rig. Am uploading the past 15 days of one minute IEOD for spot Nifty & BankNifty for people who may have need of it, and also as proof of concept
For sharing the utility, will now have to look at and learn how make a distributable with user input & error handling built in, as it'd be unreasonable to expect all users to have python installed on their rigs. Plus I'm seriously not upto 'hand holding' due constraints of time for people who will decidedly need guidance for installing python/ modules/ Packages/ Libraries and environment to use this 'as it is' in its present form!
Let me know if any of you guys (n gals?) here would be interested in using/ testing this out, so that I know that the effort to make a distro is useful, at least!
Will likely work on and share this by tomorrow. Presently too spaced out to tackle the job
Cheerio!!!
P.S - BTW, did I happen to mention it earlier? The distro's free!!!
Edit - Forgot to attach the format file for import of uploaded spot indices IEOD to amibroker! Done now!!
). Even though the generous offerings of TB & Josh have somewhat mitigated the extent of setback, the slipshod formulation of the nest plus api coupled with ad-hoc tampering by Thompson Reuters has led to the backfill process becoming a significantly capricious affair, especially for the spot Indices!!!Despite other sources of data being available, google finance has always been the staple convenient mainstay for the source of good quality, reliable (and not to mention, free!!!
intraday historical data for backfilling and backtesting by retail tradersCame across the news that google is planning to revamp their finance offerings, come November. However, the clock is ticking on the fact that post 15 trading days after 13 Sep, the window to fill the data gap may be irretrievably lost!!!
Had heard of some python scripts on the net which (allegedly) might do the job, so decided to take a shot at rectifying the current situation!
Hence sat down last night with a nebulous idea of searching for a suitable script and tinkering by (maybe) rectifying the changed URL. All of which, ideally, ought not to have taken more than an hour or two, at the most. Turns out that I was sadly mistaken! Big time!! The couple of hours ended up in pulling an 'all nighter' ...... and then some ...... extending well into today!! Phew!!!
During the course of which I learnt how to install the python module (used 2.7), python environment, fix the environment variables, download and install the various dependencies/ libraries/ packages and rudimentary manipulation of the python IDE
All of the scrips on the net were not functional (obviously!!!). But what came as a major downer was the fact that all of them (which I could locate) were also not suitable even if I could manage to replace the URL because they pulled data only for a single ticker, and that too only to the env dataframe and not to file!!! This meant that the nebulous idea of 'minor tinkering' was misguided 'wishful thinking' to begin with!!!
However, I did come across a script from 2011 which had the 'definitions' for query, parser and
write functions. I now had to look at making requisite alterations and adding a module to interpret and download a list of tickers to file. Well... the next obvious stop was online python tutorials reference to make some sense of the definitions' code and to understand what exactly was happening under the hood!!!
To cut a long tedious story short ....... after countless false starts, (I have no formal, or even informal, coding background), finally attained the Eureka Moment!!!
The resultant outcome works flawlessly on my rig. Am uploading the past 15 days of one minute IEOD for spot Nifty & BankNifty for people who may have need of it, and also as proof of concept
For sharing the utility, will now have to look at and learn how make a distributable with user input & error handling built in, as it'd be unreasonable to expect all users to have python installed on their rigs. Plus I'm seriously not upto 'hand holding' due constraints of time for people who will decidedly need guidance for installing python/ modules/ Packages/ Libraries and environment to use this 'as it is' in its present form!
Let me know if any of you guys (n gals?) here would be interested in using/ testing this out, so that I know that the effort to make a distro is useful, at least!
Will likely work on and share this by tomorrow. Presently too spaced out to tackle the job
Cheerio!!!
P.S - BTW, did I happen to mention it earlier? The distro's free!!!
Edit - Forgot to attach the format file for import of uploaded spot indices IEOD to amibroker! Done now!!
Last edited:
