PyDownloader Reloaded

[tradermodified]

Mike bro
I have some suggestions for the new version, if easy to implement, then please include these.

- include other time frames, like 5 min, 15 mins, 30/60 mins (as these TF's provide data for more than 7 days at a time) and daily, with an option to choose
- increase numbers of scrips to 200 or 300 in a go, instead of 100.

Thanks

 

[biggles]

"This would be compiled on a Win 10 rig as I have donated all my older rigs with previous versions of windows to underprivileged kids to facilitate online education! "

Bravo Mike. You rise stratospherically in our esteem. More power to you

 

[rmike]

yfinance is nice library that you may want to look at. Works well and seems to be actively maintained. All i had to do was update package when previous version broke

Am aware of the quoted library

It's OK to use that for personal purposes if one has python installed on the rig and knows what to do with that. However, when it comes to using that library for building an app for generic distribution (to people who don't have python installed and don't wish to either), there are a few very 'real considerations', which engender food for thought:-

> The lib is based on 3.6 environment and has a few packages which are not current and may soon be legacy in near future

> My installation is 3.8 and fully updated

> The lib has many other attendancies which cater to building a database which is not specifically (out of the box) compatible to any trading platform

> Converting python code to app produces a bulky executable because it has to 'bundle in' the python environment

> Hence, any such app will typically take time to be 'ready for action' on the first run because the temp environment has to be loaded first for it perform. By using the lib 'as it is' will not only increase the bulk of the executable, but will also increase the 'ready for action' timelag manifold!!!

> It takes time and effort to examine & experiment with all features in someone elses' code to filter out the superfluous. Further, the modification & distribution of someone elses' code as an app not only requires according 'due credit' (which can be easily done, with no issues) but also, does require taking 'due permission' from the owner, if not as a point of strict legality but certainly as a point of ethic!!! That process can take days, if not weeks!!! Going by past personal experience (the very first ver of PyDownloader), at times one receives no answer at all!!! Most people will probably not understand what I am talking about here because most are under the impression that everything posted on Github is 'fair game' - It certainly may be public, but it still has a licence attached to it!!! You being a developer yourself will understand where 'I am coming from' here

> Lastly, since I am a 'self learnt' coder, I started out as being unsure of everything. Hence tested out every line of code for not only correctness but also for elegance, performance and efficiency. This 'repetitive doing' has transmutated into an ingrained habit and process flow. This way not only do I know what the code is doing, but also what is exactly going on under the hood . Most of the free code available on the net is not very elegant/ efficient and hence may not be exactly optimal for a trader in a live fast moving intraday environment. For that kind of environment, my endeavour would be to write the leanest, meanest possible kind of code which can best assist even a scalper to extract the max possible advantage!!! Being a trader myself and having studied data nuances 'in-depth' does help in better understanding of traders' requirements!!!

Regards,

 

[TracerBullet]

Am aware of the quoted library

It's OK to use that for personal purposes if one has python installed on the rig and knows what to do with that. However, when it comes to using that library for building an app for generic distribution (to people who don't have python installed and don't wish to either), there are a few very 'real considerations', which engender food for thought:-

> The lib is based on 3.6 environment and has a few packages which are not current and may soon be legacy in near future

> My installation is 3.8 and fully updated

> The lib has many other attendancies which cater to building a database which is not specifically (out of the box) compatible to any trading platform

> Converting python code to app produces a bulky executable because it has to 'bundle in' the python environment

> Hence, any such app will typically take time to be 'ready for action' on the first run because the temp environment has to be loaded first for it perform. By using the lib 'as it is' will not only increase the bulk of the executable, but will also increase the 'ready for action' timelag manifold!!!

> It takes time and effort to examine & experiment with all features in someone elses' code to filter out the superfluous. Further, the modification & distribution of someone elses' code as an app not only requires according 'due credit' (which can be easily done, with no issues) but also, does require taking 'due permission' from the owner, if not as a point of strict legality but certainly as a point of ethic!!! That process can take days, if not weeks!!! Going by past personal experience (the very first ver of PyDownloader), at times one receives no answer at all!!! Most people will probably not understand what I am talking about here because most are under the impression that everything posted on Github is 'fair game' - It certainly may be public, but it still has a licence attached to it!!! You being a developer yourself will understand where 'I am coming from' here

> Lastly, since I am a 'self learnt' coder, I started out as being unsure of everything. Hence tested out every line of code for not only correctness but also for elegance, performance and efficiency. This 'repetitive doing' has transmutated into an ingrained habit and process flow. This way not only do I know what the code is doing, but also what is exactly going on under the hood . Most of the free code available on the net is not very elegant/ efficient and hence may not be exactly optimal for a trader in a live fast moving intraday environment. For that kind of environment, my endeavour would be to write the leanest, meanest possible kind of code which can best assist even a scalper to extract the max possible advantage!!! Being a trader myself and having studied data nuances 'in-depth' does help in better understanding of traders' requirements!!!

Regards,

yeah, you are releasing binary. Did not consider that. Anyway, just wrote it as fyi.

1) It works without issue on 3.8. But i don't really use it, so testing is limited. More of a backup if everything else goes to hell.
2) License is very permissive, it usually is for open source projects. This one is even more open than gpl. https://github.com/ranaroussi/yfinance/blob/main/LICENSE.txt.
But yahoo finance license will probably be much more restrictive.
3) You have to take a call. Popular libraries are usually well maintained and will have more reliable code than i could write in limited time, if that. For ex, there is no real point in trying to write requests/pandas/etc on your own. Most of the power of python for me comes from fast well designed libraries, both internal and 3rd party.
For me, most important is for code to be readable and reasonably logically organized. Performance is best managed through removing bottlenecks rather than line by line checks which does not really work beyond a point( miss the forest for the trees kind of thing). But yeah i get the need of having control over different parts. I use good popular libraries as building blocks but generally do not use other people's tools and code as-it-is for day to day work. Having your own gives freedom to change it as exactly as needed. Good to hear from you ..

 

[rmike]

Also fyi, Fyers are going to provide free realtime+historical data via api for their users. Would likely be better than yahoo since yahoo does put some garbage minutes in data. Still yahoo is decent as backup.

Actually, this is what I have come to now 'euphemistically categorize' as an urban myth amongst most (if not all ) traders. Bear with me for the moment when I state that there's NO such entity called as 'Perfect Data'!!!

Let alone unauthorized data vendors/ sources of data, even authorized data vendors will have relational variances in their data in comparison! This is not only due to the sampling/ filtering method employed 'in-house' and at NSE's end but also due to the fact of which vendor/ institution got attached to what port in NSE server's 'plumbing'!!! [if you'd recollect, there was an exposé of a major scam related to this issue not very long ago which culminated in the resignation (read - firing!) of the previous NSE Head]

Hence all data vendors will have minute (sometimes even major) variances in terms of price, vol etc attributes depending upon specific timestamp(s)! Here, we have vendors who subscribe to and provide 'one sec snapshot data'!!! Millisecs of difference would result in some trades shifting to a prior or following data point amongst vendors!!! Brokers are NOT authorized data vendors and the data supplied to brokers is subjected to far restrictive sampling protocol at NSE's end!!!

At the end of the day, the only Benchmark of what may be considered as 'Perfect Data' is what is finally published by the exchange after verification of all trades via 'clearing'!!! For intraday purposes, the only available benchmark is the one minute data published in TAME charts (which cannot be accessed now via browsers post phasing out of Adobe Flash. However, I had developed a custom utility for intraday backfill from that source for personal purposes and hence am in a position to regularly compare. AND no PM's or requests for that from anyone please - as with NSE there are too many legalities involved!!! A related case in point is the Manshirt RT debacle!!!)

There have been times in extremely fast moving markets/ flash crashes that even exchanges, at times, fail to record rogue ticks which are discovered only much later after the fact!!! If one is in the habit of employing a keen eye, one would notice that NSE keeps updating the Bhavcopy long after market close - Typically, the last update is near about 8 PM!!! And that may be considered as the final Benchmark!!!

Yahoo too pays for data from NSE for display on their website and the data is OK catering for the fact of relational comparison differences (like I mentioned, I am in a position to compare) for backfill. The 'garbage' which you mentioned is merely Yahoo's way of warehousing data in a common format catering for the requirement of four digits after the decimal in respect of currencies. But one can accordingly design a fn to tackle that, post studying of the pattern employed!!!

Regards,

 

[rmike]

Did not consider that. Anyway, just wrote it as fyi

Much appreciated!!!

there is no real point in trying to write requests/pandas/etc on your own

True that!!!

Performance is best managed through removing bottlenecks rather than line by line checks

The 'line by line' phrase was merely an expression (though, to be honest, I did start out like that )

Good to hear from you

Likewise!!!

Regards,

 

[rmike]

Mike bro
I have some suggestions for the new version, if easy to implement, then please include these.

- include other time frames, like 5 min, 15 mins, 30/60 mins (as these TF's provide data for more than 7 days at a time) and daily, with an option to choose
- increase numbers of scrips to 200 or 300 in a go, instead of 100.

Thanks

Actually, the current 'logical' intraday intervals are 1 min, 2min and 60min

1 min data is for max 7 days. For 2 min and above till (including 5 min, 15 min) 30 min, the data is for max 60 days

For 60 min (or 1 hr) data is max for 2 years

Furthermore, since Yahoo provides only equity data and considering NSE equity market timings, the final logical intervals would be 1m, 2m, 5m, 15m because the 30min and 60min interval would not generate equitime bars due to the awkward market timing

Including 'daily' has no real value addition. The same can be downloaded from NSE or via many other popular and available utilities. Also, for daily there's a more efficient, different query than the one for intraday!

With regard to increasing the number of scrips I, personally speaking, would have been far more inclined towards reducing it to 50. This is because, not only has Yahoo incorporated roadblocks (which, though, can be negotiated , by redesigning the app) but also the feature for reporting an IP employing multiple ticker bulk download (this cannot be negotiated). The 'reporting IP' feature is disparate from extant ratelimiting and immediately logs/ reports such an IP, at times after the 10th or 20th ticker itself!!! Tested this with a barebones 'query only' prototype and had it freezing midway!!! The only option is to then restart with a separate tickerlist of the remaining tickers!!!

The thing is, once the utility is released, the manner of it's employment rests in the hands of people using it. Some are wise and some are otherwise!!! The wise take only what they need. The otherwise load up to the max on whatever they can get!!! Little realizing that the needless extra is just hubris (I've seen people going to the extent of attempting to build a 1 min database for all the tickers in NSE)!!! In that process they unduly and needlessly abuse a good viable resource of free data!!! This results in drying up of that resource over a period of prolonged abuse!!!

Intraday trading does not require an extensive dataset of tickers. I trade everyday and the parameters which I employ for vehicle selection typically encompass a dataset of about 100, out of which the best three candidates are earmarked as per trading system criteria. Even on a good day (i.e clear one sided day) I'd be extremely wary of putting on more than three concurrent trades unless there was a way to ensure double fail-safe end-to-end complete automation which catered for all possible contingencies - market as well technology related!!! Which, of course, is not the case . This would also not be the case for most (if not all) people here, because traders with that kind of firepower, know-how and wherewithal at their disposal generally do not hang around forums and neither do they use such humble utilities!!!

Anyway, since you've requested - will increase the number to 200. I sincerely hope that people do pay heed to what I've said above - for their own good as well as for the good of others!!!

Regards,

 

[biggles]

I feel you should restrict the securities in the ticker list to a reasonable number. Even 100 is way too much for an individual's trading.
Collective good vs avarice.

 

[rmike]

I feel you should restrict the securities in the ticker list to a reasonable number. Even 100 is way too much for an individual's trading.
Collective good vs avarice.

Since the request has been tendered by a previous 'enthusiastic user', hence it's been considered!

Although, it's been pointed out that the issue of limit of number of scrips in no way hinders a genuine retail trader

https://www.traderji.com/community/threads/pydownloader.105364/post-1253752

This limit is an inbuilt sensible safeguard for the user's own good. The time taken to manually edit/ substitute TickerLists enforces a much needed hiatus to keep under the ratelimiting radar!!!

Hence the caution to keep it sane with TickerLists comprising of higher number of scrips, with the facility incorporated!!!

 

[rmike]

Python Yahoo Downloader - Version Upgrade V 1.2

With Yahoo Finance Fixes!

* Incorporated Multi Interval, Multi Period Data download from Multiple Exchanges as per user selection and TickerList config (Exchange suffixes as per Yahoo Finance).

* Incorporated TickerList scrips increase to 200 scrips.

https://www.traderji.com/community/threads/pydownloader-reloaded.107309/post-1447709

* Incorporated Auto Conversion of ^NSEI and ^NSEBANK to NIFTY 50 and NIFTY BANK for NSE index data for user import convenience.

https://www.traderji.com/community/threads/pydownloader-reloaded.107309/post-1350431

* Incorporated Metastock compatibility and addition of Ticker Column.

https://www.traderji.com/community/threads/pydownloader.105364/post-1441281

1. Didn't have time to write detailed 'help' but previous and even new users ought not to face any issues as the utility is fairly intuitive to use.
2. Download from attached link. Start 'Setup', accept default prompts and the ultility will get installed in C:\PyDownloader. Don't be tempted to change the installation path during Setup because the utlity is configured to work only from that location!!!
3. * If you still have the previous installation in your C:\ drive, then copy/ backup/ preseve your TickerList.txt because it may get overwriitten!!!
4. Place the 'PyD_v1.2.format' file in your Amibroker installation's 'Formats' folder and you are Good To Go!
5. A sample TickerList is included for the guidance of new users for scrip and exchange suffix config. Sample data files are included in the Data folder as output format examples.

Will probably look in from time to time to see how users are getting on. If you like using the utility and are a previous experienced user, then you may like to contribute by guiding newbies who may need hand holding

Note:- New users will benefit by going through this entire thread and the one indicated below, for troubleshooting or for answers to general queries!

https://www.traderji.com/community/threads/pydownloader.105364/

Regards,