How to clean data for Amibroker

#1
Hello experts and forum users....

As of current i used manual method to correct the database i use for amibroker. the data that i get in NEST saved as csv file is purposely hammered like


Does any one knows how to automatically remove all the lines that has
"0.0000,0.0000,0.0000, " data ..OR
i want some code/software that can remove entire line if data is 0.0000,0.0000,0.0000, ......

thx
Trading Symbol,Time,Open,High,Low,Close/Price,Volume
GOLDM15MAYFUT,23-04-2015 22:35:00,26693.0000,26699.0000,26690.0000,26699.0000,1700
GOLDM15MAYFUT,23-04-2015 22:34:00,26682.0000,26688.0000,26682.0000,26688.0000,2200
GOLDM15MAYFUT,23-04-2015 22:33:00,26686.0000,26687.0000,26677.0000,26677.0000,400
GOLDM15MAYFUT,23-04-2015 22:32:00,26679.0000,26680.0000,26679.0000,26680.0000,500
GOLDM15MAYFUT,23-04-2015 22:31:00,0.0000,0.0000,0.0000,26677.0000,100 ---=========================> error lines and i want to get it deleted
GOLDM15MAYFUT,23-04-2015 22:30:00,26679.0000,26679.0000,26677.0000,26677.0000,400
GOLDM15MAYFUT,23-04-2015 22:29:00,26675.0000,26677.0000,26675.0000,26677.0000,200
GOLDM15MAYFUT,23-04-2015 22:28:00,26682.0000,26685.0000,26676.0000,26676.0000,1300
GOLDM15MAYFUT,23-04-2015 22:27:00,26665.0000,26676.0000,26665.0000,26676.0000,800
GOLDM15MAYFUT,23-04-2015 22:26:00,26670.0000,26670.0000,26666.0000,26666.0000,700
GOLDM15MAYFUT,23-04-2015 22:25:00,26666.0000,26669.0000,26666.0000,26669.0000,400
GOLDM15MAYFUT,23-04-2015 22:24:00,26668.0000,26669.0000,26665.0000,26667.0000,500
GOLDM15MAYFUT,23-04-2015 22:23:00,0.0000,0.0000,0.0000,26669.0000,600
GOLDM15MAYFUT,23-04-2015 22:22:00,26669.0000,26669.0000,26669.0000,26669.0000,300
GOLDM15MAYFUT,23-04-2015 22:21:00,26672.0000,26672.0000,26670.0000,26670.0000,1600
GOLDM15MAYFUT,23-04-2015 22:20:00,26675.0000,26675.0000,26674.0000,26674.0000,200
GOLDM15MAYFUT,23-04-2015 22:19:00,26680.0000,26680.0000,26677.0000,26677.0000,800
GOLDM15MAYFUT,23-04-2015 22:18:00,26685.0000,26685.0000,26685.0000,26685.0000,100
GOLDM15MAYFUT,23-04-2015 22:17:00,26688.0000,26688.0000,26684.0000,26688.0000,1500
GOLDM15MAYFUT,23-04-2015 22:16:00,26689.0000,26689.0000,26688.0000,26689.0000,500
GOLDM15MAYFUT,23-04-2015 22:15:00,26694.0000,26695.0000,26690.0000,26695.0000,400
GOLDM15MAYFUT,23-04-2015 22:13:00,0.0000,0.0000,0.0000,26690.0000,300
GOLDM15MAYFUT,23-04-2015 22:12:00,26688.0000,26695.0000,26688.0000,26690.0000,500
i use notepad and use find function to manually delete 0.0000,0.0000,0.0000 but it;s tiring work if u miss a day then everyday same process..
I wish there was some software or code that could just filter output the errortic data in red .. i am sure c program can be made or VB code idk

Thanks for read and replies :)





Solution given by Master KelvinHand and it works
I made a little change in extension it works Thanks again mate ^^^

1. Save the following code into "test123.js" in a folder
2. Save your datafile in post #1 as mydata.csv the same folder. (Try out first)
3. run the jscript in that folder
4. The code will produce mydataready.csv where all those errors should be filter.

5. mydataready.csv is the filtered file without lag data

Note: do not create in root dir eg. c:\
Code:
filename ="mydata";


           var iFile = filename+".csv";
           var oFile = filename+"ready.csv";

           var fso, fi, fo, r;
           var ForReading = 1;

           var xO, xH, xL;


           /* ... and file system object */
           fso = new ActiveXObject( "Scripting.FileSystemObject" );


           if (fso.FileExists(oFile)) fso.DeleteFile(oFile);
 

           /* open ASCII file */
           fi = fso.OpenTextFile(iFile, ForReading);

       fo= fso.CreateTextFile(oFile, false);


           /* skip first line which contains format definition */
           r =  fi.ReadLine(); 
           fo.WriteLine(r);


           /* read the file line by line */
           var cnt =0;

           while ( !fi.AtEndOfStream )
           {  
              r =  fi.ReadLine();
             
              
              /* split the lines using coma as a separator */
              fields = r.split(","); 
          xO = parseFloat( fields[2]);
          xH = parseFloat( fields[3]);
          xL = parseFloat( fields[4]);


           if (xO>0.0 || xH>0 || xL>0)
          {
         fo.WriteLine(r);
              }
          else
              {
          cnt++;
              }
     
              

             
           }
           /* refresh ticker list and windows */
           /* notify the user */
           WScript.Echo( "Finished! Error Count = " + cnt );
 
Last edited:

manishchan

Well-Known Member
#2
Hello experts and forum users....

As of current i used manual method to correct the database i use for amibroker. the data that i get in NEST saved as csv file is purposely hammered like


Does any one knows how to automatically remove all the lines that has
"0.0000,0.0000,0.0000, " data ..OR
i want some code/software that can remove entire line if data is 0.0000,0.0000,0.0000, ......

thx


i use notepad and use find function to manually delete 0.0000,0.0000,0.0000 but it;s tiring work if u miss a day then everyday same process..
I wish there was some software or code that could just filter output the errortic data in red .. i am sure c program can be made or VB code idk

Thanks for read and replies :)

In the csv file itself, use filter. Sort it by opening price (which seems to be 0.000000.. select all the row and shift+delete
 
#3
ty for reply problem is 0.0000,0.0000,0.0000 are not one after another they are random ..... and there is no filter to select all those at once / idk how to select all lines
hence i am searching for code..../ software or any other known method .
I am sure i am not alone that is facing this problem and this problem occurs in mcx data table only
 

manishchan

Well-Known Member
#4
In the csv file itself, use filter. Sort it by opening price (which seems to be 0.000000.. select all the row and shift+delete
ty for reply problem is 0.0000,0.0000,0.0000 are not one after another they are random ..... and there is no filter to select all those at once / idk how to select all lines
hence i am searching for code..../ software or any other known method .
I am sure i am not alone that is facing this problem and this problem occurs in mcx data table only

Not sure what you are refering to .... do you know what the sort function does in CSV/excel ??? It will put all the 0.0000 to either top or bottom of the sheet. Then just select all of those row and delete.
 

amitrandive

Well-Known Member
#5
ty for reply problem is 0.0000,0.0000,0.0000 are not one after another they are random ..... and there is no filter to select all those at once / idk how to select all lines
hence i am searching for code..../ software or any other known method .
I am sure i am not alone that is facing this problem and this problem occurs in mcx data table only
Can try this

http://www.amibroker.com/kb/category/data/
http://www.wises tocktrade r.com/indicators/4222-remove-data-between-the-dates(remove spaces)
 

KelvinHand

Well-Known Member
#6
ty for reply problem is 0.0000,0.0000,0.0000 are not one after another they are random ..... and there is no filter to select all those at once / idk how to select all lines
hence i am searching for code..../ software or any other known method .
I am sure i am not alone that is facing this problem and this problem occurs in mcx data table only

1. Save the following code into "test123.js" in a folder
2. Save your datafile in post #1 as mydata.dat the same folder. (Try out first)
3. run the jscript in that folder
4. The code will produce mydata.out where all those errors should be filter.

5. if ok copy mydata.out to mydata.dat or whatever you wish

Note: do not create in root dir eg. c:\


PHP:
filename ="mydata";


           var iFile = filename+".dat";
           var oFile = filename+".out";

           var fso, fi, fo, r;
           var ForReading = 1;

           var xO, xH, xL;


           /* ... and file system object */
           fso = new ActiveXObject( "Scripting.FileSystemObject" );


           if (fso.FileExists(oFile)) fso.DeleteFile(oFile);
 

           /* open ASCII file */
           fi = fso.OpenTextFile(iFile, ForReading);

	   fo= fso.CreateTextFile(oFile, false);


           /* skip first line which contains format definition */
           r =  fi.ReadLine(); 
           fo.WriteLine(r);


           /* read the file line by line */
           var cnt =0;

           while ( !fi.AtEndOfStream )
           {  
              r =  fi.ReadLine();
	     	
              
              /* split the lines using coma as a separator */
              fields = r.split(","); 
	      xO = parseFloat( fields[2]);
	      xH = parseFloat( fields[3]);
	      xL = parseFloat( fields[4]);


 	      if (xO>0.0 || xH>0 || xL>0)
	      {
 		fo.WriteLine(r);
              }
	      else
              {
  		cnt++;
              }
 	
              

             
           }
           /* refresh ticker list and windows */
           /* notify the user */
           WScript.Echo( "Finished! Error Count = " + cnt );
 
Last edited:
#7
KelvinHand
Guruji it worked like charm woooooooooooo!!!!!

dmmmm man you must be a skilled programmer ^^^^

Many thanks. you have saved me a daily hazzel ....One click and job done now

KelvinHand man if ever visit mumbai food and drinks on me :) man u made my day


@sumosanammain
Trust me that code will help a lot of ppls that use nest data table option for Amibroker graphs.
lool Good places to eat ?
Code:
http://www.tripadvisor.in/Restaurants-g304554-Mumbai_Bombay_Maharashtra.html
personally i prefer ...
Status Veg Restaurant, Nariman Point
Ramanayak Udipi, Matunga
shri krishna nana chowk
Aditi parel
and at odd day even stall at road side food ^^
 
Last edited:

sumosanammain

Well-Known Member
#8
Guruji it worked like charm woooooooooooo!!!!!

dmmmm man you must be a skilled programmer ^^^^

Many thanks. you have saved me a daily hazzel ....One click and job done now

KelvinHand man if ever visit mumbai food and drinks on me :) man u made my day
At least one happy trader on the forum today :)
BTW which are in mumbai are you located in? I am there frequently... know a lot of good places to eat... could point out a few to you :)
 

Similar threads