How to extract some lines from a large EDA report?

[jacobus]

hello. everyone.

I have a large EDA report about several giga bytes,
and I need to extract some lines from the report, for example, the last 100 lines and then print these lines in an reverse order in another file.

Since for some line, it may contain about 1M byte data, it seems very inefficient to use fseek to locate the file, read the line and then write the file word by word.

So I want to know if there is any more efficient way to extract the last 100 lines from the EDA file?
And due to my platform and knowledge limitation, I only can use C or Perl to realize such a script.

Thanks a lot!

 

[echo47]

I'm not sure what you mean by "EDA report" or "lines". Is it a text file?

If it's a text file, try finding an efficient version of the UNIX/Linux/GNU "tail" utility. It outputs the last few lines of a file. Then pipe it through something like "tac" to reverse the lines. Some versions of "tail" include a line reverse option.

You could open the file in binary mode (maybe faster than text mode), then fseek(fp, 0, SEEK_END) to the end of the file, and then use fseek() and fread() to step backwards and read a few kilobytes at a time until you find the 100 lines you want. I'm guessing this is how "tail" works.

If you know approximately how many bytes you need to read from the end of the file, then you could open it in binary mode, fseek(fp, offset, SEEK_END) to that position, and then fread() the whole tail end into memory. That should go really fast.

 

[jacobus]

I'm not sure what you mean by "EDA report" or "lines". Is it a text file?

If it's a text file, try finding an efficient version of the UNIX/Linux/GNU "tail" utility. It outputs the last few lines of a file. Then pipe it through something like "tac" to reverse the lines. Some versions of "tail" include a line reverse option.

You could open the file in binary mode (maybe faster than text mode), then fseek(fp, 0, SEEK_END) to the end of the file, and then use fseek() and fread() to step backwards and read a few kilobytes at a time until you find the 100 lines you want. I'm guessing this is how "tail" works.

If you know approximately how many bytes you need to read from the end of the file, then you could open it in binary mode, fseek(fp, offset, SEEK_END) to that position, and then fread() the whole tail end into memory. That should go really fast.

thanks very much for your reply.
It is a text file.

If I write the script myself, should the algorithm be like this?

fseek(fp,-1*buffersize,SEEK_END);
fgetpos(fp, &filepos);
while( n< number of lines){
fread(buffer, buffersize,fp);
reverse the array and calculate the existence of "\n"
fwrite(buffer,buffersize,fp);
fsetpos(fp, &filepos)
fseek(fp,-1*buffersize,SEEK_CUR);
fgetpos(fp, &filepos);
}


Does that mean I should set the buffersize to some large amount, otherwise it will need to read file many times if the line size is very much?

thanks again

 

[banh]

hello. everyone.

Since for some line, it may contain about 1M byte data, it seems very inefficient to use fseek to locate the file, read the line and then write the file word by word.

So I want to know if there is any more efficient way to extract the last 100 lines from the EDA file?
And due to my platform and knowledge limitation, I only can use C or Perl to realize such a script.

Thanks a lot!

how about using some splitter utilities to cut your file into chunks first.
you only need to handle the last chunks which are of smaller sizes.

 

[artem]

i never used this - you can probably get the file size directly via fstat() if not wrong or other which gives you stat structure holding the file size and go directly to position you want by fseek() .
In windows here is GetFileSize() function and others, but i am not sure in applicability to this case - must be checked.

btw , when fseek is executed , OS does not go through entire file content but ionodes - file headers till last where position does point. So whole function should not take long time to complete .


I got tha wrong - you need lines not position please discard this message .

 

[jacobus]

hello. everyone.

Since for some line, it may contain about 1M byte data, it seems very inefficient to use fseek to locate the file, read the line and then write the file word by word.

So I want to know if there is any more efficient way to extract the last 100 lines from the EDA file?
And due to my platform and knowledge limitation, I only can use C or Perl to realize such a script.

Thanks a lot!

how about using some splitter utilities to cut your file into chunks first.
you only need to handle the last chunks which are of smaller sizes.

thanks first
I did not quite catch 'chunk' here.
do you mean I first add some split in the file or read the whole and then split?

 

[banh]

what i mean is you can find a file splitter program to cut your file first.

google "file splitter" :
https://www.google.com.sg/search?so...rls=GGGL,GGGL:2005-09,GGGL:en&q=file+splitter

 

[echo47]

Aarrgh! I just learned that fseek(... SEEK_END) is not guaranteed to work. It says so in the ANSI C standard. It may work in some systems, but apparently there is no portable method in C to determine the size of a file (or seek to its end) without reading the whole thing. It looks like you need to try it and see, or make some operating system specific file access calls to help you.

Or go find "tail".

 

[artem]

that case my suggestion would help - find file end by file size then seek to that positioin .

 

[jacobus]

thanks for all your suggestions!
I would have some try.