Welcome to EDAboard.com

Welcome to our site! EDAboard.com is an international Electronic Discussion Forum focused on EDA software, circuits, schematics, books, theory, papers, asic, pld, 8051, DSP, Network, RF, Analog Design, PCB, Service Manuals... and a whole lot more! To participate you need to register. Registration is free. Click here to register now.

Register Log in

How to extract some lines from a large EDA report?

Status
Not open for further replies.

jacobus

Newbie level 5
Joined
Jan 11, 2006
Messages
9
Helped
1
Reputation
2
Reaction score
0
Trophy points
1,281
Activity points
1,376
hello. everyone.

I have a large EDA report about several giga bytes,
and I need to extract some lines from the report, for example, the last 100 lines and then print these lines in an reverse order in another file.

Since for some line, it may contain about 1M byte data, it seems very inefficient to use fseek to locate the file, read the line and then write the file word by word.

So I want to know if there is any more efficient way to extract the last 100 lines from the EDA file?
And due to my platform and knowledge limitation, I only can use C or Perl to realize such a script.

Thanks a lot!
 

echo47

Advanced Member level 5
Joined
Apr 7, 2002
Messages
3,942
Helped
636
Reputation
1,270
Reaction score
88
Trophy points
1,328
Location
USA
Activity points
33,183
I'm not sure what you mean by "EDA report" or "lines". Is it a text file?

If it's a text file, try finding an efficient version of the UNIX/Linux/GNU "tail" utility. It outputs the last few lines of a file. Then pipe it through something like "tac" to reverse the lines. Some versions of "tail" include a line reverse option.

You could open the file in binary mode (maybe faster than text mode), then fseek(fp, 0, SEEK_END) to the end of the file, and then use fseek() and fread() to step backwards and read a few kilobytes at a time until you find the 100 lines you want. I'm guessing this is how "tail" works.

If you know approximately how many bytes you need to read from the end of the file, then you could open it in binary mode, fseek(fp, offset, SEEK_END) to that position, and then fread() the whole tail end into memory. That should go really fast.
 

    jacobus

    points: 2
    Helpful Answer Positive Rating

jacobus

Newbie level 5
Joined
Jan 11, 2006
Messages
9
Helped
1
Reputation
2
Reaction score
0
Trophy points
1,281
Activity points
1,376
echo47 said:
I'm not sure what you mean by "EDA report" or "lines". Is it a text file?

If it's a text file, try finding an efficient version of the UNIX/Linux/GNU "tail" utility. It outputs the last few lines of a file. Then pipe it through something like "tac" to reverse the lines. Some versions of "tail" include a line reverse option.

You could open the file in binary mode (maybe faster than text mode), then fseek(fp, 0, SEEK_END) to the end of the file, and then use fseek() and fread() to step backwards and read a few kilobytes at a time until you find the 100 lines you want. I'm guessing this is how "tail" works.

If you know approximately how many bytes you need to read from the end of the file, then you could open it in binary mode, fseek(fp, offset, SEEK_END) to that position, and then fread() the whole tail end into memory. That should go really fast.
thanks very much for your reply.
It is a text file.

If I write the script myself, should the algorithm be like this?

fseek(fp,-1*buffersize,SEEK_END);
fgetpos(fp, &filepos);
while( n< number of lines){
fread(buffer, buffersize,fp);
reverse the array and calculate the existence of "\n"
fwrite(buffer,buffersize,fp);
fsetpos(fp, &filepos)
fseek(fp,-1*buffersize,SEEK_CUR);
fgetpos(fp, &filepos);
}


Does that mean I should set the buffersize to some large amount, otherwise it will need to read file many times if the line size is very much?

thanks again
 

banh

Advanced Member level 1
Joined
Dec 16, 2004
Messages
458
Helped
17
Reputation
34
Reaction score
5
Trophy points
1,298
Activity points
3,939
jacobus said:
hello. everyone.

Since for some line, it may contain about 1M byte data, it seems very inefficient to use fseek to locate the file, read the line and then write the file word by word.

So I want to know if there is any more efficient way to extract the last 100 lines from the EDA file?
And due to my platform and knowledge limitation, I only can use C or Perl to realize such a script.

Thanks a lot!

how about using some splitter utilities to cut your file into chunks first.
you only need to handle the last chunks which are of smaller sizes.
 

artem

Advanced Member level 4
Joined
May 22, 2003
Messages
1,350
Helped
126
Reputation
252
Reaction score
32
Trophy points
1,328
Location
Turkey
Activity points
13,461
i never used this - you can probably get the file size directly via fstat() if not wrong or other which gives you stat structure holding the file size and go directly to position you want by fseek() .
In windows here is GetFileSize() function and others, but i am not sure in applicability to this case - must be checked.

btw , when fseek is executed , OS does not go through entire file content but ionodes - file headers till last where position does point. So whole function should not take long time to complete .


I got tha wrong - you need lines not position please discard this message .
 

jacobus

Newbie level 5
Joined
Jan 11, 2006
Messages
9
Helped
1
Reputation
2
Reaction score
0
Trophy points
1,281
Activity points
1,376
banh said:
jacobus said:
hello. everyone.

Since for some line, it may contain about 1M byte data, it seems very inefficient to use fseek to locate the file, read the line and then write the file word by word.

So I want to know if there is any more efficient way to extract the last 100 lines from the EDA file?
And due to my platform and knowledge limitation, I only can use C or Perl to realize such a script.

Thanks a lot!

how about using some splitter utilities to cut your file into chunks first.
you only need to handle the last chunks which are of smaller sizes.
thanks first
I did not quite catch 'chunk' here.
do you mean I first add some split in the file or read the whole and then split?
 

echo47

Advanced Member level 5
Joined
Apr 7, 2002
Messages
3,942
Helped
636
Reputation
1,270
Reaction score
88
Trophy points
1,328
Location
USA
Activity points
33,183
Aarrgh! I just learned that fseek(... SEEK_END) is not guaranteed to work. It says so in the ANSI C standard. It may work in some systems, but apparently there is no portable method in C to determine the size of a file (or seek to its end) without reading the whole thing. It looks like you need to try it and see, or make some operating system specific file access calls to help you.

Or go find "tail".
 

artem

Advanced Member level 4
Joined
May 22, 2003
Messages
1,350
Helped
126
Reputation
252
Reaction score
32
Trophy points
1,328
Location
Turkey
Activity points
13,461
that case my suggestion would help - find file end by file size then seek to that positioin .
 

jacobus

Newbie level 5
Joined
Jan 11, 2006
Messages
9
Helped
1
Reputation
2
Reaction score
0
Trophy points
1,281
Activity points
1,376
thanks for all your suggestions!
I would have some try.
 

Status
Not open for further replies.
Toggle Sidebar

Part and Inventory Search

Welcome to EDABoard.com

Sponsor

Top