Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Categories

Welcome to the new platform of Programmer's Heaven! We apologize for the inconvenience caused, if you visited us from a broken link of the previous version. The main reason to move to a new platform is to provide more effective and collaborative experience to you all. Please feel free to experience the new platform and use its exciting features. Contact us for any issue that you need to get clarified. We are more than happy to help you.

Reading text files

tj29tj29 Posts: 45Member
I want to know what is the best way to read text files that are 500 megs in size. each file contains more or less 500,000 rows. I need to read this file line by line to extract data from it.

do I load the file to a string list?

do i read the file using end of file and end of line while loops?

or

is there a better way to ready quickly and efficiently with out using so much memroy?

thank you very much.

Comments

  • zibadianzibadian Posts: 6,349Member
    : I want to know what is the best way to read text files that are 500 megs in size. each file contains more or less 500,000 rows. I need to read this file line by line to extract data from it.
    :
    : do I load the file to a string list?
    :
    : do i read the file using end of file and end of line while loops?
    :
    : or
    :
    : is there a better way to ready quickly and efficiently with out using so much memroy?
    :
    : thank you very much.
    :
    I would use a while-not-eof() loop with a readln() for this. This way you have a memory efficient program, but there is a downside: this only works if you want to read the lines sequentially. If you want to "jump around" then a TStrings object is better. The eoln() function is not necessary, since it is better to read each line individually instead of each character individually. If you want to access the file on a per character basis, then a TFileStream is better.
  • dolev9dolev9 Posts: 147Member
    reading line by line would take hours with 500 mb file.

    as larger the blocks you read faster it will be.
    read it as larger blocks as you can - try 5 mb block.

    check the filesize and then decrement the filesize by 50 mb each time.

    after that you will need to seperate the 50 mb data to lines by your self - more work but much faster.

    dolev

  • tj29tj29 Posts: 45Member
    : : I want to know what is the best way to read text files that are 500 megs in size. each file contains more or less 500,000 rows. I need to read this file line by line to extract data from it.
    : :
    : : do I load the file to a string list?
    : :
    : : do i read the file using end of file and end of line while loops?
    : :
    : : or
    : :
    : : is there a better way to ready quickly and efficiently with out using so much memroy?
    : :
    : : thank you very much.
    : :
    : I would use a while-not-eof() loop with a readln() for this. This way you have a memory efficient program, but there is a downside: this only works if you want to read the lines sequentially. If you want to "jump around" then a TStrings object is better. The eoln() function is not necessary, since it is better to read each line individually instead of each character individually. If you want to access the file on a per character basis, then a TFileStream is better.
    :
    :::
    Thank you very much i really appreciate you help thanks again
Sign In or Register to comment.