Partial File Loading

bluntelk · Post by **bluntelk** » Fri Nov 21, 2003 5:00 am

if this has been suggested, sorry for not seeing the existing thread.

The Problem:
Opening large files makes textpad go non responsive as it tries to load the file into memory.

Possible sollution:
Only load the first x megabytes of the file (when opening) the file

eg - 100 meg log file
user opens it
textpad load the first 2 meg.
as the user scrolls down (or across) the log textpad keeps a track of where they are in the file - loading the relavant portions of the files as they go

so in effect - if you loaded up a 100 meg file, it would load 2 meg at a time as you scrolled down.

as for updated files, the trick will still work.

as for "the size" of the files to do this, let it be user definable.

The End Result:
you will be able to load up 1gb files, make changes in your favourite editor and save them without hassle.

what do you think? a good or bad idea?

ramonsky · Post by **ramonsky** » Fri Nov 21, 2003 9:59 am

Doesn't this happen already, thanks to the miracle of Virtual RAM (aka Swap-File) that is built into Windows?

The problem you describe is a problem which affects all sorts of programsL image editors; music editors; 3D-model editors, etc. They all eventually have to face the possibility that the document might not fit in RAM. Almost universally, the application simply relies on the operating system to manage the swap file. It's a good system. It works. Sure - things slow down when you hit the out-of-real-RAM point, but that would still be true even if TextPad handled the situation internally.

But text editors are the LEAST affected by such considerations, because text files are relatively small compared with most other kinds of document. In general, text files are tiny compared to (for example) video clips. If you're running out of RAM when editing a text file, then either you have humungously huge text files or a miniscule RAM,

Just buy more RAM. Seriously.

Jill

maniac · Post by **maniac** » Fri Nov 21, 2003 3:20 pm

First of all, I don't think he was actually talking about loading it from memory, rather, streaming it from the file, which wouldn't deal with a swapfile. However, this would be really hard to implement, at least in my opinion.

kaimiddleton · Post by **kaimiddleton** » Sat Nov 22, 2003 7:24 pm

vedit, which has been around for a while, purports to handle large files well. It ain't cheap tho'.

http://www.vedit.com/

Quickly edit huge, tricky files:
Database, log, mainframe files
Binary, control & graphics chars
Fast, flexible, multi-key sorting
Huge 100+ Megabyte files
Lines with 100,000+ columns