Dear lazy web,
Friend of mine ask me for help to find out bottleneck of performance problems he’s dealing with.
Basically it looks like the most problematic are I/O operations and main reason for that are thousands of read/write operations on directory having about 150k files.
Solution is easy right? Spread those files among some hashed subdirectories and we’re fine.
But it is not so easy. There are about 200 bash/perl scripts operating on those files.
So it’s not so simple as `mv aaaaaaaaaaaaa.txt a/aa/aaaaaaaaaaaaaa.txt` and so on.
Those scripts were developed for the past 10 or more years… so yes… noone really knows how and why do they work.
So I’m thinking about another approach which should be much easier and cheaper than rewriting hundreds of scripts.
Something like proxy filesystem between those scripts and files.
So we could still move those files to hashed subdirectories but show them as they are in single directory for the scripts.
Anyone knows about such filesystem? Or is it something that has to be written?
And another question… is it worth to go this way? Is there a chance that some kind of FUSE proxy filesystem with really fast hashed subdirectories will be faster than filesystem with one directory with a lot of files?
Anyone tried such approach?