Multi Path Upload (MPU) / dynamic upload folder

[onoehring]

Hi,

just as a teaser:
I am working on a solution where images are stored in subfolders.
This solution "multi path upload" can be implemented with ease in existing applications.
Currently one client of mine has about 10 GB of files (approx. 18.000) in the /uploads folder. This definitely slows the request, which is actually at a medium time of 11 seconds now, before the page has been sent from the server to the browser. The rendering in the browser is no problem - this takes less than a second. The problem is the shear amount of files the file-system and server OS has to handle. Thus, my tool will create e.g. 1.000 subfolders and move uploaded files into those.
This will hold just about 1.8 files per (sub)folder and should speed up the application a lot again.

Installing the tool into the app (test-server) means adjusting the AG files where uploads can be done in 3 hooks-functinos and doing a single line change in /thumbnails.php. Then, one can call the MPU and let the install-section do it's work.
Installing and moving those aprx. 18.000 files is a matter of less than 30 minutes.

Stay tuned.

Olaf

[ckebbell]

Hi Olaf

I would be very interested in the MPU you have outlined here!

I have over 121000 files stored in my images folder and I have been trying to find a way to change the upload path based on the table. I have experimented with changing the $Translation on tablename_init, changing it in beforeinsert and afterinsert and changing it in Header-extras but nothing has worked!

I just wonder if your proposal will allow me to redirect files to images subfolders based on the table? I don't have a complex requirement, just for 7-10 subfolders which are named after those tables.

Thank you

[onoehring]

Hi chkbell and all others,

actually my script works using the numeric primary key to place images in specific subfolder. This way the amount of files should be pretty much equal in every subfolder. In your setting, having multiple tables and, if I under stand correct, you want something like this

Code: Select all

/uploads/tablenameAAA/img01.jpg 
/uploads/tablenameAAA/img20.jpg
/uploads/tablenameBB/img99.jpg

it could easily happen, that still very many files reside in a /uploads/tablename folder (making access to this slow again.)

If you manage to adjust my script to work for tablenames ... please publish so we can all make us of it

How MPU works
This I choose a different route which is described quickly with this example:

Code: Select all

 * Example: 
 *   regular upload path: /images
 *   primary key (pk) is integer, autonumber 
 *   MPUUPLOADPATHLENGTHNEW is set to be 3
 * Example results:
 * pk =     1 -> new upload path will be /images/001
 * pk =     2 -> new upload path will be /images/002
 * pk =    15 -> new upload path will be /images/015
 * pk =   370 -> new upload path will be /images/370
 * pk =  4891 -> new upload path will be /images/891
 * pk = 94891 -> new upload path will be /images/891
 * pk = 96001 -> new upload path will be /images/001
 * pk = 96015 -> new upload path will be /images/015

You will note, that my script will NOT work if primary key is not numeric (I always use autoincrement as internal PK, not showing this to the user and I have tried to use a string as PK in AG which made "funny things").

MPU works in my test(s) and real-time environment for uploaded images (png+jpg) and PDF (I did not test other filetypes but I suppose they should also work). Images can be zoomed in tableview (TV) and detailview (DV) if this has been set in the field properties in AG.

Installation ans setup
For all interested, I present a first release here (universal link will forward to the latest release).
Be aware, that the setup - especially for an existing AG application is quite extensive but actually easy. Everything is documented in the beginning of the PHP file - and, if you want nicer layout - in the PDF.

Installation will require changing some files a) in the hooks b) of the always (re)generated and setting some variables in the PHP file.

Release 1.04
Your reply reminded me to publish the current result.
You can download here: https://url.olaf-noehring.de/agmpu
(1 Zip: 1 PDF and 1 PHP)

Feedback appreciated. I would like to hear how many files you have in your uploads folder.

Olaf

[ckebbell]

Hi Olaf

Thank you! I will study your solution script this weekend and will see if I can adapt / implement into my use case / workflow.

I see your point about the growing number of files. As my database grows, I can see that this will become unmanageable! I have a number of child tables (e.g. contacts, reports, photos, drawings) linked to a parent table (e.g. Clients). The parent table does not have a numeric PK, but the child tables all do, so adaptation should work for me

For example, the Reports table has 18000 entries, so I am considering if I can adapt your script so that Reports are directed to a Reports subfolder, then a further subfolder based on the PK as in your example.

Over the last 10 years or so of using Appgini, I have added over 121000 files in the images folder, so I will need to 1) separate those files into subfolders based on the uploadtable, which is a timeconsuming task and 2) ensure that future uploads are directed to the correct tablename / subfolder. I think I will do that by year, to make task 1) easier and smaller chunks...

I will report back as soon as I am able to and if I can find a way to redirect to a tablename, I will share my code with everyone!

I have spent a lot of time adapting Appgini to my own uses but, if anyone needs code on how to capture / download a signature, integrate Reportico / PHPWord or use ajax calls to update / calculate fields as they are populated, please send me a private message and I will post something up as soon as I can. I am not on very often though, as I own a business that I run with Appgini, so I am mainly here at weekends trying new combinations of code!

Thank you

[onoehring]

Hi ckebbell,

I am looking forward to see your changed to the script.

As the script is integrated on a "per table base", e.g. in the Hooke/tablename.php files it should be no problem to skip a table (like your parenttable with no numeric pk).

I do not understand your 4th paragraph (over the last 10 years...), the list 1..4 you create. My script will create all subfolders based.on PK. There year should not matter and probably be saved as an extra field in your table (of needed).
Even if all your files would be from the sane year, the script would split them into 1,000 subfolders (if folder length is set to 3), resulting in just 120 files each folder as average.
Why put them in tablename/reports subfolder?

I.may extend the script to use the tablename as (additional) subfolder. What are the forums opinions about that? Would that be worth the time, e.g. a useful extension?

Olaf