How to clone an Apache file directory from a website?

[Jackster]

Doing a bit of file archival work and we have a few public-facing Apache file directories like in this screenshot.



The directory structure is pretty simple but there are many of them.
The files are pretty standard but some are text files and some are binary.

What I would like to do is basically clone those directories onto my local storage.


I have tried WGET so far with some success.

Code: [Select]

wget -m -p -E -k -K -np "http://domain.ltd/dir/"As well as an assortment of flags.
But I am finding that it does not download the txt files correctly and sometimes the binary files are downloaded incorrectly also.

Does anyone have a better solution?
I don't need to generate the index pages either.

[Jackster]

A few solutions have poped up such as https://github.com/ArchiveBox/ArchiveBox and Archive Team's self hosted Warrior.

[retiredfeline]

It's been years since I had to do anything like this but I think I used httrack.

[Jackster]

It's been years since I had to do anything like this but I think I used httrack.

That is what I used first. Tried and tested bit of software. But it also had issues with the binary files and text files.

[Whales]

What exactly are the issues with the files you have downloaded?  Please give as much detail as you can.

Are you trying to clone that site's code assets including .php files so you can rehost it yourself?  You probably can't do that from a web browser, the Apache instance will likely "run" the php pages when you try to access them (rather than serve you the raw file contents).  You will need a different form of access to the server (eg a shared hosting control panel or ssh/scp/sftp).

[retiredfeline]

What sorts of issues? Linux doesn't have attributes for binary or text files but you can configure the Apache server your end to associate MIME types with particular file extensions. This is a configuration setup on the origin server that isn't exported to the world.

I noticed now that it looks like a Wordpress site. Dynamic sites like that where content is programmatically generated cannot be cloned from the website. You need access to the underlying Wordpress installation.

[AndyBeez]

Gain root access and FTP / SSH all of the files into a zip or tar archive.

[Jackster]

Should have added more info..

It is a publicly accessible directory for a game's downloadable mods and maps.
The files are compiled data files and txt configs.

I don't have access to the actual server's backend.

[golden_labels]

It is a publicly accessible directory for a game's downloadable mods and maps.
The files are compiled data files and txt configs.

I don't have access to the actual server's backend.

You still did not answer Whale’s question about the problems you are experiencing. We do not know, what you did download. Neither we do know what you consider “incorrect” and — most importantly — what is your reasoning behind considering something “incorrect”.

On a separate note: while dumping websites please consider using wget’s --limit-rate, --wait and --random-wait. That will decrease load variance, making it affect normal users less.

[Nominal Animal]

The only problem I can think of wrt. plain text files is that the server supplies a Content-Type: text/plain; charset=encoding with different encoding than you use on your local machine.  (Typically, the server provides UTF-8, and you use e.g. Windows Western European aka win-1252 aka Codepage 1252.)
The effect is that all ASCII characters are fine, but non-ascii characters are mangled.
One other quirk is that you might use a text editor that requires Windows CR-LF newline convention, whereas Linux, BSDs, Macs use LF-only.

There is no "fix" nor "error".

What you need to do, is view the file in your browser, and change the encoding to what the server used.
(Browser developers are stupid, and don't have any way of setting local text files' encoding to UTF-8, "because it is not a legacy encoding and is therefore unsuitable as a default".  Yeah, it is one of my pet peeves.)
Because you intend to do an archival, I recommend against recoding the files with e.g. iconv.

[Jackster]

Thanks to a suggestion by someone on ArchiveTeam's IRC, OpenDirectoryDownloader to index the URLs and then use WGET to download all the files.
It is working well and not messing up the config files.

https://github.com/KoalaBear84/OpenDirectoryDownloader