Author Topic: Server Error Reports  (Read 188277 times)

0 Members and 1 Guest are viewing this topic.

Offline gnif

  • Administrator
  • *****
  • Posts: 1178
  • Country: au
Re: Server Error Reports
« Reply #325 on: September 01, 2020, 06:00:04 am »
Oh it certainly is, but it's not something that can be fixed easily.

SMF is dumb and just allows you to upload any file, and when it's an image attempts to create a thumbnail at render time (not at upload time), not only does this mean that multiple concurrent page loads will try to generate the thumbnail at once, if there are multiple images on the page, it will do them all at once (which can be slow) and cause a timeout, preventing the database from being updated with the information about the thumbnails.

The core workflow is very broken and fixing it will require overhauling how things work here. Such a "fix" would make SMF incompatible for future updates/upgrades.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 2624
  • Country: pl
Re: Server Error Reports
« Reply #326 on: September 01, 2020, 06:04:44 am »
At least the previewer should be fixed not to choke on malformed PNGs.
Is it exploitable for RCE too? ::)
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1178
  • Country: au
Re: Server Error Reports
« Reply #327 on: September 01, 2020, 06:11:24 am »
I did add some idiot checks for this some time back but upgrades to SMF and/or modules have undone it. I will schedule some time to investigate and re-instate what I can.

I am not sure if it's a RCE issue, IIRC it's the thumbnail preview feature which I think is an "addon" (if you can call SMF addons 'addons', more like diffs/patches)
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 2624
  • Country: pl
Re: Server Error Reports
« Reply #328 on: September 01, 2020, 06:25:36 am »
Hmm, I found this, which suggests that thumbnails are core SMF functionality. I have no idea, never administrated SMF myself.
https://wiki.simplemachines.org/smf/SMF2.0:Attachments_and_Avatars

RCE could possibly be an option if it's some old lousy C program which generates those thumbnails. If they actually wrote their own PHP scripts to parse those PNGs then maybe not. Again, no idea how they actually do it.
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 2624
  • Country: pl
Re: Server Error Reports
« Reply #329 on: September 01, 2020, 06:40:37 am »
Isn't this the culprit?
https://github.com/SimpleMachines/SMF2.1/blob/release-2.1/Sources/Subs-Graphics.php#L395

It seems to use one of three libraries, depending on availability. I suppose you could see if installing a different backend solves it, or file a bug with the backend's vendor. I'm not sure if I would want to run remotely executable C binary which can't even recognize that it's being fed a BMP instead of PNG and tries to process it instead of bailing out.

Or maybe the backend does bail out and SMF goes nuts? Then it would be an SMF bug.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1178
  • Country: au
Re: Server Error Reports
« Reply #330 on: September 01, 2020, 06:47:12 am »
It needs to be fixed in SMF, it just hands everything to `imagecreatefrom` based on the extension.

```
Code: [Select]
        // A known and supported format?
        if (isset($default_formats[$sizes[2]]) && function_exists('imagecreatefrom' . $default_formats[$sizes[2]]))
        {
                $imagecreatefrom = 'imagecreatefrom' . $default_formats[$sizes[2]];
                if ($src_img = $imagecreatefrom($destination))
                {
                        resizeImage($src_img, $destination, imagesx($src_img), imagesy($src_img), $max_width === null ? imagesx($src_img) : $max_width, $max_height === null ? imagesy($src_img) : $max_height, true, $preferred_format);
                        $success = true;
                }
        }

This will call `imagecreatefrom` + extension, ie `imagecreatefrompng` with invalid data. It's just plain dumb. Tell a library that this is a PNG and hand it something else, expect things to break.

The solution here is to actually open the file and read the header to determine it's type. I am in the middle of writing a fix in now :)
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 2624
  • Country: pl
Re: Server Error Reports
« Reply #331 on: September 01, 2020, 07:02:08 am »
And what if I take a BMP file and patch the PNG magic number into it? :P
Calling wrong function is one thing, the library being rubbish and failing to sanitize untrusted input is another :scared:
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1178
  • Country: au
Re: Server Error Reports
« Reply #332 on: September 01, 2020, 07:10:55 am »
That's just it, the library is not failing, it's doing exactly what it's supposed to. The issue is that it's slow at it, and consumes a ton of ram, hitting the PHP limits and it gets terminated.
A highly compressed PNG with large dimensions will do exactly the same thing.

> And what if I take a BMP file and patch the PNG magic number into it?

gd will bomb out and stop, the issue is that GD can and will load a BMP as it's a generic image processing library that PHP wraps.

At it's core, the issue is SMF's poor design, image thumbnail generation should be a background task done by the server, NOT as part of a HTTP request.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 
The following users thanked this post: mnementh

Offline magic

  • Super Contributor
  • ***
  • Posts: 2624
  • Country: pl
Re: Server Error Reports
« Reply #333 on: September 01, 2020, 07:22:26 am »
This function should return an error when it's presented with a BMP file, not eat RAM like crazy. A BMP is not a highly compressed PNG.

the issue is that GD can and will load a BMP as it's a generic image processing library that PHP wraps
Does that mean that uploading the very same BMP with BMP extension causes the same problem?
Because if not, that's still a bug in libgd's PNG loader, rather than any problem with BMPs.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1178
  • Country: au
Re: Server Error Reports
« Reply #334 on: September 01, 2020, 07:28:38 am »
>Does that mean that uploading the very same BMP with BMP extension causes the same problem?
>Because if not, that's still a bug in libgd's PNG loader, rather than any problem with BMPs.

No idea, feel free to debug this and chase it with them. Simple fact of the matter is, we need a fix now. Fixing this issue is already well beyond the scope of the services I render for dave as it is.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 16600
  • Country: gb
Re: Server Error Reports
« Reply #335 on: September 01, 2020, 09:29:51 am »
It's definitely libgd's fault. A DoS attack from malformed input is definitely CVE-worthy. It should return E_WHAT_THE_FUCK_DID_YOU_GIVE_ME error or something if it can't parse the file.

Workaround....

Assumption: this is a FPM process or something running with nginx in front of it. Issue is memory ballooning

1. Set proper memory limits in systemd for the FPM process. See: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LimitCPU=
2. Set systemd to auto restart the FPM process: Restart=always
3 (optional but recommended): run more than one FPM instance and use nginx as a balancer so that if one fails there's more left.

This will mean if it does crash or start gobbling RAM it'll restart.

Edit: can someone send me the file? I'll create a marketing web site for it "phpb0rk" and make £50k from selling it to some terrorists in NK  :-DD :-DD
« Last Edit: September 01, 2020, 09:32:13 am by bd139 »
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1178
  • Country: au
Re: Server Error Reports
« Reply #336 on: September 01, 2020, 09:32:37 am »
Quote
This will mean if it does crash or start gobbling RAM it'll restart.

Why do you think the site didn't die? Solutions/protections for these issues are already in place. It doesn't however stop a bunch of people consuming all available PHP processes because of a "slow page load" hitting refresh.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 16600
  • Country: gb
Re: Server Error Reports
« Reply #337 on: September 01, 2020, 09:39:40 am »
Well it's hard to debug from the outside. I'm pissing in the dark here :-DD

Add more cycles is about all you can do then.
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 2624
  • Country: pl
Re: Server Error Reports
« Reply #338 on: September 01, 2020, 09:55:53 am »
I did a quick test on a Debian 10 system with PHP 7.3.19. Not sure how to test libdg version, I don't have root on that box and can't find a file like that in /usr/lib.

Code: [Select]
<?php
$i
=imagecreatefrompng("test.bmp");
for(;;);
?>

PHP Warning:  imagecreatefrompng(): 'test.bmp' is not a valid PNG file in
Standard input code on line 2
^C
Memory usage didn't increase at all after hitting CTRL+D to execute the typed code. Maybe I need some special BMP file, maybe it's fixed in this version.
Memory usage did increase a few megs if I used frombmp instead or converted the file to PNG.

So dunno, check if everything is up to date and pray that there are no serious vulns in there, I guess.
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 16600
  • Country: gb
Re: Server Error Reports
« Reply #339 on: September 01, 2020, 10:01:03 am »
It's probably a broken BMP file that just happens to skip whatever validation is. It might even be intentionally malformed. I hope your FPM is running as an unprivileged process  :popcorn:

To note libgd has a hell of a lot of CVEs against it which are similar: https://www.cvedetails.com/vulnerability-list/vendor_id-6668/Libgd.html
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 2624
  • Country: pl
Re: Server Error Reports
« Reply #340 on: September 01, 2020, 10:02:21 am »
Well, I could try to upload my test.bmp here as killeevblog.png and see what happens :-DD
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 16600
  • Country: gb
Re: Server Error Reports
« Reply #341 on: September 01, 2020, 10:06:16 am »
 :-DD

Really to debug this we'd need:

1. PFP FPM version
2. OS version
3. Libgd version
4. Exploding BMP of doom.
5. Probably about 3 hours of pain

 

Offline gnif

  • Administrator
  • *****
  • Posts: 1178
  • Country: au
Re: Server Error Reports
« Reply #342 on: September 01, 2020, 10:11:56 am »
Quote
1. PFP FPM version
2. OS version
3. Libgd version
4. Exploding BMP of doom.
5. Probably about 3 hours of pain

Not going to happen for obvious reasons.

Please though stop posting about this here as this topic is special and I am alerted to responses to this topic. It's intended for actual outage reports.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 
The following users thanked this post: bd139

Offline mansaxel

  • Frequent Contributor
  • **
  • Posts: 965
  • Country: se
  • SA0XLR
    • My very static home page
Re: Server Error Reports
« Reply #343 on: September 01, 2020, 10:45:57 am »
Yup,

Issue fixed, another windows bitmap uploaded with a png extension. SMF goes nuts trying to load it and just hangs consuming tons of RAM.

Code: [Select]
FILE(1)                   BSD General Commands Manual                  FILE(1)

NAME
     file -- determine file type

<snip>

     The magic tests are used to check for files with data in particular fixed
     formats.  The canonical example of this is a binary executable (compiled
     program) a.out file, whose format is defined in <elf.h>, <a.out.h> and pos-
     sibly <exec.h> in the standard include directory.  These files have a
     ``magic number'' stored in a particular place near the beginning of the file
     that tells the UNIX operating system that the file is a binary executable,
     and which of several types thereof.  The concept of a ``magic'' has been
     applied by extension to data files.  Any file with some invariant identifier
     at a small fixed offset into the file can usually be described in this way.
     The information identifying these files is read from the compiled magic file
     /usr/share/file/magic.mgc, or the files in the directory
     /usr/share/file/magic if the compiled file does not exist.

<snip>

HISTORY
     There has been a file command in every UNIX since at least Research Version
     4 (man page dated November, 1973). 


Why does SMF degrade itself to using MS-DOS heuristics, when this is and has been available for some time?


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf