They've borked my homepage and screw up

[capt bullshot]

I'm having this homepage http://wunderkis.de since many years now. Hosted by a rather smallish provider, always working fine, having quick and competent response to support questions. Some interesting features like ssh access and DynDNS support so I could point subdomains to my home server. Worked as a charme.

Now, sometime ago this company was sold to a (presumably) larger one, they stated everything will continue as usual. It did, I wasn't concerned. Then they had some kind of a server failure, they reported this properly to me and after decent period of time, everything was up and running again. So what, shit happens, I wasn't concerned.

Now, a few days ago they announced to move the admin interface from SphereConf to their standard product called "Cloudpit", and everything is supposed to work as before.

Today they proudly announced "We moved your homepage to our way much better standard". No, nothing is the same as before, neither works as before. All the data moved to a spanking new VM environment, I was urged to create a new login. ssh and other access ceased, I guess I'm supposed to create new logins here. Didn't check the mailboxes yet. DynDNS failed, can't find any means to access NS entries in the colourful spanking new user interface.
Wrote a support ticket. Got an automated answer: Sorry we're on overload, a qualified answer will take at least 72 hours. Bwargh! Old provider had response times from minutes to 2 hours (while business hours).

Last but not least, Websites don't display properly anymore. OK, my handwritten html codes doesn't validate at all, I don't care as long as it is working proper in Firefox on Debian or Ubuntu. The old companies and my local apache server deliver the pages and Firefox renders, but the new one delivers in a subtle different way so Firefox doesn't render anymore. WTF? BTW, it does work on another domain hosted by the same package, but these pages aren't created by me and use a totally different approach (some CMS involved).

So for now, I've got to use my redundant DynDNS to get to my home server and try to find out somewhen what's the reason for their new apache to not properly deliver my pages. Bwargh! Awaiting their answer regarding the name server settings and DynDNS possibilities - or make the change to some other hoster immediately?

[fubar.gr]

I clicked your link and my browser (latest Firefox on win 10) apparently tries to download the index.php file instead of displaying it.

This is probably due to a misconfigured httpd.conf file.

[Richard Crowley]

I clicked your link and my browser (latest Firefox on win 10) apparently tries to download the index.php file instead of displaying it.

This is probably due to a misconfigured httpd.conf file.

Same with Chrome and Edge on Win10.

[rrinker]

Same. Fubar.gr probably nailed the cause. I'd guess their support is overloaded because this same issue is affecting ALL of their customers. Ooops.  Guess someone didn't test enough prior to trying to migrate every customer.

[Mr. Scram]

Same. Fubar.gr probably nailed the cause. I'd guess their support is overloaded because this same issue is affecting ALL of their customers. Ooops.  Guess someone didn't test enough prior to trying to migrate every customer.

It's called a live test 

[GreyWoolfe]

Same. Fubar.gr probably nailed the cause. I'd guess their support is overloaded because this same issue is affecting ALL of their customers. Ooops.  Guess someone didn't test enough prior to trying to migrate every customer.

No one ever tests enough.  I watch our client send out updates to fix an issue and usually causes 2 more issues.  Thank God it's their software and not ours, though we sold them the engine.  Sad to say, my company is just as guilty.  The stuffed shirts at corporate decided the call generating/inventory tracking software needed to be the same across the different divisions.  So naturally, what worked in our division and was customizable by program was replaced by software that is no longer customizable, harder to use and has parts that doesn't make sense.

[capt bullshot]

I clicked your link and my browser (latest Firefox on win 10) apparently tries to download the index.php file instead of displaying it.

This is probably due to a misconfigured httpd.conf file.

Yes, that's my educated guess too. Try http://cb.wunderkis.de/wk/ (my local copy of the homepage) - the index page isn't a .php at all but a .html with a little bit of embedded php.

[capt bullshot]

Wrote another ticket to them, demanding to restore all previous features.

BTW the new server is Nginx, before was Apache

mg@l7:~$ wget --save-headers -O - wunderkis.de | head
--2017-11-07 20:07:07--  http://wunderkis.de/
Resolving wunderkis.de (wunderkis.de)... 89.22.104.201
Connecting to wunderkis.de (wunderkis.de)|89.22.104.201|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1011 [application/x-httpd-php]
Saving to: ‘STDOUT’

-                   100%[===================>]    1011  --.-KB/s    in 0s     

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 07 Nov 2017 19:07:07 GMT
Content-Type: application/x-httpd-php
Content-Length: 1011
Connection: keep-alive
X-Accel-Version: 0.01
Last-Modified: Tue, 04 Jul 2017 11:12:26 GMT
ETag: "3f3-5537bf55aba80"
Accept-Ranges: bytes
X-Powered-By: PleskLin

Here's the culprit, I guess: Content-Type: application/x-httpd-php

[capt bullshot]

Oh, wow, got a response on my first ticket about ssh (new server, as I guessed) and DynDNS (no more supported) - doesn't look good for their future with me.

[idpromnut]

Nginx is a http proxy (that can additionally serve http requests). Highly likely they installed nginx as a reverse proxy to multiple backend http servers (apache or other) and got the configuration subtly wrong (or not 100% correct in all cases). From experience, this can be a pain to sort out, especially if they are not experienced with troubleshooting these types of setups.

As a side note, this type of change can *usually* be tested reasonably thoroughly before rolling out the changes to the masses at large.

Cheers,
id

Wrote another ticket to them, demanding to restore all previous features.

BTW the new server is Nginx, before was Apache

mg@l7:~$ wget --save-headers -O - wunderkis.de | head
--2017-11-07 20:07:07--  http://wunderkis.de/
Resolving wunderkis.de (wunderkis.de)... 89.22.104.201
Connecting to wunderkis.de (wunderkis.de)|89.22.104.201|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1011 [application/x-httpd-php]
Saving to: ‘STDOUT’

-                   100%[===================>]    1011  --.-KB/s    in 0s     

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 07 Nov 2017 19:07:07 GMT
Content-Type: application/x-httpd-php
Content-Length: 1011
Connection: keep-alive
X-Accel-Version: 0.01
Last-Modified: Tue, 04 Jul 2017 11:12:26 GMT
ETag: "3f3-5537bf55aba80"
Accept-Ranges: bytes
X-Powered-By: PleskLin

Here's the culprit, I guess: Content-Type: application/x-httpd-php

[capt bullshot]

So, by chance I found a PHP related setting in the net Cloudpit Tool, randomly changing it made the main domain (http://wunderkis.de) work again. Other stuff still screwed up:
ssh login to new server doesnt work due to "ssh: connect to host xxx port 22: Connection refused" - did someone forget to start sshd or didn't open port 22 through the firewall? WTF
ftp login works, but nothing more than the plain login, any attempt to actually access says "connection refused" - another firewall issue?
CNAME records to point subdomains to another dyndns service (freedns.org) are not accessible although promised so
The CloutPit's link to set the SSH password (WTF password, I had public key auth previously) points to a 404

That's where I stopped digging, wrote another support ticket, offered them to hire me as a beta tester 

[Mjolinor]

Well 22 is open.

rDNS record for 89.22.104.201: web125.dogado.net
Not shown: 984 closed ports
PORT     STATE    SERVICE
21/tcp   open     ftp
22/tcp   open     ssh
25/tcp   open     smtp
80/tcp   open     http
110/tcp  open     pop3
143/tcp  open     imap
443/tcp  open     https
465/tcp  open     smtps
554/tcp  open     rtsp
587/tcp  open     submission
993/tcp  open     imaps
995/tcp  open     pop3s
1124/tcp filtered hpvmmcontrol
3306/tcp open     mysql
7070/tcp open     realserver
8443/tcp open     https-alt

[capt bullshot]

Well 22 is open.

Right, it is open now, apparently someone fixed it ...
Still no login possible, I've set a password through their CloudPit Tool, but get "Permission denied, please try again."

[capt bullshot]

Ok, support gets somewhat more responsive, last answer within 5 hours, and some things fixed. Still not everything working as before, especially editing CNAME records isn't available and DynDNS appears not to be supported anymore - OK, there are others like http://freedns.afraid.org/ (that I'm already using for fall-back), but I need to edit the CNAME records to point some subdomains there ...

[technix]

Afraid.org can have some limitations on free accounts. I have migrated to dns.he.net for all three of my domains.

While you are at it, you may want to set up TLS using something free like Let’s Encrypt. TLS can protect both you and your visitors from some peeping Toms in ISPs.

[capt bullshot]

Thanks, dns.he.net looks interesting, but afraid.org fulfilled all my needs. I'm already using TLS for my home based server (with self-signed certificates), not for the hosted content. That's because they didn't offer it until now. TLS (https) is one of the new features I still have to explore with their new service. I consider switching to a smallish virtual server, they also offer such stuff at about the same price. Moving my domain to something like dns.he.net or afraid.org is a new aspect, so I could get independent of their DNS configuration tools, which is quite limited now in comparison to what it has been before the migration.

Where's your domain registered then?

[kulla]

For the certificate use letsencrypt.org which is free.

Regarding nginx, it cannot serve .php by default, php-fpm needs to be installed and running.

It usually listens on port 9000 or unix socket and in configuration you need to have proxy pass for .php files to php-fpm server.

[technix]

Thanks, dns.he.net looks interesting, but afraid.org fulfilled all my needs. I'm already using TLS for my home based server (with self-signed certificates), not for the hosted content. That's because they didn't offer it until now. TLS (https) is one of the new features I still have to explore with their new service. I consider switching to a smallish virtual server, they also offer such stuff at about the same price. Moving my domain to something like dns.he.net or afraid.org is a new aspect, so I could get independent of their DNS configuration tools, which is quite limited now in comparison to what it has been before the migration.

Where's your domain registered then?

All three of my domains are registered at GoDaddy and resolved at dns.he.net.

[capt bullshot]

Regarding nginx, it cannot serve .php by default, php-fpm needs to be installed and running.

OK, that's their desk, not mine, they've successfully resolved it yesterday.

[capt bullshot]

Did just a quick check: now there's the same 502 glitch available at my homepage that has been seen here at the forum from time to time ...

[chriswebb]

Thanks, dns.he.net looks interesting, but afraid.org fulfilled all my needs. I'm already using TLS for my home based server (with self-signed certificates), not for the hosted content. That's because they didn't offer it until now. TLS (https) is one of the new features I still have to explore with their new service. I consider switching to a smallish virtual server, they also offer such stuff at about the same price. Moving my domain to something like dns.he.net or afraid.org is a new aspect, so I could get independent of their DNS configuration tools, which is quite limited now in comparison to what it has been before the migration.

Where's your domain registered then?

All three of my domains are registered at GoDaddy and resolved at dns.he.net.

don't do business with GoDaddy. There are many, many more reputable and reliable registrars out there.

Did just a quick check: now there's the same 502 glitch available at my homepage that has been seen here at the forum from time to time ...

This is a proxy timeout. Your nginx frontend is having timeouts while talking to your php backend. Have them make sure your php backend is still up and running? This is one of the issues they may not realize they've introduced by moving to their new architecture. In your old setup, since php was running through modphp on apache, you really didn't have to worry about these types of issues because it didn't have to go through a proxy connection. If apache was crashed by your php code, they just had to restart apache but the site wouldn't respond at all or give a connection refused or something. Most likely it just wrote the error in the error log and didn't crash apache at all and you saw a 500 error. Not sure whats going on with php-fpm, maybe it is crashing on an exception?

[capt bullshot]

OK, the 502 was a unique glitch until now, I won't bother too much for the moment. Thanks for the technical explanation, that's quite new stuff for me.

Found out why the browser wouldn't display my pages properly:
After adding rsync myself to the server (I've asked the support to do so, but no response), I've re-synced the content with my local copy. Afterwards it was broken again
Found the culprit in this line of .htaccess: "AddType application/x-httpd-php .htm .html" - Apparently I had to add this to make the old server work, but breaks the new setup.

Seems the support fixed my .htaccess file due to my first complaint, but didn't tell me what they've done.

[chriswebb]

That is interesting. Would need to have a look at the configuration for your host specifically to see how theyve setup the proxy, but I'd imagine you don't need to trigger this via a .htaccess now.  More importantly the .htaccess you setup previously was for Apache. Nginx doesn't like them: https://www.nginx.com/resources/wiki/start/topics/examples/likeapache-htaccess/

Is your rsync messing with read/write permissions, like removing global reads? Nginx has a lot of problems trying to access paths with any part not globally readable or at least readable by the user running the nginx process. These will show up as 403s.

Whats the error you are seeing after uploading the new files?

Do you have shell access?

[capt bullshot]

Yes, I've got a shell to a limited container, have access to logfiles, my content, but none of the configurations.

I didn't experience problems with read permissions yet, but I'm aware of this so I'm always testing after syncing new files. With the old setup, I never had problems.
To be clear: I had to remove the line "AddType ..." to make things work again

I've seen a 403 on a URL that is supposed to show an automatic directory listing (the old apache setup was configured to allow this), not yet investigated or fixed.
A quick "chmod -R ugo+r *" didn't fix it, so I guess I have to lookup the right directives to write into .htaccess. It's not my day job to maintain webpages, I've got a good understanding of how things work here, but for details I'll always have to look up them up. I just hate them now for this effort they put on me 

[chriswebb]

Yes, I've got a shell to a limited container, have access to logfiles, my content, but none of the configurations.

I didn't experience problems with read permissions yet, but I'm aware of this so I'm always testing after syncing new files. With the old setup, I never had problems.
To be clear: I had to remove the line "AddType ..." to make things work again

I've seen a 403 on a URL that is supposed to show an automatic directory listing (the old apache setup was configured to allow this), not yet investigated or fixed.
A quick "chmod -R ugo+r *" didn't fix it, so I guess I have to lookup the right directives to write into .htaccess. It's not my day job to maintain webpages, I've got a good understanding of how things work here, but for details I'll always have to look up them up. I just hate them now for this effort they put on me 

Are your error logs telling you anything?  I am so confused about how your .htaccess is affecting nginx, unless they specifically import it into the nginx config unless php-fpm is affected by your .htaccess file. Nginx shouldn't be using your .htaccess file at all.

To enable directory listings (if nginx is the one serving the static content), you add an "autoindex: on" to the location directive in the config of the virtual host.

Code: [Select]

location /sample {
     autoindex: on;
}
I am really confused about what their intentions are. They are going to need to give you a way of modifying the configuration, I'd imagine, but I've never had to use nginx in a shared hosting environment.  I've only seen Apache used in conjunction with htaccess and usually some sort of web-based server administration solution. Is this just a work in progress for them? If you had the time for the initial setup, you could probably get this up and running on your own vm on something like aws, google or a small cheap vps provider. You'd also have full root over the system, so you could lock down a lot of those open services to just your ip address. That type of setup wouldn't require a ton of maintenance. You could probably just run the updates on a cron job. And if you don't use anything more than apache, php, mysql, rsync and a few other services, it may not take a ton of time to setup either. But I can definitely understand the hesitation, especially if you can leave it to capable people to manage... But I am seriously  :palm:ing at the way this hosting provider has handled this migration to nginx.