Author Topic: RANT! Security in development.  (Read 1885 times)

0 Members and 1 Guest are viewing this topic.

Offline paulcaTopic starter

  • Super Contributor
  • ***
  • Posts: 4247
  • Country: gb
RANT! Security in development.
« on: August 20, 2024, 12:28:21 pm »
Year on year this gets worse and worse and worse.

Iron Man: You have a REST API you are testing.  You send it a request and it says:

400 Bad Request

Now.  You would like to know why it was a bad request,yes?  Well, that is not available.  If it was to be made available on the response message then a bad actor could use this to learn more about the system.

So all errors are mute, void, pointless, omitted and all reasons obsfucated.

As it turns out, it was NOT a bad request, of course it wasn't.  It's just that 400 Bad Request is the "lying to the hacker" response. 

So off you go into the logs and find what really happened

"Parse error on access token".

Aha!  Now how did that happen.  Fast forward 20 minutes of investigation and into the logs of the auth provider and...

"access denied".

Finally.  We find the root cause.

So, why did the auth provider not return a 403 forbidden or a 401 authentication required?  Because, you guess it, that would give the bad actor information.  So instead of handling errors they are allowed to propogate and produce a much more fundamental, but completely unrelated error.

I am not a bad actor, I am the god damn developer, why, why do we make this bed of nails for ourselves?

Can we not have proper error handling, proper error repsonse bodies, log snippets and tracing information when we are running dev?  Is this too much to ask?

Answering my own questions....  you try getting that ticket into the tight timelines with the customer and management all bearing teeth for deadlines.  It will get bumped down and down teh list until it's not done at all.   That or the InfoSec guys will reject the approach if it is even in the final build, regardless if it was disabled.

So we are stuck with

"Something went wrong."
"What?"
"Meh."
« Last Edit: August 20, 2024, 12:30:49 pm by paulca »
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline eutectique

  • Frequent Contributor
  • **
  • Posts: 445
  • Country: be
Re: RANT! Security in development.
« Reply #1 on: August 20, 2024, 01:18:26 pm »
Raise a ticket with IT infrastructure dept, mark its priority as "blocking" and set your ticket dependent on that one.

Then go to the next ticket on your list.
 

Offline dferyance

  • Regular Contributor
  • *
  • Posts: 198
Re: RANT! Security in development.
« Reply #2 on: August 20, 2024, 03:50:01 pm »
Usually security policies are just that, policies, and not thought. I've seen UUID generation code flagged a security risk because it didn't use a cryptographically secure random number generator, even though the UUID didn't need to be cryptographically secure and figuring out the next UUID didn't allow for anything to go wrong or be accessed. Security is done by checkboxes, not thinking through the system security.

There is merit to the whole not sending errors back approach. But it's got to be within reason. It isn't possible to do proper error handling without knowing what the error is. Usually this is handled via a flag on the web server to allow for more detailed error responses in dev. But even in prod, you need some kind of error information.

It sounds like you need better logging though. You shouldn't have to dig into multiple systems to see the log. The logs should be consolidated to a single place to see all of what happened. Without this, logging isn't of much use.
 

Offline Kjelt

  • Super Contributor
  • ***
  • Posts: 6572
  • Country: nl
Re: RANT! Security in development.
« Reply #3 on: August 20, 2024, 07:51:57 pm »
Yes security is needed, no it does not need to block normal work.
At my work we run everything in a VDI (Virtual desktop infrastructure) for security reasons.
Last year the security dept. has tightened access.
Now if I run a test and redirect the std screen output to a file, I am unable to get that file to my windows environment to share with colleagues, print it or mail it.
The only option I got left is to open a text editor inside my vdi and take screenshots and mail those.
Completely unworkable, but hey if we even can not get data binary copied to ourselves , an intruder also can not do it  |O
 

Online Bicurico

  • Super Contributor
  • ***
  • Posts: 1762
  • Country: pt
    • VMA's Satellite Blog
Re: RANT! Security in development.
« Reply #4 on: August 21, 2024, 12:37:04 pm »
Every admin's dream is a system tat users cannot access, so that it will never compromised.

https://bofh.bjash.com/
 
The following users thanked this post: vk6zgo

Offline paulcaTopic starter

  • Super Contributor
  • ***
  • Posts: 4247
  • Country: gb
Re: RANT! Security in development.
« Reply #5 on: August 21, 2024, 12:45:23 pm »
Every admin's dream is a system tat users cannot access, so that it will never compromised.

https://bofh.bjash.com/

Yep.  In a place I worked we got an email from devops that they would no longer be permitting the use of Java debugger JWP on the dev servers.

I bit the hook and replied?

"Da fu dudes?  Why?"
"Well it allows for remote code inspection, modification and execution, so it's insecure."
"Yes.  That is what it's for."
"Yes, that's why we banned it."
"Because it's useful?"
"To an attacker yes."
"If you want your organs fixed, you have to let the surgeon in, you get my drift?"

I was ignored.  We ignored them.  Then the department head sent an email with teeth.  Not only was an automated scanner installed to "kill -9" any process binding to the suspect ports, but anyone trying to run it "hidden" would get an offical warning.

When an ally intervened and asked, "What is the impact of banning this development tool, in development?"  I pointed out that without the ability to run software in a like for like environment and debug problems the two options are, debugging protocol or hundreds and hundreds of builds to deploy "probe logging" and "debug loggin" and to modify the code to try and trigger the bugs.  A change, push, deploy, test cycle takes 45 minutes.  Viewing a variable in a debugger takes seconds.  There is the impact.

No reponse.
« Last Edit: August 21, 2024, 12:48:46 pm by paulca »
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline dferyance

  • Regular Contributor
  • *
  • Posts: 198
Re: RANT! Security in development.
« Reply #6 on: August 21, 2024, 02:17:09 pm »
Quote
email from devops that they would no longer be permitting the use of Java debugger

Sounds like you also have the problem of people not knowing what devops is. The idea is combining development and operations into one. Presuming you are a developer (which is a safe assumption given the topic of a debugger), you shouldn't be getting an email from devops... you are part of devops. So do you give yourself permissions?
 

Offline paulcaTopic starter

  • Super Contributor
  • ***
  • Posts: 4247
  • Country: gb
Re: RANT! Security in development.
« Reply #7 on: August 21, 2024, 05:47:23 pm »
Quote
email from devops that they would no longer be permitting the use of Java debugger

Sounds like you also have the problem of people not knowing what devops is. The idea is combining development and operations into one. Presuming you are a developer (which is a safe assumption given the topic of a debugger), you shouldn't be getting an email from devops... you are part of devops. So do you give yourself permissions?

Like all shit buzz words it gets used to mean many things.  Would "platform", "infra", "core" team do you better?  "They whom own the build servers, CI/CD, dev servers etc."
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 3884
  • Country: us
Re: RANT! Security in development.
« Reply #8 on: August 21, 2024, 08:14:28 pm »
Year on year this gets worse and worse and worse.

Iron Man: You have a REST API you are testing.  You send it a request and it says:

400 Bad Request

Now.  You would like to know why it was a bad request,yes?  Well, that is not available.  If it was to be made available on the response message then a bad actor could use this to learn more about the system.

So all errors are mute, void, pointless, omitted and all reasons obsfucated.

This just means you are working in a defective environment.

There is a right way to do this.  You provide developers and testers the full error trace, but redact it when displaying to end users.  End users can't do anything about the difference between 400 and 500 errors anyway.  It's a pretty general rule: show people messages that are actionable.  That is different for developers vs. users.

One way to do this is to have the application server provide full, detailed error messages.  Then use a reverse proxy for access (most people do this anyway for SSL termination, load balancing, and separating static data from dynamic).   The proxy then strips off the internal diagnostic information to provide a user-relevant error message.  Developers and internal testers are then given a mechanism to access the real message.  One way is to have a seperate proxy instance that only authorized users can access, or the proxy can check the user identity internally.  Another option is for the proxy to log the error message and put an opaque log ID in the user visible message.  Then give the developers a query interface where they punch in the ID and get the log message.  This can potentially be used by customers as well if they file tickets.

Of course all of these options take some effort to set up.  If you are working at a place that doesn't prioritize developer tooling, it can be hard to get traction for this.  But guess what, places that don't prioritize developer tooling don't care about productivity, so just ignore the problem.
 

Offline radiolistener

  • Super Contributor
  • ***
  • Posts: 3984
  • Country: ua
Re: RANT! Security in development.
« Reply #9 on: August 21, 2024, 10:30:44 pm »
There is one more thing, modern servers on cloudflare with high load may use protection, and it may returns error even if all is okay, just your TLS stack is not approved to be used with server, so don't surprise if exactly the same request may work on one OS and may not work on another OS or from another application with different TLS library...  :-//
 

Offline paulcaTopic starter

  • Super Contributor
  • ***
  • Posts: 4247
  • Country: gb
Re: RANT! Security in development.
« Reply #10 on: August 22, 2024, 09:54:25 am »
There is a right way to do this.  You provide developers and testers the full error trace, but redact it when displaying to end users.  End users can't do anything about the difference between 400 and 500 errors anyway.  It's a pretty general rule: show people messages that are actionable.  That is different for developers vs. users.

Don't get me wrong I know many ways to fix it.  The tricky part is I am not the one in control of it.  The platform is customer side, on-prem, k8s.  Sure we have input, but ultimately the architecture and infrastructure is already written in stone.  Government projects, you wouldn't expect otherwise.

Truth be told the most common way, these days, is to just use a log aggregation service and transaction ids on requests so a query can pull back all related logs from all related services.  I think we might even have that in the current project.  I was just already in the k8s management UI so I could open the app logs directly.

Specific to my ironman however.  We techies are surely familiar with the basic web auth login, like the one of this forum.  Now, if you put the incorrect password in, it will not tell you that your password is incorrect.  That would be "telling".  It would validate that at least the username is correct.

In an enterprise (mciro)service architecture individual services all require authorisation.  So when service A wants to request a data look up from Service B (say) it will have to provide proof of identity/authentication/authorisation details to service B.

Having each and every service handle ident/auth routines and data flows we typically move the whole thing "out of band" and use an authentication service.  Sometimes called "IDAM".  Any service can submit a pre-shared key, usually specific certificates, to the IDAM service and ask for a token for a role.  As long as their credentials/certs are valid, they will receive an access token.

When service A now wants to request of service B, it sends the access token it requested from IDAM to service B.  Service B then asks the IDAM server is the token is valid.  (There are shortcuts with crypto validation that don't need a network hop).  If the token is valid for the authorisation Service A actually requested it will be permitted.

Bring the web login back to mind now.  Service A has something wrong with it's authorisation and ACL matrix.  When service A requests it's token for Role_ABC it is denied.  However, rather than deny the authorisation, the service will simple, ignorantly as all hell, send you a 200(ok) Empty response.

As a bad actor trying to brute force your way in, this will be exceedingly annoying and unfruitful.

However, it is also exactly that annoying and unfruitful in development.

Poor or non-existant error handling downstream of this often means the "Blank" token is then passed on verbatim to the request.  So the actual failure manifests several steps along the chain from the cause.

Not all of the "poor or non-existant error handling" is the teams fault.  A huge amount of all of the above is wrapped up in libraries.  Those libraries all are auto-magically-voodoo-coupled/injected in amazingly complicated ways so that you don't need to write much buggy code in addition to what comes as "black magic" from the frameworks.  That is until the thing detonates inside it's lovely opague black box and all the ugly complicated goo comes out in stack traces 150 deep in abstractions and it's some dumb as parsing call that throws an IllegalArgument in the bowls of spring when it tries to split a token on ".".  Where exactly would you hook that error handling code into?  (That is answerable, using event Filter chains etc. but digression).

The general ethos of development, I find, in enterprise, is to divide, compartmentalise and conquer.  If it fails bin it.  Move on.  Because projects are split up into so many components, each one is fairly cheap to replace.  So true diligence on "life expectancy" of a service is deinvested in favour of rapid application development techniques.  In fairness, the amount of effort you would need to spend to "overhaul" a 2000s REST API today would be far, far in excess of just rewriting it in a modern REST API framework.  90% of the problems you had to solve yourself bad then are gone, done, buried in extensively tested framework code.

A smaller number of people devise the required concoction of services, they spec each one broadly, define their integrations/interfaces/contracts, and then they can be farmed out and divided up for a team or multiple teams of devs.  Each component is a dozen days maybe.  Individual "projects" at the application "leaf" layer are just not that valuable individually.

Consider if you had an MCU project for a desktop product with a few interfaces and a colour display.

You "could" create one single firmware and try and get the core devs to work well with the UI devs, but you are likely to run up some bills on communications between the teams and code merge artifacts, code clobbers etc. etc.

You could split the development entirely and give each an MCU of their own.  Assuming the bean counters were asleep or that MCU devs cost as much as enterprise devs, I don't know.  Anyway, in this case, you would define the interface of communications and contracts between the core MCU and the UI MCU.  Now each is free to develop along their own path and as long as they never break that contract everything works.  The communications and contention points come down to a much more manageable scope of the integration specification.  In enterprise the economics means the later is nearly always preferred.

If you go too far and divide things up too far, it's just a sliding scale of "overhead" versus "manageability".  At some point you end up in "public toilet loo roll" territory as I call it.  You aren't actually solving any complexity anymore, you are just aimlessly smeering it around in circles.
« Last Edit: August 22, 2024, 10:01:35 am by paulca »
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline coromonadalix

  • Super Contributor
  • ***
  • Posts: 6569
  • Country: ca
Re: RANT! Security in development.
« Reply #11 on: August 22, 2024, 10:08:58 am »
evolve adapt or die  ....   this is what we live in technology
 

Offline paulcaTopic starter

  • Super Contributor
  • ***
  • Posts: 4247
  • Country: gb
Re: RANT! Security in development.
« Reply #12 on: August 22, 2024, 10:57:49 am »
Security should be optional.

The reason is becoming mandatory is because it's difficult.  If I have a more junior developer who has been tasked to "green field" spin up a new project that has, for instance, a REST API server component.  I will pressure them increasingly to pause the "fun" real functional development and incorporate the required company security aspects.  The reason is that retro-fitting security to an insecure application is actually harder than starting with the burden of security in the first place.  (Exactly as can be said for writing testable code).

So SDD or Security Driven Development has basically taken hold.

Before you turn a single CPU cycle, set ACLs to "Deny All", "Deny Everyone".  Now test you cannot access it.  Add functionality.  Test you still cannot access it.  Now configure the security to allow just the right auth tokens in just the right circumstances, with just the right SSL certs.  Now test you still cannot access it without providing every exactly correct details.

In 28 days when the company rotate those certs, you realise you need an automated way to handle that.

It's catch 22.  If you develop without security you can get a MASSIVE amount more done.  However you then have a TON of pain trying to retrofit security into code not designed for it.  If you develop with security, it is a constant drag, constant flux, constant hurdles.  Every time they need to add another layer of protection or security in the ever evolving field you have to spend time fixing your end before you can get back to work.  You spend 40% baby sitting the security just to be able to develop.

In larger enterprises, banks, gov, telecoms etc.  There is usually a very clean cut boundary between "development" and "platform".  The "platform" is usually a shared environment(s) which are strictly controlled to meat the company security and auditing requirements.  "Developers" are considered "untrusted" within the platform space.  We have no control.  For good reason.  It's a shared environment.  If you give devs access to make config changes, then TeamA will change config to suit their needs, breaking TeamB and C.  Worse.  When TeamA then advance to UAT that config is not there and their application fails flat.  Their config request is rejected and they end up weeks late.  The same argument applies for "no local run" ethos.  If you allow people to develop against their own little bubble test harness which they control, when it comes to deploying to the "real" environment, with all that lovely security, encryption in flight, on the wire and at rest... and it fails spectacularly.

At the end of the day, I think it just makes development no fun. :(  We can't have nice things, because we have bad people.  In fact we can't even assume or trust our own family are nice we must assume they are bad actors masquerading as them.  (Literally we had training on this around social engineering)
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline paulcaTopic starter

  • Super Contributor
  • ***
  • Posts: 4247
  • Country: gb
Re: RANT! Security in development.
« Reply #13 on: August 22, 2024, 11:13:50 am »
Contrary to my rant on SSL...  I have worked in two very large enterprises which did indeed "ban SSL" within their production cluster.

No names, but household names.  The reasoning behind it was that SSL "encryption inflight" was just too hard to maintain end-2-end.

Other polices said that every application needs a unique SSL cert.  There was a process to request these.  However when it came to "inter-service" connections there were multiple distinct ways of accessing other services.  You could go out the "hairpin" route to the edge node loadbalancer/proxy layer and use the services 'actual' SSL host name.  Or you could use the internal cluster URI for the service.  This route is FAR faster, it usually doesn't even transit a NAT boundary and is often "in-stack" 0 hop.  However.  That URL is dynamic and volatile.  Thus the SSL layers will not accept it.

After dozens of application teams hammered the processes for registering all their certs on the proxies and suddenly all internal traffic between services was bouncing through the proxies they had, had enough.

They managed to get approval from the regulators to declare the kubernetes clusters "opaque" and thus free from the encryption-in-flight regulations.

Immediately all teams gladly removed the SSL filter chains and linked to the internal dynamic cluster hosts.
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf