When str.lower() is a security vulnerability in Python

Join the discussion

• Hacker News
• Are there any security relevant international domain names, which require punycode? That's the real security problem: dont use IDNA 2003 for anything security relevant. redirect to an ASCII domain.
  by rurban
• That's a little over the top. string.lower() is not a security vulnerability. Not following the spec is the security vulnerability.
  by bawolff
• This is a typical clickbait title
  by orphea
• So the attack surface would be a bit flip on a domain name? Or more specifically, a unicode conversion flip, where an attacker could redirect to a malicious IP?

  Impressive to have found such a vulnerability!

  by ajd555
• Browsers (WHATWG) uses UTS #46, not IDNA2008.

  ada-url (https://github.com/ada-url/ada-python) closely tracks WHATWG spec and thus less likely to lead to parser differential vulnerabilities when interoperating with browsers.

  by hun3
• This could apply to any two systems that don't follow the same standard. not really a security issue, but the moral of the story is makes sure all components of your system use the same standard.
  by the_black_hand
• I was also startled when python did ß.upper() returns "SS". Which is kind of unsuspected in some cases (if string length changes with an upper call)
  by K0IN
• Seems kinda logical, since that’s how the Germans did it for a long time? ẞ wasn’t introduced until 2017 or so, AFAICT.
  by cpach
• That's in the standard. https://www.unicode.org/reports/tr21/tr21-5.html

  [SpecialCasing] Contains additional case mappings that map to more than one character, such as "ß" to "SS".

  by smallerize
• If this is a vulnerability every bug in every API is a vulnerability.

  This is very spacebar-heating.

  It's not a vulnerability, it's a bug. A system that used this bug in a way that relied on it to perform a security task would have a vulnerability.

  We need to stop seeing library functions that are not themselves security systems as having vulnerabilities.

  by philipwhiuk
• How else are we supposed to have claims that we are finding all the vulnerabilities in the world?
  by dazed_confused
• Reminds me of an old security incident at Spotify https://engineering.atspotify.com/2013/06/creative-usernames
  by jooon
• So someone used lower() from an unspecified version of Unicode when the standard was very specific about which to use. And they say "There's also a database of Unicode 3.2.0 data available on every version of Python (unicodedata.ucd_3_2_0) specifically for the StringPrep and IDNA algorithms", so the right version is available.

  And then the fix is to hardcode a bunch of special cases which again depend on exactly which version of Unicode is in use, and so will break again in the same way in future, rather than just using the right version?

  by ghusbands
• I'd assumed that the updating of the B3 dict takes place at runtime, either during module initialisation or lazily at the first call (that is, roughly as late as possible -- certainly, well after the wheel was built), just as a perf optimisation.

  But then I realised the code does (and must do) a lookup in the B3 table for each character anyway, so there doesn't seem to be any point. I suppose it means they can load the full 3.2.0 table once, use it to discover the exceptions and then immediately evict it from memory, keeping only the presumably smaller and faster-to-query B3 table of exceptions, but this seems pretty marginal...

  by akoboldfrying
• > The fix was to create new exceptions so that str.lower() would behave as if it was using Unicode 3.2.0 for only particular function. So, we go through each Unicode codepoint and record when the behavior of str.lower() is different when comparing the Unicode version shipped with Python and Unicode 3.2.0

  This sounds like a really hacky solution compared to implementing a separate frozen Unicode 3.2.0 lower.

  by ummonk
• To be clear (because the snippet is non-explanatory).

  For encode("idna") what they did is use lower() except where it would produce a result different to 3.2.0 and then instead use the result from 3.2.0 instead.

  Essentially they've frozen the IDNA encoding to be based on 3.2.0 by overriding any changes.

  by philipwhiuk
• The first sentence sounds as if they modified the implementemention of str.lower(). That would be bonkers, but that's not what they did.

  https://github.com/python/cpython/commit/7e109d084d55e7eb

  The important part is:

     # B.3 is mostly Python's .lower, except for a number
     # of special cases, e.g. considering canonical forms.
    +# To enforce Unicode 3.2.0 behavior of .lower instead of
    +# whatever Unicode version is included with Python we
    +# add unassigned or newly case-folding codepoints to
    +# the exception map, too.
     
     b3_exceptions = {}
     
     for k,v in table_b2.items():
         if list(map(ord, chr(k).lower())) != v:
             b3_exceptions[k] = "".join(map(chr,v))
    +for cp in range(0x110000):
    +    ch = chr(cp)
    +    # Assigned in current Unicode version
    +    # and supports case folding, but not
    +    # explicitly in B.2 or B.3 tables.
    +    if (unicodedata_current.category(ch) != "Cn"
    +            and ch.lower() != ch
    +            and cp not in table_b2
    +            and cp not in table_b3):
    +        b3_exceptions[cp] = ch  # Identity.

  by jwilk
• This idiocy is a big part of why it was so important to get Python people working on TLS implementations to understand that the defined mechanism for SANs (no the "alternative" in Subject Alternative Name doesn't mean in the sense of more than one, X.509 is originally for the X.500 system and the Internet repurposed X.509 so these are alternative names from the Internet) says that these are DNS names, they specifically are not to be understood as some sort of human readable text, and thus "decoding" them to Unicode is definitely nonsense even though Python really wanted to do that and I think used to do it or at least proposed to.

  The rule for how SAN DnsNames match againt like names, from the DNS is very, very simple so that you don't screw it up. You handle a single wildcard (ASCII * code 42 matches any single DNS label) and beyond that it's literally byte comparison. You don't care what these bytes mean, either the bytes are all identical or that's not a match and we're done.

  by tialaramex
• But it's not literally byte comparison; it's case-insensitive ASCII comparison.
  by MattSteelblade
• This sums up the whole Unicode disaster in Python3. People are conditioned to "encode" and "decode".
  by ltpoll
• > This is why calling str.lower() represents a difference in the implementation and the specification, and therefore a vulnerability:

  I wish there was some explanation how this is a vulnerability and not just a bug generating erroneous data.

  Vulnerability for me sounds like there’s a reasonable way to create an exploit from the bug, and I don’t see one here as someone who’s not very familiar with the topic.

  by echoangle
• Honestly it seems it's grabbing at straws

  There are a whole bunch of more consequential vulnerabilities before worrying about that

  by raverbashing
• Fingerprinting comes to mind.
  by amelius
• I'm too lazy to investigate further but my guess is that if there is a vulnerability here it has something to do with dns name spoofing.
  by zaphar
• I suppose I could envision a scenario like: Service A has a "reset your password" option. Someone with a "user@popular-unicode-domain.com" tries to use this option. Normally this fails when mangled-popular-unicode-domain-plus-garbage.com can't be found. Enterprising malicious actor registers mangled-popular-unicode-domain-plus-garbage.com, and now gets a hold of user password reset requests.
  by floxy
• My favourite example of this is the Chromium bug where enabling floating point flush-to-zero for WebAudio was used to cause deliberate heap corruption: https://issues.chromium.org/issues/382005099

  > We have a working exploit (OOB access in the V8 heap), our security folks put one together based on the example I posted above (and they're cleaning it up to post it here). In general, we find that correctness issues like this are pretty much always exploitable with a bit of effort (not even that much effort normally, just gluing together a few gadgets), so we treat correctness issues as security issues until they are proven not to be, rather than the other way around.

  The floating-point-to-heap-corruption chain here is... uniquely JavaScript, but in general getting two different implementations to disagree is the start of lots of interesting inconsistent behaviour.

  by wren6991
• Author here, that's a good idea. A straightforward way to exploit an implementation differential like this is if you have a software system that contains two different implementations of IDNA 2003 processing user input. One part of the process processes the domain correctly, the other incorrectly, and in this case you can have one part of a system (such as a policy/filter) "see" the data one way and the other part of the system (such as, taking an action as a result of the data) see the data in another way.

  Server-side Request Forgery (SSRF) is an example of such an exploit targeting a differential in implementations of URL parsers, which is similar to this implementation difference.

  by SethMLarson
• It creates a parser differential; two different components of the system can treat the same string as different hostnames. Things that have trusted hostnames, or privileged/admin hostnames that are screened out, or SSRF filters all depend on accurately comparing presented hostnames.

  This is pretty situational, though, isn't it? You still have to be dealing with IDN names.

  by tptacek