To reproduce the issue, create a free-style job that executes the following shell command :

echo -e '\0033[8mha:[HUDSON]\0033[0m'

Run a build, then try to view the console output : an OOM error is thrown. Or if you are running with a large heap (more than 500MB) you will see a 500MB surge in Hudson's heap usage. Which is actually worse, as the whole machine is suddenly impacted and in our case, it caused mad thrashing (see below for real-word, unintentional occurrence of the problem).

What happens is that ConsoleNote.readFrom() reads the above sequence as a serialized ConsoleNote. It first tries to decode the first few characters into an object size : this yields a very large integer, which is then used (ConsoleNote.java:220) to allocate a huge byte array.
A similar risk exists in ConsoleNote.skip() : many more bytes may be skipped than intended.

Suggested fix(es) :

• add a sanity check to ConsoleNote.readFrom() to ignore "unreasonably"-sized annotations
• make UnbufferedBase64InputStream throw when reading invalid Base64, perhaps by calling Base64.isArrayByteBase64 before Base64.decodeBase64.

Note: I did not stumble upon this bug by trying to attack the Hudson log parser
It is revealed by another issue (that I will submit later) : there seems to be a race condition somewhere that occasionally causes a log message to be appended just after the ConsoleNote preamble has been written, but before the size and object itself have been written.