Filter out errors out of mozilla build on command line

Mozilla build errors filtered once again under the build log

I wrote a small filter script that lists all build errors once again at the bottom of the whole build log, so that you don't have to look for them like for a needle in a haystack. Something I wanted mach to do natively.  But I was always told something like "filter yourself".  So here it is :)

  • Download this small script *)
  • Copy it to your source directory or somewhere your $PATH points to
  • run mach as: ./mach build | err

When there are errors during build, those will be listed under the build log and conveniently highlighted.

*) It's tuned for and mainly targeting mingw, but might well work on linux/osx too.

INACCESSIBLE BOOT DEVICE on Windows 10 boot after update of the Intel Rapid Storage Technology driver

Have a BSOD after you've installed the latest version of Intel RST from Intel's download center? During the boot, staring at the Windows logo and the spinning wheel, after a minute or two getting just INACCESSIBLE_BOOT_DEVICE error and "we must reboot" message?  Restarts, nothing helps?  Yeah, I've been there.

How to fix the BSOD

Note: can be applied only when you have updated the driver from a previously working version, since it counts with a previous driver file stored on your disk.

  • Check your BIOS and RAID setting are as expected, since I once encountered iRST update that screwed that up - actually turned off RAID!
  • During boot hold F8, if this doesn't work for you, you need to "Create a recovery drive" on a USB and boot using that
  • Choose Troubleshoot
  • Choose Advanced options
  • Choose Command Prompt, a command prompt window, as you know it, should open
  • My system drive was mounted as E:, if yours is mounted elsewhere, replace E: in below commands with that letter
  • At the prompt type:

    [text gutter="false"]
    cd /d E:\Windows\system32\drivers
    ren iaStorA.sys iaStorA.sys-bad-version
    cd ..
    dir /s iaStorA.sys
    [/text]

  • That will list something like this:

    [text gutter="false"] Volume in drive C has no label.
    Volume Serial Number is XXXX-XXXX

     Directory of E:\Windows\System32\drivers

    07/29/2015  19:44         1,462,720 iaStorA.sys
                   1 File(s)      1,462,720 bytes

     Directory of E:\Windows\System32\DriverStore\FileRepository\iastorac.inf_amd64_26544f4e51074f52

    05/28/2014  10:10           672,104 iaStorA.sys
                   1 File(s)        672,104 bytes

     Directory of E:\Windows\System32\DriverStore\FileRepository\iastorac.inf_amd64_61378e65f4f142a0

    07/29/2015  19:44         1,462,720 iaStorA.sys
                   1 File(s)      1,462,720 bytes

         Total Files Listed:
                   3 File(s)      2,806,928 bytes

    [/text]

  • For me the previous working driver file is apparently at

    [text gutter="false"]DriverStore\FileRepository\iastorac.inf_amd64_26544f4e51074f52[/text]

    , yours can be elsewhere, so update the source directory in the copy command below according that

  • Continue typing following commands, you should still be at the E:\Windows\System32 directory:

    [text gutter="false"]copy DriverStore\FileRepository\iastorac.inf_amd64_26544f4e51074f52\iaStorA.sys drivers\iaStorA.sys[/text]

  • You should see a "1 file(s) copied" message
  • [text gutter="false"]exit[/text]

  • Now normally reboot
  • Good luck!

After that Intel RST service still starts up, shows the status etc, disks are fine, everything seems to work, even after another (deliberate) reboot. Only weird thing was that System Restore for the C: drive was turned off. Not sure if it was caused by the RST update, by the boot problems, by some of my other manual changes (not listed here) or what.. Re-enabling it works fortunately well, so not an issue.

A good thing to do is yet to rollback the driver from the Device Manager (Storage Controllers/Intel(R) *** SATA RAID Controller/Properties/Driver/Roll Back Driver) to put the registry records back to the correct state.

Known iRST BIOS and iRST driver combinations

driver ↓BIOS 11.1.0.1413BIOS 14.8.0.2377
13.1.0.1058worksunknonwn
14.0.0.1143worksunknonwn
14.8.0.1042 BSOD unknonwn
15.2.0.1020 BSOD works

Dear readers, if you know about other versions combinations, please let me know so that we can fill this table with more data. Thanks in advance!

String parsing made simple with mozilla::Tokenizer

 

PL_strstr

 

I can see FindChar, Substring, ToInteger and even atoi, strchr, strstr and sscanf craziness all over the Mozilla code base. There are though much better and, more importantly, safer ways to parse even a very simple input.

I wrote a parser class with API derived from lexical analyzers that helps with simple inputs parsing in a very easy way. Just include mozilla/Tokenizer.h and use class mozilla::Tokenizer. It implements a subset of features of a lexical analyzer.  Also nicely hides boundary checks of the input buffer from the consumer.

To describe the principal briefly: Tokenizer recognizes tokens like whole words, integers, white spaces and special characters.  Consumer never works directly with the string or its characters but only with pre-parsed parts (identified tokens) returned by this class.

 

There are two main methods of Tokenizer:

  • bool Next(Token& result);

If there is anything to read from the input at the current internal read position, including the EOF, returns true and result is filled with a token type and an appropriate value easily accessible via a simple variant-like API.  The internal read cursor is shifted to the start of the next token in the input before this method returns.

  • bool Check(const Token& tokenToTest);

If a token at the current internal read position is equal (by the type and the value) to what has been passed in the tokenToTest argument, true is returned and the internal read cursor is shifted to the next token.  Otherwise (token is different than expected) false is returned and the read cursor is left unaffected.

Few usage examples:

Construction

  #include "mozilla/Tokenizer.h"

  mozilla::Tokenizer p(NS_LITERAL_CSTRING("Sample string 2015."));

Reading a single token, examining it

  mozilla::Tokenizer::Token t;
  bool read = p.Next(t);
  // read == true, we have read something and t has been filled
  // Following our example string...
  if (t.Type() == mozilla::Tokenizer::TOKEN_WORD) {
    t.AsString(); // returns "Sample"
  }

Checking on a token value and automatically skipping on a positive test

  if (!p.CheckWhite()) {
    throw "I expect a space here!";
  }

  read = p.Next(t);
  // read == true
  t.Type() == mozilla::Tokenizer::TOKEN_WORD;
  t.AsString() == "string";

  if (!p.CheckWhite()) {
    throw "A white space is expected here!";
  }

Reading numbers

  read = p.Next(t);
  // read == true
  t.Type() == mozilla::Tokenizer::TOKEN_INTEGER;
  t.AsInteger() == 2015;

Reaching the end of the input

  read = p.Next(t);
  // read == true
  t.Type() == mozilla::Tokenizer::TOKEN_CHAR;
  t.AsChar() == '.';

  read = p.Next(t);
  // read == true
  t.Type() == mozilla::Tokenizer::TOKEN_EOF;

  read = p.Next(t);
  // read == false, we are behind the EOF
  // t is here undefined!

More features

To learn more enhanced features of the Tokenizer - there is not that many, don't be scared ;) - look at the well documented Tokenizer.h file under xpcom/ds.

As a teaser you can go through this more enhanced example or check on a gtest for Tokenizer:

#include "mozilla/Tokenizer.h"

using namespace mozilla;

{
  // A simple list of key:value pairs delimited by commas
  nsCString input("message:parse me,result:100");

  // Initialize the parser with an input string
  Tokenizer p(input);
  // A helper var keeping type and value of the token just read
  Tokenizer::Token t;

  // Loop over all tokens in the input
  while (p.Next(t)) {
    if (t.Type() == Tokenizer::TOKEN_WORD) {
      // A 'key' name found
      if (!p.CheckChar(':')) {
        // Must be followed by a colon
        return; // unexpected character
      }

      // Note that here the input read position is just after the colon
      // Now switch by the key string
      if (t.AsString() == "message") {
        // Start grabbing the value
        p.Record();
        // Loop until EOF or comma
        while (p.Next(t) && !t.Equals(Tokenizer::Token::Char(',')))
          ;
        // Claim the result
        nsAutoCString value;
        p.Claim(value);
        MOZ_ASSERT(value == "parse me");

        // We must revert the comma so that the code bellow recognizes the flow correctly
        p.Rollback();
      } else if (t.AsString() == "result") {
        if (!p.Next(t) || t.Type() != Tokenizer::TOKEN_INTEGER) {
          return; // expected a value and that value must be a number
        }

        // Get the value, here you know it's a valid number
        uint32_t number = t.AsInteger();
        MOZ_ASSERT(number == 100);
      } else {
        // Here t.AsString() is any key but 'message' or 'result', ready to be handled
      }

      // On comma we loop again
      if (p.CheckChar(',')) {
        // Note that now the read position is after the comma
        continue;
      }
      // No comma?  Then only EOF is allowed
      if (p.CheckEOF()) {
        // Cleanly parsed the string
        break;
      }
    }

    return; // The input is not properly formatted
  }
}

 

Currently works only with ASCII inputs but can be easily enhanced to also support any UTF-8/16 coding or even specific code pages if needed.

New Gecko performance tool: Backtrack

Backtrack aims to show a complete code path flow from any point back to its source, crossing asynchronous callbacks, threads, processes, network requests, timers and any kind of implementation specific queuing plus capturing any I/O or mutex blockade.  The 'critical flow execution path' is put to a context of all the remaining concurrent execution flows.  It's then easy to examine how the critical flow is blocked and delayed by concurrent tasks.

The work is tracked in this bug, where you also find patches and build instructions.  There is also an add-on that, in Backtrack enabled builds, allows you to view actual captured data.

Click the screenshot bellow to view an interactive previewIt's capture of load of my blog main page till the first-paint notification (no e10s and no network predictor to demonstrate the capture capabilities.)

backtrack-preview-1

Backtrack combines*) Gecko Profiler and Task Tracer.

Gecko Profiler (PSP) provides instrumentation (already spread around the code base) to capture static call stacks.  I've enhanced the PSP instrumentation to also capture objects (i.e. 'this' pointer value) and added a simple base class to easily monitor object life time (classes must be instrumented.)

Task Tracer (TT) on the other hand provides a generic way to track back on runnables - but not on e.g. network poll results, network requests or implementation specific queues.  It was easy to add a hook into the TT code that connects the captured object inter-calls with information about runnables dispatch source and target.

The Backtrack experimental patch:

  • Captures object lifetime (simply add ProfilerTracked<class Derived> as a base class to track the object lifetime and class name automatically)
  • Annotates objects with resource names (e.g URI, host name) they work with at run-time
  • Connects stack and object information using the existing PROFILER_LABEL_FUNC instrumentation recording this pointer value automatically ; this way it collects calls between objects
  • Measures I/O and mutex wait time ; an object holding a lock can be easily found
  • Sticks receipt of a particular network response exactly to its actual request transmission (here I mainly mean HTTP but also applies to connect() and DNS resolution)
  • Joins network polling "ins" and "outs"
  • Binds code-specific queuing and dequeuing, like our DNS resolver, HTTP request queues.  Those are not just 'dispatch and forget' like nsIEventTarget and nsIRunnable but rather have priorities, complex dequeue conditions and may not end up dispatched to just a single thread.  These queues are very important from the resource scheduling point of view.

Followups:

  • IPC support, i.e. cross also processes
  • Let the analyzes also mark anything 'related' for achieving a selected path end (e.g. my favorite first-paint time and all CSS loads involved)
  • Probably persist the captured raw logs and allow the analyzes be done offline

Disadvantages: just one - significant memory consumption.

*) The implementation is so far not deeply bound to SPS and TT memory data structures.  I do the capture my own - actually a third data collection, side by SPS and TT.  I'm still proving the concept this way but if found useful and bearable to land in this form as a temporary way of collecting the data, we can optimize and cleanup as a followup work.