New Firefox HTTP cache now enabled on Nightly builds

Yes, it's on!  After a little bit more than a year of a development by me and Michal Novotný all bugs we could find in our labs, offices and homes were fixed.  The new cache back-end is now enabled on Firefox Nightly builds as of version 32 and should stay like that.

The old cache data are for now left on disk but we have handles to remove them automatically from users' machines to not waste space since it's now just a dead data.  This will happen after we confirm the new cache sticks on Nightlies.

The new HTTP cache back end has many improvements like request prioritization optimized for first-paint time, ahead of read data preloading to speed up large content load, delayed writes to not block first paint time, pool of most recently used response headers to allow 0ms decisions on reuse or re-validation of a cached payload, 0ms miss-time look-up via an index, smarter eviction policies using frecency algorithm, resilience to crashes and zero main thread hangs or jank.  Also it eats less memory, but this may be subject to change based on my manual measurements with my favorite microSD card which shows that keeping at least data of html, css and js files critical for rendering in memory may be wise.  More research to come.

Thanks to everyone helping with this effort.  Namely Joel Maher and Avi Halachmi for helping to chase down Talos regressions and JW Wang for helping to find cause of one particular hard to analyze test failure spike.  And also all early adopters who helped to find and fix bugs.  Thanks!

 

New preferences to play with:

 

browser.cache.disk.metadata_memory_limit
Number of kBs we reserve for keeping recently loaded cache entries metadata (i.e. response headers etc.) for quick access and re-validation or reuse decisions.  By default this is at 250kB.
browser.cache.disk.preload_chunk_count
Number of data chunks we always preload ahead of read to speed up load of larger content like images.  Currently size of one chunk is 256kB, and by default we preload 4 chunks - i.e. 1MB of data in advance.

 

Load times compare:

Since these tests are pretty time consuming and usually not very precise, I was only testing with page 2 of my blog that links some 460 images.  Media storage devices available were: internal SSD, an SDHC card and a very slow microSD via a USB reader on a Windows 7 box.

 

[ complete page load time / first paint time ]

SSD
Cache versionFirst visitCold go to 1)Warm go to 2)Reload
cache v17.4s / 450ms880ms / 440ms510ms / 355ms5s / 430ms
cache v26.4s / 445ms610ms / 470ms470ms / 360ms5s / 440ms

 

Class 10 SDHC
Cache versionFirst visitCold go to 1)Warm go to 2)Reload
cache v17.4s / 635ms760ms / 480ms545ms / 365ms5s / 430ms
cache v26.4s / 485ms1.3s / 450ms530ms / 400ms*5.1s / 460ms*

 

Edit: I found one more place to optimize - preload of data sooner in case an entry has already been used during the browser session (bug 1013587).  We are winning around 100ms for both first paint and load times!  Also stddev of first-paint time is smaller (36) than before (80).  I have also measured more precisely the load time for non-patched cache v2 code.  It's actually better.

Slow microSD
Cache versionFirst visitCold go to 1)Warm go to 2)Reload
cache v113s / 1.4s1.1s / 540ms560ms / 440ms5.1s / 430ms
cache v26.4s / 450ms1.7s / 450ms710ms / 540ms*5.4s / 470ms*
cache v2 (with bug 1013587)--615ms / 455ms*-

* We are not keeping any data in memory (bug 975367 and 986179) what seems to be too restrictive.  Some data memory caching will be needed.

 

"Jankiness" compare:

The way I have measured browser UI jank (those hangs when everything is frozen) was very simple: summing every stuck of the browser UI, taking more then 100ms, between pressing enter and end of the page load.

 

[ time of all UI thread events running for more then 100ms each during the page load ]

SSD
Cache versionFirst visitCold go to 1)Warm go to 2)Reload
cache v10ms600ms0ms0ms
cache v20ms0ms0ms0ms

 

Class 10 SDHC
Cache versionFirst visitCold go to 1)Warm go to 2)Reload
cache v1600ms600ms0ms0ms
cache v20ms0ms0ms0ms

 

Slow microSD
Cache versionFirst visitCold go to 1)Warm go to 2)Reload
cache v12500ms740ms0ms0ms
cache v20ms0ms0ms0ms

 

All load time values are medians, jank values averages, from at least 3 runs without extremes in attempt to lower the noise.

 

1) Open a new tab and navigate to a page right after the Firefox start.

2) Open a new tab and navigate to a page that has already been visited during the browser session.

 

Posted on by Honza Bambas 58

NTLMv1 and Firefox

In Firefox 30 the internal fallback implementation of the NTLM authentication schema talking only NTLMv1 has been disabled by default for security reasons.

If you are experiencing problems with authentication to NTLM or Negotiate http proxies or http servers since Firefox 30 you may need to switch network.negotiate-auth.allow-insecure-ntlm-v1 in about:config to true.


Firefox (Necko) knows two ways to authenticate to a proxy or a server that requires NTLM or LM authentication:

  • System API or library like SSPI, GSSAPI or ntlm_auth binary ; on the Window platform by default the SSPI is always attempted, on non-Windows systems must be allowed by modifying some of the Firefox preferences, see bellow
  • Our own internal NTLM implementation module that is currently disabled since it talks only NTLMv1 ; we may have some plans to implement NTLMv2 in the future

Note that if you are in an environment where the system API can be used we have no handles to influence what NTLM version is used. It's fully up to your local system and network setting, Firefox has no control over it.

 

Preference list influencing NTLM authentication in Firefox

(Note: there is a similar list dedicated to the Negotiate schema)

EDIT: network.negotiate-auth.allow-insecure-ntlm-v1-https
Introduced in Forefox 31 on June 23th, 2014, enabled use of our own internal implementation of NTLM module that talks only NTLMv1 for connections to HTTPS servers (not proxies). Note: this preference influences both NTLM and Negotiate authentication scheme.
deafult: true
true: The NTLMv1 internal implementation module is enabled and used as a fallback for connecting secure HTTPS servers when system API authentication fails or cannot be used.
false: Usage of the NTLMv1 module is controlled by network.negotiate-auth.allow-insecure-ntlm-v1.
network.negotiate-auth.allow-insecure-ntlm-v1
Introduced in Forefox 30 on April 25th, 2014, disables use of our own internal implementation of NTLM module that talks only NTLMv1. Note: this preference influences both NTLM and Negotiate authentication scheme.
deafult: false
true: The NTLMv1 internal implementation module is enabled and used as a fallback when system API authentication fails or cannot be used.
false: Usage of the NTLMv1 module is hard-disabled, it won't be used under any circumstances.
network.automatic-ntlm-auth.allow-proxies
Allows use of the system (e.g. SSPI) authentication API when talking to (and only to) a proxy requiring NTLM authentication, this also allows sending user's default credentials - i.e. the credentials user is logged in to the system - to the proxy automatically without prompting the user.
deafult: true
true: The system API (like SSPI) will be used to talk to the proxy, default credentials will be sent to the proxy automatically.
false: Disallows send of default credentials to the proxy. On non-Windows platforms the fallback internal implementation, which is currently disabled, would be used. Hence with this setting you will not be able to authenticate to any NTLM proxy.
network.automatic-ntlm-auth.trusted-uris
This is a list of URLs or schemes that you trust to automatically send the system default credentials without any prompts to when NTLM authentication is required, the system API like SSPI will be used. On non-Windows platforms without filling this list you cannot use the system NTLM API to authenticate and since the internal NTLM v1 is disabled, you will not be able to authenticate at all.
deafult: an empty string
example: "https://, http://intranet.com/" - this will allow sending the default credentials to any https: host and any address that starts with http://intranet/ you are connecting to automatically without prompts - BE CAREFUL HERE.
network.automatic-ntlm-auth.allow-non-fqdn
Influences automatic send of default system credentials to hosts with a non fully qualified domain names (e.g. http://corporate/, http://intranet/).
deafult: false
true: Allow automatic sending without prompts, this setting is examined before network.automatic-ntlm-auth.trusted-uris check. With this setting there is no need to list your non-FQDN hosts in the network.automatic-ntlm-auth.trusted-uris preference string.
false: Automatic sending is not allowed to non-FQDN hosts, although particular hosts can be manually allowed in network.automatic-ntlm-auth.trusted-uris.
network.auth.force-generic-ntlm
Forces in all cases and on all platforms use of the internal NTLM implementation. This effectively bypasses use of the system API and never sends the default system credentials.
deafult: false
true: In all cases and on all platforms always only use the internal NTLM implementation. With network.negotiate-auth.allow-insecure-ntlm-v1 at false this will actually completely turn of any attempts to do NTLM authentication to any server or proxy.
false: When NTLM internal implementation is not disabled with network.negotiate-auth.allow-insecure-ntlm-v1 (default is disabled) it is only used when you are not on the Windows platform and the host being connected is neither a proxy nor an allowed non-FQDN host nor a listed trusted host.

 

Disclaimer: NO WARRANTY how accurate or complete this list is. I don't know the Kerberos preferences (if there are any) at all. I am not the original author of this code, I'm only occasionally maintaining it as part of my HTTP work.

Posted on by Honza Bambas 6

Brendan Eich - what the heck?

Mozilla has few hundred employes and even more volunteers and outside contributors.  Worldwide.  I don't know them all by person - and I never will.  I even don't know Brendan Eich in person.

When you look at this just purely statistically, it's clear that all these people creating Mozilla are of different religion, opinions to global warming, occupation of Tibet, human rights in Russia, China, global economical system, sexual orientation, or just of different favorite color.  And no one says "you don't like pink color?  Then I can't work with you!"

This is the same with any larger company and actually any group of people that do something as a team.

We all in Mozilla work together and do it as good as we can and create something that I think has an impact on the world and most of you out there like it.  For instance, one of my managers is a Catholic - strongly believing.  I am an atheist - also very strongly believing.  Do you think it matters?  No!  What only I'm interested in is how anyone I cooperate with does its job - that's the only thing that matters - at least to me.

I don't agree with Brendan Eich's opinion on forbidding gay marriage but I very much respect him as a technical mind.  Period.

 

(Closed for comments, if you want to say something more on this topic, please say it somewhere else)

Interesting articles:
Reflex - Politická korektnost zabila šéfa americké Mozilly, vyjádřit názor se stává smrtící (in Czech)
FAQ on CEO Resignation (on The Mozilla Blog)

Posted on by Honza Bambas 5

Get NSPR log from tryserver run

Mozilla tryserver nspr log

 

Get an NSPR log with your choice of modules from a try run.  So far works only for mochitests but I think can be extended for other test harness as well.

Based on an advice from Phil Ringnalda at this bug comment. The feature has just landed on mozilla-central.

  • Open file testing/mochitest/runtests.py
  • Search for NSPR_LOG_MODULES = "", should look as:
    # Set the desired log modules you want an NSPR log be produced by a try run for, or leave blank to disable the feature.
# This will be passed to NSPR_LOG_MODULES environment variable. Try run will then put a download link for the log file
# on tbpl.mozilla.org.

NSPR_LOG_MODULES = ""
  • Change the empty string "" to a string with list of all your modules and levels to produce the log for
  • Push this modification to try along with your patch(es)
  • All produced NSPR logs in a single zip file are then uploaded for each completed run to amazonaws, you will find the link in the results window at the bottom of tbpl.mozilla.org


Note: doesn't work for B2G so far, there is no way to upload the logs.

Posted on by Honza Bambas 0

Building mozilla code directly from Visual Studio IDE

visual-studio-mozilla-build-ide-error-list

 

Yes, it's possible!  With a single key press you can build and have a nice list of errors in the Error List window, clickable to get to the bad source code location easily.  It was a fight, but here it is.  Tested with Visual Studio Express 2013 for Windows Desktop, but I believe this all can be adapted to any version of the IDE.

 

  • Create a shell script, you will (have to) use it every time to start Visual Studio from mozilla-build's bash prompt:

export MOZ__INCLUDE=$INCLUDE
export MOZ__LIB=$LIB
export MOZ__LIBPATH=$LIBPATH
export MOZ__PATH=$PATH
export MOZ__VSINSTALLDIR=$VSINSTALLDIR
# This is for standard installation of Visual Studio 2013 Desktop, alter the paths to your desired/installed IDE version
cd "/c/Program Files (x86)/Microsoft Visual Studio 12.0/Common7/IDE/"
./WDExpress.exe &

  • Create a solution 'mozilla-central' located at the parent directory where your mozilla-central repository clone resides.  Say you have a structure like C:\Mozilla\mozilla-central, which is the root source folder where you find .hg, configure.in and all the modules' sub-dirs.  Then C:\Mozilla\ is the parent directory.
  • In that solution, create a Makefile project 'mozilla-central', again located at the parent directory.  It will, a bit unexpectedly, be created where you probably want it - in C:\Mozilla\mozilla-central.
  • Let the Build Command Line for this project be (use the multi-line editor to copy & paste: combo-like arrow on the right, then the <Edit...> command):

call "$(MOZ__VSINSTALLDIR)\VC\bin\vcvars32.bat"
set INCLUDE=$(MOZ__INCLUDE)
set LIB=$(MOZ__LIB)
set LIBPATH=$(MOZ__LIBPATH)
set PATH=$(MOZ__PATH)
set MOZCONFIG=c:\optional\path\to\your\custom\mozconfig
cd $(SolutionDir)
python mach --log-no-times build binaries


 

Now when you make a modification to a C/C++ file just build the 'mozilla-central' project to run the great build binaries mach feature and quickly build the changes right from the IDE.  Compilation and link errors as well as warnings will be nicely caught in the Error List.

BE AWARE: There is one problem - when there is a typo/mistake in an exported header file, it's opened as a new file in the IDE from _obj/dist/include location.  When you miss that and modify that file it will overwrite on next build! Using CreateHardLink might deal with this issue.

With these scripts you can use the Visual Studio 2013 IDE but build with any other version of VC++ of your choice.  It's independent, just run the start-up script from different VS configuration mozilla-build prompt.

I personally also create projects for modules (like /netwerk, /docshell, /dom) I often use.  Just create a Makefile project located at the source root directory with name of the module directory.  The project file will then be located in the module - I know, not really what one would expect.  Switch Solution Explorer for that project to show all files, include them all in the project, and you are done.

Few other tweaks:

  • Assuming you properly use an object dir, change the Output Directory and Intermediate Directory to point e.g. to $(SolutionDir)\<your obj dir>\$(Configuration)\.  The logging and other crap won't then be created in your source repository.
  • Add:
    ^.*\.vcproj.*
    ^.*\.vcxproj.*
    .sln$
    .suo$
    .ncb$
    .sdf$
    .opensdf$
    to your custom hg ingnore file to prevent the Visual Studio project and solution files interfere with Mercurial.  Same suggested for git, if you prefer it.

 

Note: you cannot use this for a clobbered build because of an undisclosed Python Windows-specific bug.  See here why.  Do clobbered builds from a console, or you may experiment with clobber + configure from a console and then build from the IDE.

Posted on by Honza Bambas 9