Recently a user encountered the following error message when refreshing a query in Excel that used PowerQuery to connect to Microsoft SQL Server using Windows authentication:
Could not load file or assembly ‘System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. Either a required impersonation level was not provided, or the provided impersonation level is invalid. (Exception from HRESULT: 0x80070542)
What’s going on? Read on for all the gory details.
Spoiler: The problem was caused by attempting to “Use alternate credentials” for Windows authentication in PowerQuery. It was solved by switching to “Use my current credentials” in Data Source Settings, as described in “Manage data source credentials” in Manage data source settings and permissions (Power Query).
The first part of the error message “Could not load file or assembly
‘System.EnterpriseServices, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies.” can be a bit
misleading. It indicates that the System.EnterpriseServices .NET assembly
failed to load. Often this is caused by missing or corrupted DLL files (in
this case, System.EnterpriseServices.dll
or any of its dependencies). I
found many well-intentioned suggestions to run System File
Checker
or reinstall .NET framework to address the failure. However, in this case the
error is not caused by missing or corrupted DLLs.
The rest of the error message indicates what caused the assembly load failure.
We can use the Microsoft Error Lookup
Tool
to decode the
HRESULT
and determine that it is 1346 (0x542)
ERROR_BAD_IMPERSONATION_LEVEL
,
which has a description that matches the preceding error message. Although
this doesn’t tell us much that we didn’t already know, it fills in the details
a bit.
Since the error message refers to impersonation, I suspected it had something to do with database authentication. I used the following tests, which you may find useful:
I was able to connect using
sqlcmd
, SQL Server
Management
Studio,
and PowerShell (by running $c =
New-Object System.Data.Odbc.OdbcConnection "DRIVER=SQL
Server;SERVER=$myServerName;Trusted_Connection=Yes;DATABASE=$myDbName";
$c.Open()
) which confirmed that the issue was specific to Excel.
I tried creating a new workbook and querying data from the same database, using the same connection type (PowerQuery in my case). It failed with the same error message, confirming that the issue is not specific to a particular file.
If you are able to connect from other programs using Windows authentication as the current user, the issue may be caused the data source credentials for the SQL Server database connection in PowerQuery being configured to “Use alternate credentials” for Windows authentication rather than “Use my current credentials”. This can be changed in the Data Source Settings, as described in “Manage data source credentials” in Manage data source settings and permissions (Power Query) and reproduced below:
If the SQL Server connection does require using alternate credentials, the issue may be that User Rights Assignment does not allow Impersonate a client after authentication in the Local Security Policy. See Configure security policy settings for how to change this setting, as suggested by How do I make sure that my windows impersonation level is valid? on Stack Overflow.
I recently configured a Windows 11 guest virtual machine on libvirt with the VirtIO drivers. This post is a collection of my notes for how to configure the host and guest. Most are applicable to any recent version of Windows.
For the impatient, just use my libvirt domain XML.
Many configuration guides recommend disabling hyper-threading on Intel chipsets before Sandy Bridge for performance reasons. Additionally, if the VM may run untrusted code, it is recommended to disable SMT on processors vulnerable to Microarchitectural Data Sampling (MDS).
To keep RTC time in the guest
accurate across suspend/resume, it is advisable to set SYNC_TIME=1
in
/etc/default/libvirt-guests
, which calls virsh domtime
--sync
after the guest
is resumed. This causes the QEMU Guest
Agent to call w32tm /resync
/nowait
in the guest, which synchronizes the clock with the configured w32time
provider (usually NTP, although VMICTimeProvider could be used to sync with
the Hyper-V
host).
Ignore the comment in older libvirt versions that SYNC_TIME is not supported
on Windows, which was fixed in
qemu/qemu@105fad6bb22.
To send keyboard shortcuts (i.e. key combinations) to the virtual machine viewer that has focus, rather than sending them to the Wayland compositor, the compositor must support the Wayland keyboard shortcut inhibition protocol. For example, Sway gained support for for this protocol in Sway 1.5 (swaywm/sway#5021). When using Sway 1.4 or earlier in the default configuration, pressing Win + d would invoke dmenu rather than display or hide the desktop in the focused Windows VM.
There are trade-offs to consider when choosing between BIOS and UEFI:
If UEFI is selected, an image must be chosen for the pflash device firmware. I
recommend OVMF_CODE_4M.ms.fd
(which pairs with OVMF_VARS_4M.ms.fd
which
enables Secure Boot and includes Microsoft keys in KEK/DB) or
OVMF_CODE_4M.fd
if Secure Boot is not desired. See
ovmf.README.Debian
for details.
Note: Be aware that UEFI does not support the QEMU -boot order=
option.
It does support the bootindex
properties. For
example, to boot from win10.iso
, use -drive
id=drive0,file=win10.iso,format=raw,if=none,media=cdrom,readonly=on -device
ide-cd,drive=drive0,bootindex=1
instead of -cdrom win10.iso -boot order=d
.
It may be preferable to choose a CPU model which satisfies the Windows Processor Requirements for the Windows edition which will be installed on the guest. As of this writing, the choices are Skylake, Cascadelake, Icelake, Snowridge, Cooperlake, and EPYC.
If the VM may be migrated to a different machine, consider setting
check='full'
on <cpu/>
so enforce
will be added to the QEMU -cpu
option and the domain will not start if the created vCPU doesn’t match the
requested configuration. This is not currently set by default. (Bug
822148)
If topology is not specified, libvirt instructs QEMU to add a socket for each
vCPU (e.g. <vcpu placement="static">4</vcpu>
results in -smp
4,sockets=4,cores=1,threads=1
). It may be preferable to change this for
several reasons:
First, as Jared Epp pointed out to me via email, for licensing reasons Windows 10 Home and Pro are limited to 2 CPUs (sockets), while Pro for Workstations and Enterprise are limited to 4 (possibly requiring build 1903 or later to use more than 2). Similarly, Windows 11 Home is limited to 1 CPU while 11 Pro is limited to 2. Therefore, limiting sockets to 1 or two on these systems is strongly recommended.
Additionally, it may be useful, particularly on a NUMA system, to specify a topology matching (a subset of) the host and pin vCPUs to the matching elements (e.g. virtual cores on physical cores). See KVM Windows 10 Guest - CPU Pinning Recommended? on Reddit and PCI passthrough via OVMF: CPU pinning on ArchWiki Be aware that, on my single-socket i5-3320M system, the matching configurations I tried performed worse than the default. Some expertise is likely required to get this right.
It may be possible to reduce jitter by pinning vCPUs to host cores, emulator
and iothreads to other host cores and using a hook script with cset shield
to ensure host processes don’t run on the vCPU cores. See Performance of
your gaming VM.
Note that it is possible to set max CPUs in excess of current CPUs for CPU hotplug. See Linux KVM – How to add /Remove vCPU to Guest on fly ? Part 9.
QEMU supports several Hyper-V
Enlightenments for
Windows guests. virt-manager/virt-install enables some Hyper-V Enlightenments
by default, but is missing several useful recent additions
(virt-manager/virt-manager#154).
I recommend editing the libvirt domain XML to enable Hyper-V
enlightenments
which are not described as “nested specific”. In particular, hv_stimer
,
which reduces CPU usage when the guest is
paused.
When configuring the memory size, be aware of the system requirements (4GB for Windows 11, 1GB for 32-bit, 2GB for 64-bit Windows 10) and Memory Limits for Windows and Windows Server Releases which vary by edition.
If shared memory will be used (e.g. for virtio-fs discussed below), define a (virtual) NUMA zone and memory backing. The memory backing can be backed by files (which are flexible, but can have performance issues if not on hugetlbfs/tmpfs) or memfd (since QEMU 4.0, libvirt 4.10.0). The memory can be Huge Pages (which have lower overhead, but can’t be swapped) or regular pages. (Note: If hugepages are not configured, Transparent Hugepages may still be used, if THP is enabled system-wide on the host system. This may be advantageous, since it reduces translation overhead for merged pages while still allowing swapping. Alternatively, it may be disadvantageous due to increased CPU use for defrag/compact/reclaim operations.)
If memory ballooning will be used, set current memory to the initial amount
and max memory to the upper limit. Be aware that the balloon size is not
automatically managed by KVM. There was an Automatic
Ballooning project
which has not been merged. Unless a separate tool, such as oVirt Memory
Overcommitment Manager, is
used, the balloon size must be changed manually (e.g. using virsh --hmp
"balloon $size"
) for the guest to
use more than “current memory”. Also be aware that when the balloon is
inflated, the guest shows the memory as “in
use”
which may be counter-intuitive.
The Q35 Machine Type adds support for
PCI-E, AHCI, PCI hotplug, and probably many other features, while removing
legacy features such as the ISA bus. Historically it may have been preferable
to use i440FX for stability and bug avoidance, but my experience is that it’s
generally preferable to use the latest Q35 version (e.g. pc-q35-6.1
for QEMU
6.1).
Paravirtualized storage can be implemented using either SCSI with
virtio-scsi
and the vioscsi
driver or bulk storage with virtio-blk
with
the viostor
driver. The choice is not obvious. In general, virtio-blk
may be faster while virtio-scsi
supports more
features (e.g. pass-through,
multiple LUNs, CD-ROMs, more than 28 disks). Citations:
virtio-blk
is faster than virtio-scsi
in Fam Zheng’s LC3-2018
presentation.virtio-scsi
“rough numbers: 6% slower [than virtio-blk
] on iozone
with a tmpfs-backed disk”.vioscsi
has supported discard for long time (pre 2015, when changelog
starts?). viostor
only added support for discard recently (in
virtio-win/kvm-guest-drivers-windows#399
for 0.1.172-1). Although #399 is described as “preliminary support” the
author clarified that it is now full support on par with
vioscsi
.When choosing a format for the virtual disk, note that qcow2
supports
snapshots. raw
does not. However, raw
is likely to have better
performance due to less overhead.
Alberto Garcia added support for Subcluster allocation for qcow2
images
in QEMU 5.2. When using 5.2 or later, it may be prudent to create qcow2
disk images with extended_l2=on,cluster_size=128k
to reduce wasted space and
write amplification. Note that extended L2 always uses 32 sub-clusters, so
cluster_size
should be 32 times the filesystem cluster size (4k for NTFS
created by the Windows installer).
I find it generally preferable to set discard
to unmap
so that guest
discard/trim requests are passed through to the disk image on the host
filesystem, reducing its size. For Windows guests, discard/trim requests are
normally only issued when Defragment and Optimize
Drives
is run. It is scheduled to run weekly by default.
I do not recommend enabling detect_zeroes
to detect write requests with all
zero bytes and optionally unmap the zeroed areas in the disk image. As the
libvirt docs note:
“enabling the detection is a compute intensive operation, but can save file
space and/or time on slow media”.
Jared Epp also informed me of an incompatibility between the virtio drivers
and defrag
in Windows 10 and 11
(virtio-win/kvm-guest-drivers-windows#666)
which causes defragment and optimize to take a long time and write a lot of
data. There are two workarounds suggested:
Use a recent version of the virtio-win drivers (0.1.225-1 or later?) which
includes
virtio-win/kvm-guest-drivers-windows#824
and set discard_granularity
to a large
value
(Hyper-V uses 32M).
For libvirt, discard_granularity
can be set using
<qemu:property>
on libvirt 8.2 and later, or
<qemu:arg>
on earlier versions, as demonstrated by Pau
Rodriguez-Estivill.
Note: There was a patch to add discard_granularity
to
<blockio>
but it was never merged, as far as I can tell.
Emulate an SSD rather than a Thin Volume, as suggested by Pau
Rodriguez-Estivill
by setting rotation_rate=1
(for SSD
detection)
and discard_granularity=0
(to change the MODE PAGE POLICY to
“Obsolete”?).
These settings were inferred from QEMU
behavior.
It’s not clear to me why this avoids the slowness issue.
For libvirt, rotation_rate
can be set on <target>
of
<disk>
.
As above, discard_granularity
can be set using
<qemu:property>
on libvirt 8.2 and later, or
<qemu:arg>
on earlier versions.
I am unsure which approach to recommend, although I am currently using
discard_granularity=32M
. Stewart Bright noted some differences between SSD
and Thin Provisioning behavior in Windows
guests.
In particular, I’m curious how slabs and slab consolidation behave. Interested
readers are encouraged to investigate further and report their findings.
There are several options for graphics
cards. VGA and
other display devices in qemu by Gerd
Hoffmann has
practical descriptions and recommendations (kraxel’s
news is great for following progress).
virtio-drivers 0.1.208 and later include the viogpudo
driver for
virtio-vga
. (Bug 1861229)
Unfortunately, it has some limitations:
height x width <=
4x1024x1024
.viogpuap.exe
and vgpusrv.exe
to a permanent location.vgpusrv.exe -i
as Administrator to register the “VioGpu Resolution Service” Windows Service.However, unless the above limitations are critical for a particular use case,
I would recommend virtio-vga
over QXL based on the understanding that it is
a better and more promising approach on technical grounds and that it is where
most current development effort is directed.
Warning: When using BIOS firmware, the video device should be connected to
the PCI Express Root Complex (i.e. <address type='pci'
bus='0x00'>
) in order
to access the VESA BIOS Extensions (VBE)
registers.
Without VBE modes, the Windows installer is limited to grayscale at
640x480,
which is not pleasant.
Note that QEMU and libvirt connect video devices to the Root Complex by default, so no additional configuration is required. However, if a second video device is added using virt-manager or virt-xml, it is connected to a Root Port or PCIe-to-PCI bridge, which creates problems if the first device is removed (virt-manager/virt-manager#402).
Note: If 3D acceleration is enabled for virtio-vga
, the VM must have a
Spice display device with OpenGL enabled to avoid an “opengl is not available”
error when the VM is started. Since the viogpudo
driver does not support 3D
acceleration, I recommend disabling both.
I recommend adding a “Virtio Keyboard” and “Virtio Tablet” device in addition to the default USB or PS/2 Keyboard and Mouse devices. These are “basically sending linux evdev events over virtio”, which can be useful for keyboard or mouse with special features (e.g. keys/buttons not supported by PS/2). Possibly also a latency or performance advantage. Note that it is not necessary to remove the USB or PS/2 devices, since QEMU will route input events to virtio-input devices if they have been initialized by the guest and virtio input devices are not supported without drivers, which can make setup and recovery more difficult if the PS/2 devices are not present.
Windows 11 requires TPM 2.0. Therefore, I recommend adding a QEMU TPM Device to provide one. Either TIS or CRB can be used. “TPM CRB interface is a simpler interface than the TPM TIS and is only available for TPM 2.” If emulated, swtpm must be installed and configured on the host. Note: swtpm was packaged for Debian in 2022 (Bug 941199), so it is not available in Debian 11 (Bullseye) or earlier releases.
It may be useful to add a
virtio-rng
device to provide
entropy to the guest. This is particularly true if the vCPU does not support
the RDRAND
instruction or if it is
not trusted.
There are several options for sharing files between the host and guest with various trade-offs. Some common options are discussed below. My recommendation is to use SMB/CIFS unless you need the feature or performance offered by virtio-fs (and like living on the bleeding edge).
Libvirt supports sharing virtual filesystems using a protocol similar to FUSE over virtio. It is a great option if the host and guest can support it (QEMU 5.0, libvirt 6.2, Linux 5.4, Windows virtio-drivers 0.1.187). It has very high performance and supports many of the filesystem features and behaviors of a local filesystem. Unfortunately, it has several significant issues including configuration difficulty, lack of support for migration or snapshot, and Windows driver issues, each explained below:
Virtio-fs requires shared memory between the host and guest, which in turn requires configuring a (virtual) NUMA topology with shared memory backing: See Sharing files with Virtio-FS. Also ensure you are using a version of libvirt which includes the apparmor policy patch to allow libvirtd to call virtiofsd (6.7.0 or later).
Migration with virtiofs device is not supported by libvirt, which also prevents saving and creating snapshots while the VM is running. This is difficult to work around since live detach of device ‘filesystem’ is not supported by libvirt for QEMU.
The Windows driver has released with several severe known bugs, such as:
iommu_platform=on
My offer to assist with adding tests (virtio-win/kvm-guest-drivers-windows#531 has seen very little interest or action. It’s not clear to me who’s working on virtio-fs and how much interest it has at the moment.
Although it is not an option for Windows guests due to lack of a driver (virtio-win/kvm-guest-drivers-windows#126), it’s worth nothing that virtio-9p is similar to virtio-fs except that it uses the 9P distributed file system protocol which is supported by older versions of Linux and QEMU and has the advantage of being used and supported outside of virtualization contexts. For a comparison of virtio-fs and virtio-9p, see the virtio-fs patchset on LKML.
SPICE Folder Sharing is a relatively easy way to share directories from the
host to the guest using the WebDAV protocol
over the org.spice-space.webdav.0
virtio channel. Many libvirt viewers
(remote-viewer, virt-viewer, Gnome Boxes) provide built-in support. Although
virt-manager does not
(virt-manager/virt-manager#156),
it can be used to configure folder
sharing
(by adding a org.spice-space.webdav.0
channel) and other viewers used for
running the VM and serving files. Note that users have reported performance
is not great and the SPICE WebDAV
Daemon must be installed in the guest to share files.
Since Windows supports SMB/CIFS (aka “Windows File Sharing Protocol”) natively, it is relatively easy to share files between the host and guest if networking is configured on the guest. Either the host (with Samba or KSMBD) or the guest can act as the server. For a Linux server, see Setting up Samba as a Standalone Server. For Windows, see File sharing over a network in Windows 10. Be aware that, depending on the network topology, file shares may be exposed to other hosts on the network. Be sure to adjust the server configuration and add firewall rules as appropriate.
I recommend adding the following Channel Devices:
There are some differences between the “legacy” 0.9/0.95 version of the virtio
protocol and the “modern” 1.0 version. Recent versions (post-2016) of QEMU
and libvirt use 1.0 by default. For older versions, it may be necessary to
specify disable-legacy=on,disable-modern=off
to force the modern version.
For details and steps to confirm which version is being used, see Virtio 1.0
and Windows
Guests.
I recommend configuring the guest with two SATA CD-ROM devices during installation: One for the Windows 10 ISO or Windows 11 ISO, and one for the virtio-win ISO. At the “Where would you like to install Windows?” screen, click “Load Driver” then select the appropriate driver as described in How to Install virtio Drivers on KVM-QEMU Windows Virtual Machines.
If the guest does not satisfy the Windows 11 System Requirements, you can bypass the checks by:
regedit
.HKEY_LOCAL_MACHINE\SYSTEM\Setup\LabConfig
with one or more of
the following DWORD values:
BypassRAMCheck
set to 1 to skip memory size checks.BypassSecureBootCheck
set to 1 to skip SecureBoot checks.BypassTPMCheck
set to 1 to skip TPM 2.0 checks.regedit
.Be aware that Windows 11 is not supported in this scenario and doing so may prevent some features from working.
Drivers for VirtIO devices can be installed by running the
virtio-win-drivers-installer,
virtio-win-gt-x64.msi
(Source),
available on the virtio-win ISO) or by using Device Manager to search for
device drivers on the virtio-win ISO.
The memory ballooning service is installed by virtio-win-drivers-installer. To install it manually (for troubleshooting or other purposes):
blnsrv.exe
from virtio-win.iso to somewhere permanent (since install
command defines service using current location of exe).blnsrv.exe -i
as AdministratorNote that the virtio-win-drivers-installer does not currently support Windows 11/Server 2022 (Bug 1995479). However, it appears to work correctly for me. It also does not support Windows 7 and earlier (#9). For these systems, the drivers must be installed manually.
To use virtio-fs for file sharing, in addition to installing the viofs
driver, complete the following steps (based on a comment by
@FailSpy](https://github.com/virtio-win/kvm-guest-drivers-windows/issues/126#issuecomment-667432487)):
winfsp-x64.dll
from C:\Program Files (x86)\WinFSP\bin
to
C:\Program Files\Virtio-Win\VioFS
.VirtioFSService
created by virtio-win-drivers-installer is
stopped and has Startup Type: Manual or Disabled. (Enabling this service
would work, but would make shared files only accessible to elevated
processes.virtiofs.exe
at logon using the following
PowerShell:
$action = New-ScheduledTaskAction -Execute 'C:\Program Files\Virtio-Win\VioFS\virtiofs.exe'
$trigger = New-ScheduledTaskTrigger -AtLogon
$principal = New-ScheduledTaskPrincipal 'NT AUTHORITY\SYSTEM'
$settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -ExecutionTimeLimit 0
$task = New-ScheduledTask -Action $action -Principal $principal -Trigger $trigger -Settings $settings
Register-ScheduledTask Virtio-FS -InputObject $task
The QEMU Guest Agent can be
used to coordinate snapshot, suspend, and shutdown operations with the
guest,
including post-resume RTC synchronization. Install it
by running
qemu-ga-x86_64.msi
(available in the guest-agent
directory of the virtio-win ISO).
If the virtual machine is configured with QXL graphics instead of
virtio-vga
, as discussed in the Video section, a QXL driver should
be installed. For Windows 8 and later, install the QXL-WDDM-DOD
driver
(Source). On
Windows 7 and earlier, the QXL
driver
(Source) can be used. The
driver can be installed from the linked MSI, or from the qxldod
/qxl
directory of the virtio-win ISO.
For clipboard sharing and display size changes, install the SPICE Agent (Source).
Note: Some users have reported problems on Windows 11 (spice/win32#11). However, it has been working without issue for me.
To use SPICE folder sharing, install the SPICE WebDAV daemon (Source).
Instead of installing the drivers/agents separately, you may prefer to install the SPICE Guest Tools (Source) which bundles the virtio-win Drivers, QXL Driver, and SPICE Agent into a single installer.
Warning: It does not include the QEMU Guest Agent and is several years out of date at the time of this writing (last updated on 2018-01-04 as of 2021-12-05).
Another alternative to installing drivers/agents separately is to install the
QEMU Guest
Tools
(Source)
which bundles the virtio-win Drivers, QXL
Driver, SPICE Agent, and QEMU Guest
Agent into a single installer.
virtio-win-guest-tools.exe
is available in the virtio-win ISO.
Once Windows is installed, one or both CD-ROM drives can be removed. If both are removed, the SATA Controller may also be removed.
For a low-overhead CD-ROM drive, a virtio-scsi
drive can be added by adding
a VirtIO SCSI controller (if one is not already present) then a CD-ROM on the
SCSI bus.
If discard was enabled for the virtual disk, Defragment and
Optimize
Drives
in the Windows guest should show the drive with media type “Thin provisioned
drive”
(or “SSD”, if configured with rotation_rate=1
). It may be useful to
configure a disk optimization schedule to trim/discard unused space in the
disk image.
discard_granularity=32M
workaround based on updates in
virtio-win/kvm-guest-drivers-windows#666.
Move it to a new subsection of Discard.-boot order=
, bootindex, and UEFI.virtio-vga
with the viogpudo
driver instead of QXL with the
qxldod
or qxl
driver.Suppose you are using Microsoft Exchange Online with Azure AD Connect Sync to synchronize users between an on-premises Active Directory and Azure Active Directory. Further suppose that there are some users for whom you do not want to create an Exchange Online mailbox, but would like to forward email to an external address. This might occur for part-time employees, contractors, partners, or other users for whom it is convenient to have a company email address, but a mailbox to hold the email is not required or desired. How would you accomplish this?
The best solution I have come up, based on the Microsoft Answers question Convert on-premise AD users to MailUsers (Exchange online) and other similar discussions, is to create a Mail User for each AD user. Unfortunately, the Mail User can not be created using the steps in the Mail User documentation because the user already exists, causing a conflict:
error
The proxy address “SMTP:user@example.com” is already being used by the proxy addresses or LegacyExchangeDN. Please choose another proxy address.
Instead, the user must be mail-enabled in the on-premises AD, then
synchronized to Azure AD. If Exchange is installed on the server, this can be
accomplished with
Enable-MailUser
:
Enable-MailUser -Identity UserName -ExternalEmailAddress user@otherdomain.example
If Exchange is not installed, the same effect can be accomplished by setting the necessary AD user object properties:
mail
proxyAddresses
mailNickName
targetAddress
These can be set by any LDAP or AD property editor, such as ADSI Edit or
Set-ADUser
:
Set-ADUser -Identity UserName -Replace @{mail='username@company.example';mailNickName='username';proxyAddresses=@('SMTP:username@company.example','SMTP:username@companyalt.example');targetAddress='SMTP:user@otherdomain.example'}
Unfortunately, if Exchange has not been installed, these properties may not exist in the AD Schema, which would cause errors such as the following:
Set-ADUser : The specified directory service attribute or value does not exist Parameter name: mailNickName
Be aware that “when directory synchronization is enabled for a tenant and a user is synchronized from on-premises, most of the attributes cannot be managed from Exchange Online and must be managed from on-premises”. To facilitate this, Microsoft provides the Hybrid Configuration Wizard to license a locally-installed Exchange server by validating your O365 tenant. (Note that Exchange doesn’t need to be fully configured to manage mailboxes.) Alternatively, it is possible to use the Exchange installer to extend the AD Schema without installing Exchange:
setup.exe /prepareschema /iacceptexchangeserverlicenseterms
After /prepareschema
completes, it should be possible to set the properties
described above on user objects.
Once the properties have been set, simply wait for AD Connect Sync to synchronize the properties to Azure or start a sync cycle to do so immediately. Once synchronized, the mail-enabled users should appear as Mail Users in the Exchange Admin Center.
I was surprised to find that the version of Vim which ships with Git for Windows does not load my vimfiles/vimrc. This post has the explanation and an easy workaround.
By default, Vim loads personal initializations from $HOME/_vimrc
,
$HOME/vimfiles/vimrc
, or
$VIM/_vimrc
whichever is found
first. The version of Vim which ships with Git for Windows instead loads
personal initializations from $HOME/.vimrc
or $HOME/.vim/vimrc
.
Presumably this was done to match Vim’s default behavior on Unix (see
git-for-windows/git#658
(comment)).
The difference can be observed by comparing
:version
from Vim installed
system-wide:
user vimrc file: "$HOME\_vimrc"
2nd user vimrc file: "$HOME\vimfiles\vimrc"
3rd user vimrc file: "$VIM\_vimrc"
to :version
from Vim in Git Bash:
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
One way to use the same vimfiles
in both versions of Vim (with minimal
confusion or clutter) is to create a $HOME/.vimrc
file with the following
content:
" Git for Windows Vim user initialization file
" GFW uses ~/.vimrc and ~/.vim/vimrc instead of ~/_vimrc and ~/vimfiles/vimrc
" See https://github.com/git-for-windows/git/issues/658#issuecomment-184269470
" This file configures GFW Vim to behave like Windows Vim
let &runtimepath = '~/vimfiles,'
\ . join(filter(split(&runtimepath, ','), 'v:val !~? "/\\.vim"'), ',')
\ . ',~/vimfiles/after'
let &packpath = &runtimepath
source ~/vimfiles/vimrc
This replaces ~/.vim
with ~/vimfiles
in
'runtimepath'
and
'packpath'
, then
loads ~/vimfiles/vimrc
. Once this file is saved, both versions of Vim will
use the same paths for user configuration and runtime files.
Michael Dayah of Ptable.com asked about how to extend the technique from Serving XHTML with Apache MultiViews and Serving Pre-Compressed Files with Apache MultiViews to serve files for a language requested using a query parameter. This post outlines the slick technique we worked out.
Initially we investigated using
mod_rewrite and
mod_headers in
various ways. Unfortunately, it appears that
mod_negotiation
(which implements MultiViews
) performs negotiation in the type_checker
hook of the preparation phase of request
processing
(see
mod_negotiation.c:3212)
which is before the fixup
hook used by mod_headers (see
mod_headers.c:1007)
and mod_rewrite (see
mod_rewrite.c:5315).
This prevents header or environment changes made by those modules from
affecting MultiViews negotiation, unless other trickery (e.g. sub-requests) is
used.
The solution we came up with is to use
SetEnvIfExpr
from
mod_setenvif to
set the prefer-language
environment
variable.
For example, to use the language from a query parameter named lang
if it
contains a value which might be a valid Basic Language
Range for
Accept-Language:
Options +MultiViews
SetEnvIfExpr "%{QUERY_STRING} =~ /(?:^|&)lang=([A-Za-z]{1,8}(?:-[A-Za-z0-9]{1,8})*)(?:&|$)/" prefer-language=$1
This combination should be sufficient for requests such as GET
/index.html?lang=fr
to return index.html.fr
regardless of the value in the
Accept-Language
request header, if present.
Note: The mapping from language tags to file extensions is configured by
AddLanguage
.
On Debian, the default mapping is defined in
/etc/apache2/mods-available/mime.conf
(linked from
/etc/apache2/mods-enabled/mime.conf
when mod_mime is enabled) and can be
changed using RemoveLanguage
/AddLanguage
as desired.
Best of luck!
Recently I started using the Sway window manager, with
occasional fallback to XFCE. Having both
mako and
xfce4-notifyd installed causes a
conflict over the org.freedesktop.Notifications
D-Bus service name (see Red
Hat Bug 484945). This post describes the
workaround I am currently using, until dynamic activation
directories or another
solution is implemented.
The Desktop Notifications
Specification
defines a protocol that applications can use to generate popups to notify the
user of events. It requires a notification server to implement the
org.freedesktop.Notifications
service on the D-Bus session bus. The D-Bus
Specification
defines service activation based on information in service description files.
This mechanism is used by both xfce4-notifyd and mako use to start on demand.
However, this poses a problem. From Message Bus Starting Services
(Activation):
On the well-known session bus, if two .service files in the same directory offer the same service name, the result is undefined. Distributors should avoid this situation, for instance by naming session services’ .service files according to their service name.
On my system, the result is that xfce4-notifyd is always started, which provides a poor experience on Sway.
If all notification services are daemons (i.e. they only exit at the end of
the user session), an easy workaround is to avoid D-Bus Activation altogether
by starting whichever daemon is desired at the beginning of the user session.
This can be done from systemd units, Desktop
Autostart, or
traditional startup scripts (e.g. .xsessionrc
). My understanding is that
both GNOME and KDE notification services have moved to this approach.
Unfortunately, this approach forfeits the benefits of D-Bus activation, such as deferring the costs of running the notification service until/unless required and avoiding delay and contention at the start of the user session. It also doesn’t work for notification services like mako which exit after displaying a notification.
The next paragraph in the D-Bus Specification hints at a potential workaround:
If two .service files in different directories offer the same service name, the one in the higher-priority directory is used: for instance, on the system bus, .service files in /usr/local/share/dbus-1/system-services take precedence over those in /usr/share/dbus-1/system-services.
The service directories are defined by the dbus-daemon
configuration, as
described in
dbus-daemon(1), with
the standard session directories:
$XDG_RUNTIME_DIR/dbus-1/services
$XDG_DATA_HOME/dbus-1/services
(default ~/.local/share
)$XDG_DATA_DIRS/dbus-1/services
for each directory in $XDG_DATA_DIRS
(default /usr/local/share:/usr/share
)${datadir}/dbus-1/services
compiled-in default (default /usr/share
)This behavior can be used to implement a workaround by placing a service
definition file in $XDG_DATA_HOME/dbus-1/services
for a service which
activates the appropriate notification service for the current desktop
environment.
The D-Bus Specification also defines systemd
Activation
for starting a service via systemd instead of executing a binary. Although
the specification does not describe the details, systemd activation in
bus/activation.c
is reasonably easy to follow: Send an ActivationRequest
signal to
org.freedesktop.systemd1.Activator
at path /org/freedesktop/DBus
on the
session bus with a string argument containing the service name to activate.
To implement the workaround, create a service definition file named
~/.local/share/dbus-1/services/org.freedesktop.Notifications.service
with
the following content:
[D-BUS Service]
Name=org.freedesktop.Notifications
Exec=/home/username/.local/lib/notify-dispatch
Create an executable script named ~/.local/lib/notify-dispatch
which runs
the preferred notification service based on the environment:
#!/bin/sh
if [ "$XDG_SESSION_DESKTOP" = sway ]; then
exec /usr/bin/mako "$@"
fi
if [ "$XDG_SESSION_DESKTOP" = xfce ]; then
exec dbus-send --session \
--dest=org.freedesktop.systemd1 \
/org/freedesktop/DBus \
org.freedesktop.systemd1.Activator.ActivationRequest \
string:xfce4-notifyd.service
fi
echo "Error: No notification service for $XDG_SESSION_DESKTOP in $0." >&2
exit 1
Then log out and back in (or killall -HUP dbus-daemon
) to apply the changes.
To test, send a desktop notification (e.g. notify-send Test
).
The notify-dispatch
script inherits its environment from the D-Bus
activation environment. For $XDG_SESSION_DESKTOP
to be set,
dbus-update-activation-environment
must be called (with --all
or XDG_SESSION_DESKTOP=...
explicitly) after
the session bus is started, before a notification is sent. On Debian, this is
done by the Xsession script
/etc/X11/Xsession.d/95dbus_update-activation-env
(from the
dbus-x11
package). (Note:
Although, as its name implies, Xsession is a component of the X Window System,
some display managers also run Xsession when starting Wayland sessions.
LightDM is one example.)
Some quick notes about connecting Subaru STARLINK to a home wireless network (e.g. for firmware updates):
These issues together significantly complicated the connecting and troubleshooting process for me. I hope that by reading this you can more easily correct or avoid these issues. Good luck!
Not long ago I helped a few people transfer their landline phone number from Charter Spectrum to Google Voice. The general process is straight-forward, but the devil is in the details. Several steps are prone to failure and delays if not done correctly. This post is my notes about the exact steps required.
The process of transferring a phone number from one carrier to another is called “porting”. (For details, see local number portability.) Google Voice only supports porting numbers from some wireless carriers. One way to port a landline to Google Voice is to first port the number to a wireless carrier. The carrier I chose to use is AT&T Prepaid.
Before beginning the porting process, you will need:
To port a landline from Charter Spectrum to Google Voice, use the following steps:
Activate an AT&T prepaid plan for the SIM Card. This can be done by
online, or by dialing
*123*ZIP Code*Plan Code#
from the cell phone, where ZIP Code
is your
5-digit ZIP Code and Plan Code
is the code for your preferred prepaid
plan. (I’d recommend 02
for “$2 Daily Plan” or 17
for “25¢ Per Minute
Plan”.)
Write down the phone number assigned to your plan. Note that the last 4 digits of the phone number are the AT&T account PIN.
Call 611 from the AT&T phone (or 1-800-901-9878 from any phone). Navigate to “more options”, then “tech support” in the current phone tree to speak to a tech support representative. Request to “port a number from a different carrier to AT&T” and answer the questions they ask.
Be sure to specify the name and address on the Charter Spectrum account exactly as it appears on the account. If the information in the port request does not match the information in Charter Spectrum’s system exactly, the port request may be rejected, which will delay the process.
If the port doesn’t complete immediately, ask for the estimated completion date, then check att.com/port/ or call the AT&T Ports Department at 1-888-898-7685 to check the port status.
If the port is rejected or doesn’t complete by the date given, call AT&T Ports Department at 1-888-898-7685 to inquire and/or retry. If the support rep. is unable to determine the reason the port failed, call the Charter Spectrum ports department at 1-844-881-2092 to have them determine the reason.
Note that Charter Spectrum contracts with Syniverse for landline porting. If the Charter support rep. is not able to unable to explain the failure, they can call Syniverse to inquire. (Customers are not allowed to call them directly.)
Once the port is completed (“Confirmed” on att.com/port) you may need to “activate service” by confirming a phone plan and funding the account. This will be done automatically by the support rep. if the port completes during the call or can be done by calling AT&T customer support and requesting to “activate service on a ported number”.
Note that it may be possible to decline service activation (e.g. by deferring “until later”) and save a few dollars. However, I have not tried it.
IEEE 802.11w-2009 defines a mechanism for cryptographically protecting 802.11 management frames to prevent deauthentication attacks (such as the deauthentication attacks recently discussed on Hacker News). The feature is often referred to as “Protected Management Frames (PMF)” or “Management Frame Protection (MFP)” and is required by WPA3.
While configuring and testing 802.11w, I found it difficult to determine whether 802.11w was supported, enabled, and/or required on each device and network. This post documents the methods that I used.
Linux drivers set the
IEEE80211_HW_MFP_CAPABLE
flag for hardware which supports 802.11w. On kernels built with
CONFIG_MAC80211_DEBUGFS
and debugfs
mounted at /sys/kernel/debug
, this
can be checked by running:
grep MFP_CAPABLE /sys/kernel/debug/ieee80211/phy0/hwflags
(Replacing phy0
with the PHY device of interest.)
On kernels built without CONFIG_MAC80211_DEBUGFS
, the flags
member of
struct ieee80211_hw
does not appear to be exposed to user-space. The best
indicator I have found is to check for cipher suites which are only supported
on MFP-capable
hardware
(AES-CMAC
, BIP-CMAC-256
, BIP-GMAC-128
, and BIP-GMAC-256
) using the
iw
command.
For example, to check whether phy0
supports AES-CMAC
(00-0f-ac:6
):
iw phy phy0 info | grep 00-0f-ac:6
If 00-0f-ac:6
is included in the output, 802.11w is supported. Note that
listing ciphers requires the iw-full
package on
OpenWRT. Also note that phy0 info
can be omitted
from the above command to check for support on any PHY.
Additionally, to check whether the driver+firmware supports optional 802.11w,
ensure you are using iw
5.0 or later and look for MFP_OPTIONAL
in
Supported extended features:
in the iw phy
output.
To check whether a network within scanning range supports or requires 802.11w, you can run:
iw dev wlan0 scan ssid NetworkName
(Replacing wlan0
with the desired wireless interface to use for scanning.)
If RSN Capabilities
includes MFP-required
, the station requires 802.11w.
If RSN Capabilities
includes MFP-capable
, 802.11w is available, but
clients are not required to use it. Note that even network cards which do
not support 802.11w can report this information. Also note that ssid
NetworkName
can be omitted to view all networks.
To check which stations managed by interfaces on an access point support or require 802.11w, run:
iw dev wlan0 station dump
(Replacing wlan0
with the wireless interface upon which the station is
configured.) MFP: yes
indicates that 802.11w is required by the station.
MFP: no
indicates that 802.11w is either optional or not supported
by the station. (Unfortunately, I have not yet found a way to distinguish
optional from unsupported.)
GnuPG 2 uses a
pinentry program to prompt
the user for passphrases and PINs. The standard pinentry
collection includes
executables for GNOME, plain GTK+, Qt, Curses, and TTY user interfaces. By
default, the graphical programs will fall back to Curses when $DISPLAY
is
not available. For my own use, I would like the opposite behavior: Present a
text UI if a terminal is available, otherwise fall back to a graphical UI.
This post describes one way to accomplish that behavior.
gpg-agent
invokes the pinentry executable configured by pinentry-program
in
gpg-agent.conf
(default: pinentry
, which is managed by the Debian
Alternatives System on
Debian-based distros) whenever the user must be prompted for a passphrase
or PIN. The standard input and output of pinentry are pipes over which the
configuration and response information is sent in the Assuan
Protocol. (See the
pinentry
Manual
for specifics.) Additionally, environment variables which contain
configuration information passed via Assuan (e.g. $DISPLAY
, $GPG_TTY
,
$TERM
) are not passed to pinentry. (See
stdenvnames
for a full list and mapping.)
This architecture keeps pinentry simple and self-contained, but it makes environment detection and conditional execution difficult:
stdin
is always a pipe.$DISPLAY
and $GPG_TTY
are never set.To achieve the desired behavior in a robust way, without additional configuration, subject to the above constraints, likely requires implementing a pinentry program using libassuan or modifying an existing pinentry program to present a UI based on the configuration information passed via Assuan. However, I am too lazy to write and maintain my own pinentry program, so I came up with a different solution which requires a little configuration:
$PINENTRY_USER_DATA
for ConfigurationAs a result of Task 799, GnuPG 2.08 and later
pass the PINENTRY_USER_DATA
environment variable from the calling
environment to gpg-agent to pinentry. The format of this variable is not
specified (and not used by any programs in the standard pinentry collection
that I can find). pinentry-mac
assumes it is a comma-separated sequence of NAME=VALUE pairs with no quoting
or
escaping
and recognizes USE_CURSES=1 to control curses
fallback. I adopted this
convention for a simple pinentry script which chooses the UI based on the
presence of USE_TTY=1
in $PINENTRY_USER_DATA
:
#!/bin/sh
# Choose between pinentry-tty and pinentry-x11 based on whether
# $PINENTRY_USER_DATA contains USE_TTY=1
#
# Based on:
# https://kevinlocke.name/bits/2019/07/31/prefer-terminal-for-gpg-pinentry
#
# Note: Environment detection is difficult.
# - stdin is Assuan pipe, preventing tty checking
# - configuration info (e.g. ttyname) is passed via Assuan pipe, preventing
# parsing or fallback without implementing Assuan protocol.
# - environment is sanitized by atfork_cb in call-pinentry.c (removing $GPG_TTY)
#
# $PINENTRY_USER_DATA is preserved since 2.08 https://dev.gnupg.org/T799
#
# Format of $PINENTRY_USER_DATA not specified (that I can find), pinentry-mac
# assumes comma-separated sequence of NAME=VALUE with no escaping mechanism
# https://github.com/GPGTools/pinentry-mac/blob/v0.9.4/Source/AppDelegate.m#L78
# and recognizes USE_CURSES=1 for curses fallback
# https://github.com/GPGTools/pinentry-mac/pull/2
#
# To the extent possible under law, Kevin Locke <kevin@kevinlocke.name> has
# waived all copyright and related or neighboring rights to this work
# under the terms of CC0: https://creativecommons.org/publicdomain/zero/1.0/
set -Ceu
# Use pinentry-tty if $PINENTRY_USER_DATA contains USE_TTY=1
case "${PINENTRY_USER_DATA-}" in
*USE_TTY=1*)
# Note: Change to pinentry-curses if a Curses UI is preferred.
exec pinentry-tty "$@"
;;
esac
# Otherwise, use any X11 UI (configured by Debian Alternatives System)
# Note: Will fall back to curses if $DISPLAY is not available.
exec pinentry-x11 "$@"
To use this script for pinentry:
~/bin/pinentry-auto
).chmod +x pinentry-auto
).pinentry-program /path/to/pinentry-auto
to ~/.gnupg/gpg-agent.conf
.export PINENTRY_USER_DATA=USE_TTY=1
in environments where prompting via
TTY is desired (e.g. alongside $GPG_TTY
in ~/.bashrc
).The script and settings are also available as a Gist.
Recently I reimplemented client-side (i.e. in-browser) JavaScript error reporting for a web application that I had written years ago. This post outlines some of the things I discovered and provides a basic implementation.
For client-side error reporting, as with many things, it is relatively simple to create a basic implementation and quite difficult to create a complete and robust one. Developers without an existing error collection system are encouraged to consider existing services, such as Bugsnag (JS client source), Rollbar (JS client source), Sentry (JS client source), and others which have (presumably) already solved the problems in this post along with many others.
error
and unhandledrejection
eventsUncaught exceptions cause error
events
while unhandled
Promise
rejections cause
unhandledrejection
(but only on Chrome 49+, Edge, Firefox 69+, and
Bluebird at the time of this writing). Consider
listening for unhandledRejection
(with a capital R) events to catch
unhandled rejections from when.js and
yaku promise libraries, if they may be used.
Reporting unhandled exceptions from other promise implementations requires
calling the reporting function explicitly.
Websites which support any version of Internet Explorer (or Edge on a local
network) should consider how to handle errors on unsupported IE versions which
might be triggered by Compatibility
View
(from Compatibility View
List
(by user, admin, or MS), Security
Zone,
or intranet site detection settings),
EMIE,
X-UA-Compatible
(in document or HTTP header), and/or Edge Enterprise
Mode.
Be aware that the value sent in the User-Agent
header of the HTTP request
does not accurately reflect the browser version in these modes (e.g. IE 11
sends the User-Agent
for IE 7 in Compatibility View, but will still operate
in IE 11 document mode based on X-UA-Compatible: IE=Edge
in the response).
Should errors in unsupported modes be reported (to the user, administrator, or
webmaster) or ignored?
Be aware that IE 8 and before do not support addEventListener
, so
window.onerror
must be used. Also, the Event
object passed to error
event listeners by IE 9 does not include error information, which must be
retrieved from window.event
with non-standard property names
(errorMessage
, errorUrl
, errorLine
, errorCharacter
).
fetch
with keepalive
or sendBeacon
Using XMLHttpRequest is problematic because
the page may be unloaded before the request is made, causing the request to be
aborted. This is especially likely if the error occurs during the
unload
or
beforeunload
events. Previously this could be avoided by making a synchronous request
(calling open
with async
false
), but synchronous requests cause delays for the user and have been
deprecated
and will be disallowed during page dismissal in
Chrome
and in Firefox.
The preferred solution is to use fetch
with keepalive:
true
. Unfortunately,
this is not yet supported in Safari or Firefox (Bug
1342484). A more portable solution is
to use
navigator.sendBeacon
.
Unfortunately, CORS
support is in flux and Chrome
currently rejects non-CORS blob types, so making a
simple
request
(with application/x-www-form-urlencoded
, multipart/form-data
, or
text/plain
body) is recommended. Also note that Chrome sends
URLSearchParams
as text/plain
instead of application/x-www-form-urlencoded
due to Bug
747787, so using a Blob
may be necessary.
The JavaScript that is run in the browser is often the result of transforming (e.g. transpiling, bundling, minifying) source files in complex ways. The stack traces may be significantly more useful if they refer to names and locations used in the source files. This can be accomplished using information from source maps with libraries such as stacktrace.js.
Important Caveat: stacktrace.js fetches source maps and sources asynchronously, which reintroduces the problem solved in the previous section: That the page may unload before the sources are resolved, preventing the error from being reported. This can be avoided by resolving the sources on the error reporting server. Alternatively, the error could be reported twice, both before and after the sources are resolved, and the unresolved report discarded by the server when the resolved report is received.
With the above tips in mind, here is a simplified version of the error reporting script that I came up with (which omits source resolution due to the need for server coordination):
// JavaScript error reporting functions, including automatic window.onerror
// and unhandledrejection reporting.
//
// Based on:
// https://kevinlocke.name/bits/2019/07/30/more-robust-javascript-error-reporting/
//
// API:
// reportError(message, error):
// Report an exception with optional message and exception value.
// reportRejection(message, cause):
// Report a rejection with optional message and cause.
// setReportUrl(newReportUrl):
// Set URL to which reports are POSTed as application/x-www-form-urlencoded
// Must be called before reporting any errors unless a default reportUrl is
// defined below.
//
// Note: unhandledrejection is only raised by Chrome 49+, Edge, Firefox 69+,
// and Bluebird. Others must use .catch(errorReporting.reportRejection).
// when.js and yaku users could call reportRejection on unhandledRejection.
//
// Note: This script is intended to work with IE 6+ so that errors are reported
// for incorrect Compatibility View, EMIE, and/or X-UA-Compatible settings.
//
// To the extent possible under law, Kevin Locke <kevin@kevinlocke.name> has
// waived all copyright and related or neighboring rights to this work.
// See https://creativecommons.org/publicdomain/zero/1.0/
// Universal Module Definition (UMD) for Node, AMD, and browser globals
// https://github.com/umdjs/umd/blob/master/templates/returnExports.js
(function(root, factory) {
if (typeof define === 'function' && define.amd) {
// AMD. Register as an anonymous module.
define([], factory);
} else if (typeof module === 'object' && module.exports) {
// Node. Does not work with strict CommonJS, but
// only CommonJS-like environments that support module.exports,
// like Node.
module.exports = factory();
} else {
// Browser globals (root is window)
root.errorReporting = factory();
}
}(typeof self !== 'undefined' ? self : this, function() {
'use strict';
var reportUrl;
// hasOwnProperty is not available from global scope in IE
// eslint-disable-next-line no-shadow
var hasOwnProperty = Object.prototype.hasOwnProperty;
function logError(/* args */) {
try {
// eslint-disable-next-line no-console
console.error.apply(console, arguments);
} catch (err) {
// Ignore failures (e.g. console not available). Can't log.
}
}
/** Gets the URL-encoding of a given string.
* https://github.com/jerrybendy/url-search-params-polyfill/blob/v7.0.0/index.js#L117
* @private
*/
function urlEncode(str) {
var replace = {
'!': '%21',
"'": '%27',
'(': '%28',
')': '%29',
'~': '%7E',
'%20': '+',
'%00': '\x00'
};
return encodeURIComponent(str)
.replace(/[!'()~]|%20|%00/g, function(match) {
return replace[match];
});
}
// eslint-disable-next-line no-shadow
var URLSearchParams = window.URLSearchParams
|| function URLSearchParamsPolyfill() {
var params = {};
this.set = function URLSearchParamsPolyfill$set(param, value) {
params[param] = value == null ? '' : String(value);
};
this.toString = function URLSearchParamsPolyfill$toString() {
var query = [];
for (var param in params) {
if (hasOwnProperty.call(params, param)) {
query.push(param + '=' + urlEncode(params[param]));
}
}
return query.join('&');
};
};
function sendReport(url, report) {
// sendBeacon support for CORS is in flux. Now spec'd as no-cors mode.
// https://bugzilla.mozilla.org/1280692
// https://bugzilla.mozilla.org/1289387
// Chrome rejects non-CORS Blob types: https://crbug.com/490015
// ScriptService doesn't support multipart/form-data, so use urlencoded
var reportParams = new URLSearchParams();
for (var reportProp in report) {
if (hasOwnProperty.call(report, reportProp)) {
var reportVal = report[reportProp];
// Omit null and undefined values since urlencoded values are strings
// and URLSearchParams encodes null as 'null', undefined as 'undefined'.
if (reportVal != null) {
reportParams.set(reportProp, reportVal);
}
}
}
var reportParamsStr = reportParams.toString();
// Use sendBeacon, when available, to ensure error report is delivered,
// even when the page is unloading, without impacting UX.
// https://groups.google.com/a/chromium.org/d/msg/blink-dev/LnqwTCiT9Gs/tO0IBO4PAwAJ
// https://bugzilla.mozilla.org/980902
// https://bugzilla.mozilla.org/1542967
//
// Note: Could use fetch with keepalive:true, where supported
// (e.g. by checking whether new Request().keepalive is undefined)
// Better CORS support, but less widely implemented.
// https://bugzilla.mozilla.org/1342484
if (typeof navigator.sendBeacon === 'function') {
// Chrome sends URLSearchParams as text/plain - https://crbug.com/747787
// convert to Blob to avoid the issue
var reportBlob = new Blob(
[reportParamsStr],
{type: 'application/x-www-form-urlencoded'}
);
// Note: Chrome throws on CORS type error. Protect with try-catch.
try {
if (navigator.sendBeacon(url, reportBlob)) {
return true;
}
} catch (errSendBeacon) {
logError('Error calling sendBeacon', errSendBeacon);
}
}
try {
// Note: IE < 7 didn't provide XMLHttpRequest, but it is available in
// IE 5 compatibility mode on IE 11. Don't bother providing a fallback.
var req = new XMLHttpRequest();
// If an error occurs during unload or beforeunload, need to send
// synchronously to ensure the request is sent before the page unloads.
// FIXME: Can window.onerror be triggered from window.beforeunload?
// If so, this check likely won't detect that case.
var isAsync = !window.event
|| (window.event.type !== 'unload'
&& window.event.type !== 'beforeunload');
req.open('POST', url, isAsync);
req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
req.send(reportParamsStr);
return true;
} catch (errXHR) {
logError('Error sending XMLHttpRequest', errXHR);
return false;
}
}
function errorEventToReport(errorEvent) {
// IE 9 uses non-standard ErrorEvent property names which are available on
// window.event but not on the event argument.
var windowEvent = window.event;
if (!errorEvent.error
&& !errorEvent.message
&& windowEvent
&& windowEvent.errorMessage) {
errorEvent = {
type: errorEvent.type,
message: windowEvent.errorMessage,
filename: windowEvent.errorUrl,
lineno: windowEvent.errorLine,
colno: windowEvent.errorCharacter
};
}
// Note: error may not be instanceof Error. Handle carefully.
var error = errorEvent.error || errorEvent.cause;
var errorString = error == null ? null : String(error);
// Note: Error.prototype.toString isn't useful on IE < 8. Detect and fix.
if (error
&& typeof error.message === 'string'
&& errorString === Object.prototype.toString.call(error)) {
if (typeof error.name === 'string') {
errorString = error.name + ': ' + error.message;
} else {
errorString = error.message;
}
}
var eventMessage =
errorEvent.message == null ? null : String(errorEvent.message);
var message =
// If there was no event message, use error string (if any)
!eventMessage ? errorString
// If there was no error string, use event message (if any)
: !errorString ? eventMessage
// If error string contains event message, use error string
// e.g. On Edge, IE eventMessage === error.message
: errorString.indexOf(eventMessage) >= 0 ? errorString
// If event message contains error string, use event message
// e.g. On Chrome eventMessage === 'Unhandled ' + errorString
: eventMessage.indexOf(errorString) >= 0 ? eventMessage
// Otherwise, combine them
: eventMessage + ': ' + errorString;
// FIXME: To get more useful stack information (especially when minified),
// consider https://github.com/stacktracejs/stacktrace.js
// May want to send unresolved stack early and resolved later, in case page
// unloads before source resolution completes.
var stack = error && error.stack;
if (stack) {
// Remove redundancy between stack and message
var nlPos = stack.indexOf('\n');
var firstLine = nlPos > 0 ? stack.slice(0, nlPos) : null;
if (firstLine === errorString) {
// First line of stack is error (Chrome, Edge, IE). Remove it.
stack = stack.slice(nlPos + 1);
}
}
if (!stack) {
if (errorEvent.filename) {
stack = ' at ' + errorEvent.filename;
if (errorEvent.lineno) {
stack += ':' + errorEvent.lineno;
if (errorEvent.colno) {
stack += ':' + errorEvent.colno;
}
}
} else {
try {
throw new Error('Reported from');
} catch (err) {
stack = err.stack;
}
}
}
return {
type: errorEvent.type,
message: message,
stack: stack,
url: location.href,
referrer: document.referrer,
// Note: May differ from header due to Compatibility View + X-UA-Compat
userAgent: navigator.userAgent
};
}
function sendError(errorEvent) {
if (!reportUrl) {
logError('Unable to send error report: Report URL not set', errorEvent);
return false;
}
return sendReport(reportUrl, errorEventToReport(errorEvent));
}
/** Reports an error to the configured report URL.
* @param {string=} message Optional error message.
* @param {*} error Optional error message.
* @return {bool} Was the report successfully sent or queued to send?
* Note: An error is logged to the console if the report can not be sent.
*/
function reportError(message, error) {
if (error == null && typeof message !== 'string') {
error = message;
message = undefined;
}
// Log error to console for parity with unhandled exceptions
logError(message || 'Reporting error', error);
return sendError({
type: 'error',
message: message == null ? undefined : String(message),
error: error
});
}
/** Reports a rejection to the configured report URL.
* @param {string=} message Optional error message.
* @param {*} cause Optional rejection cause.
* @return {bool} Was the report successfully sent or queued to send?
* Note: An error is logged to the console if the report can not be sent.
*/
function reportRejection(message, cause) {
if (cause == null && typeof message !== 'string') {
cause = message;
message = undefined;
}
// Log error to console for parity with unhandledrejection events
logError(message || 'Reporting unhandledrejection', cause);
return sendError({
type: 'unhandledrejection',
message: message == null ? undefined : String(message),
cause: cause
});
}
/** Sets the URL to which errors are reported.
* @param {string} newReportUrl URL to which errors should be reported.
*/
function setReportUrl(newReportUrl) {
reportUrl = newReportUrl;
}
if (window.addEventListener) {
window.addEventListener('error', sendError, false);
window.addEventListener('unhandledrejection', sendError, false);
} else {
var oldonerror = window.onerror;
window.onerror = function(message, filename, lineno, colno, error) {
sendError({
type: 'error',
message: message,
filename: filename,
lineno: lineno,
colno: colno,
error: error
});
return oldonerror ? oldonerror.apply(this, arguments) : false;
};
}
return {
reportError: reportError,
reportRejection: reportRejection,
setReportUrl: setReportUrl,
};
}));
For a number of reasons, including the use of better cryptographic algorithms and key management using a hardware security module exclusively, I have recently set up a new OpenPGP key and will be transitioning away from my old one.
The old key will continue to be valid for some time, but I prefer that the new key be used whenever possible.
The old key was 0x5FF14E6B8CE71E1C:
pub dsa1024/0x5FF14E6B8CE71E1C 2007-03-17
Key fingerprint = 03CF 7D34 460F BCE6 E8B4 DA43 5FF1 4E6B 8CE7 1E1C
The new key is 0x498A12A1584EA6F9:
pub rsa4096/0x498A12A1584EA6F9 2019-06-11
Key fingerprint = E98C 0D0D 7CD6 8D7D DDCF 7034 498A 12A1 584E A6F9
The full text of this transition statement, signed by both keys and including instructions for use and verification, is also available.
Recently I helped a client update an ASP.NET web site project from ReportViewer 2005 to ReportViewer 2017. This post documents a few issues that I encountered during the process:
Opening a ReportViewer 2005 RDLC file in the Microsoft RDLC Report Designer for Visual Studio produces the prompt “Do you want to convert this report to the latest RDLC format?” After clicking OK, everything appears to work, except that saving the report displays “Unknown Report Version: 9.0” an error dialog box. This issue can be worked around by first saving the RDLC files using an intermediate version of ReportViewer. I tested Visual Studio Community 2015 with SQL Server Data Tools (SSDT) version 17.4 (build 14.0.61712.050), 17.2 (build 14.0.61707.300), and from the VS2015 Installer (build 14.0.60519.0). All versions worked. Visual Studio Community 2013 with either SSDT or SSDT-BI should also work, although I did not test it.
<Style>
ConversionAfter completing the above procedure and saving the RDLC files, <Style>
elements inside <Body>
are incorrectly applied to the <Page>
element. To
correct the problem, simply edit the RDLC files with a text or XML editor and
swap the empty <Style />
in <Body>
with the non-empty one in <Page>
.
Attempting to install the
Microsoft.ReportingServices.ReportViewerControl.WebForms
NuGet package
(according to the Getting
Started
directions) results in the following error:
Install-Package : An error occurred while applying transformation to 'web.config' in project 'webdb' No element in the source document matches
'/configuration/system.web'
No element in the source document matches '/configuration/system.web'
At line:1 char:1
+ Install-Package Microsoft.ReportingServices.ReportViewerControl.WebFo ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Install-Package], Exception
+ FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
This problem was due to the presence of
xmlns="http://schemas.microsoft.com/.NetConfiguration/v2.0"
on the root
<configuration>
element in web.config
. This namespace was conventional
in the past, but is no longer expected (nor supported, apparently). After
removing xmlns
from <configuration>
in web.config
, the NuGet package
installed without error.
Compiling the project produced the following error:
The report definition is not valid. Details: The report definition has an invalid target namespace 'http://schemas.microsoft.com/sqlserver/reporting/2008/01/reportdefinition' which cannot be upgraded.
This was due to my own mistake of not correctly updating the <buildProvider>
to Version=15.0.0.0
. I mention it here in case it helps anyone else who
makes the same mistake.
Best of luck with your upgrade!
A friend recently convinced me that it’s time to disable
NetBIOS (and
WINS) based in
part on Microsoft’s recommendation not to deploy
WINS,
serious unpatched WINS
vulnerabilities,
spoofability,
and because it complicates network lookups and masks DNS problems. After
reviewing Ace Fekay’s excellent post Do I need
NetBIOS? to
check for gotchas, I decided to disable NetBIOS over TCP/IP by using DHCP
server options. This is
accomplished by setting the Vendor-Specific Option Code
0x01 to the value
0x00000001
for DHCP clients matching the Microsoft Vendor Class Identifier
(using “MSFT
” for
forward-compatibility rather than the entire “MSFT 5.0
” identifier). In
dnsmasq this can be
accomplished by adding the following to /etc/dnsmasq.conf
:
dhcp-option=vendor:MSFT,1,2i
(For reference, there is more explanation of how dhcp-option
vendor options
work in a dnsmasq-discuss
post.)
Once configured, restart dnsmasq then acquire a new DHCP lease (e.g. by
running ipconfig /release && ipconfig /renew
) and confirm NetBIOS over
TCP/IP is disabled (e.g. by running ipconfig /all
). With any luck you will
be free of NetBIOS.
I am reasonably certain that the partition type
GUID
B8CB5058-C187-4719-BAF0-379CA2D4C97E
is used for
ExpressCache partitions. Since
Wikipedia articles must not contain original
research, and I
was unable to find a source to corroborate (or refute) this finding, I am
documenting it here.
While working on a Lenovo ThinkPad Twist S230u Laptop, I noticed that it has a 24GB mSATA drive (Samsung MZMPA024) with the following partitions:
Disk /dev/sdb: 22.4 GiB, 24015495168 bytes, 46905264 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Device Start End Sectors Size Type
/dev/sdb1 2048 32223231 32221184 15.4G unknown
/dev/sdb2 32223232 46903295 14680064 7G Intel Fast Flash
The sfdisk
dump contained the following information:
label: gpt
label-id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
device: /dev/sdb
unit: sectors
first-lba: 34
last-lba: 46905230
/dev/sdb1 : start= 2048, size= 32221184, type=B8CB5058-C187-4719-BAF0-379CA2D4C97E, uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/sdb2 : start= 32223232, size= 14680064, type=D3BFE2DE-3DAF-11DF-BA40-E3A556D89593, uuid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX, name="Basic data partition", attrs="RequiredPartition GUID:63"
The 7GB second partition with type D3BFE2DE-3DAF-11DF-BA40-E3A556D89593
appears to be for Intel® Rapid Start
Technology.
The first partition type is not documented anywhere that I can find.
The machine has the Lenovo ExpressCache Software for
Windows installed.
Running eccmd -info
to check the status (per Lenovo ExpressCache mSATA
troubleshooting)
yields:
ExpressCache Command Version 1.3.118.0
Copyright© 2010-2014 Condusiv Technologies.
Date Time: 10/6/2018 10:35:45:67 (LENOVO-TWIST #4)
EC Cache Info
==================================================
Mounted : Yes
Partition Size : 15.36 GB
Reserved Size : 3.00 MB
Volume Size : 15.36 GB
Total Used Size : 6.70 GB
Total Free Space : 8.67 GB
Used Data Size : 6.63 GB
Used Data Size on Disk : 6.69 GB
Tiered Cache Stats
==================================================
Memory in use : 0 Bytes
Blocks in use : 0
Read Percent : 5.96%
Cache Stats
==================================================
Cache Volume Drive Number : 1
Total Read Count : 64828
Total Read Size : 3.52 GB
Total Cache Read Count : 22994
Total Cache Read Size : 2.01 GB
Total Write Count : 9361
Total Write Size : 194.17 MB
Total Cache Write Count : 6256
Total Cache Write Size : 96.51 MB
Cache Read Percent : 57.12%
Cache Write Percent : 49.70%
Which describes an active ExpressCache on a 15.36 GB partition. Since there are no other partitions of similar size in the system, and the Lenovo documentation mentions mSATA drives specifically, it seems reasonable to conclude that this is the mystery partition.
Talk:GUID Partition Table/Archive 1 mentions this GUID “is used by SanDisk express cache”. It also links to a SuperUser answer which identifies the type as Intel Rapid Response (with a comment claiming Smart Response). Since the drive also contains a Rapid Start partition (discussed above), I suspect that may be the cause of the confusion.
Azure App Service
provides a management interface reachable through “Advanced Tools” in the
Azure Portal for controlling App Service features.
(This interface is part of the Kudu
project.) Today I discovered that if your browser does not send the HTTP
Referer
header in
cross-origin requests, you will get Error 403 with the following content:
Error 403 - This web app is stopped.
The web app you have attempted to reach is currently stopped and does not accept any requests. Please try to reload the page or visit it again soon.
If you are the web app administrator, please find the common 403 error scenarios and resolution here. For further troubleshooting tools and recommendations, please visit Azure Portal.
Although this error may occur due to the site being stopped, as the message
and linked blog post suggest, in my case the cause is the Referer
header not
being sent. I had configured Firefox with
network.http.referer.XOriginPolicy
set to a non-zero value for privacy reasons. Setting it to 0 resolved the
error.
After a recent SD Card failure on a Raspberry Pi, I decided to research storage devices and configurations to improve performance and device lifetime. This post contains the results of that research.
As a result of an enlightening comment chain on Hacker News about SD Card reliability I started researching common NAND flash storage technologies for representing bits in flash cells. In decreasing order of cost/reliability:
Due to the high cost of SLC, there are some intermediate technologies which use MLC flash cells with firmware that only stores one bit per cell instead of two. This results in better reliability and longevity than traditional MLC at cheaper cost than SLC:
For my current project I decided to use an 8GB ATP aMLC card (AF8GSD3A or AF8GUD3A with an adapter - both are available from Digi-Key, Arrow, and other suppliers).
For my current project, power failures and hard resets are not uncommon. I need a storage configuration which performs well on an SD Card and is reasonably resistant to corruption after power failure. eMMC/SSD File System Tuning Methodology (2013) by Cogent Embedded, Inc. is a wonderful source of information for this purpose.
The most performant configuration appears to be a single partition with F2FS, a filesystem which is optimized for flash storage. Unfortunately, as noted in the “Power-Fail Tolerance” section, F2FS is unsuitable in the presence of power failure. Although it now includes an fsck utility, “[the] initial version of the tool does not fix any inconsistency”.
lockheed on Unix SE provided a corruption-resistant configuration using BTRFS RAID. This approach looks promising, with the adjustment noted in the comments to use the BTRFS DUP Profile instead of RAID1. As I understand it, the primary difference is that the BTRFS DUP profile will only read one copy when not corrupted and that the distribution of the data copies on disk may differ. However, if the SD Card deduplicates data internally this approach will not actually result in any redundancy (as noted in the DUP Profiles on a Single Device section of the mkfs.btrfs man page). I do not think SD cards currently deduplicate data internally, but this is a significant concern.
Note that BTRFS DUP/RAID can be useful because the filesystem checksums indicate corruption. Using generic software RAID1 across partitions would not reduce corruption because it does not have a way to indicate which read is bad, so it was not considered.
ext4 is a very widely deployed filesystem and the default of most Raspberry Pi
distributions. “eMMC/SSD File System Tuning Methodology” notes that ext4
tolerated power failures quite well, while BTRFS did not. This result may
have changed due to BTRFS improvements since 2013 and with the use of DUP (or
RAID1 across partitions) as described above. It may also have different
results when using the ext4 metadata_csum
feature for metadata
checksums.
However, I have not conducted a comparison.
There are also other application-specific features to consider between ext4
and BTRFS. For example, BTRFS supports filesystem snapshots, subvolumes, and
compression. Also, ext4 is built-in to the Raspberry Pi Foundation-provided
kernel builds while BTRFS is not, thus necessitating an initramfs to boot from
a BTRFS root filesystem (see
raspberrypi/linux#1550,
raspberrypi/linux#1761).
Keeping such an initramfs updated to match the kernel is also complicated on
the Pi and requires custom scripting or manual filename changes on update (see
raspberrypi/firmware#608
and
RPi-Distro/firmware#1
- note that the referenced rpi-initramfs-tools
package has not yet been
created).
Conclusion: Use ext4 with metadata_csum
or BTRFS with DUP profile for
metadata (and data, if warranted) based on application-specific
considerations and willingness to deal with initramfs issues.
Another option for reducing or mitigating corruption is to use a read-only
filesystem (or a writable filesystem mounted read-only). This can be done on
a per-directory basis (e.g. read-only root with read-write /var
) or using an
overlay filesystem such as unionfs with
either read-write partitions or tmpfs for ephemeral information. However,
this adds configuration complexity in addition to more complicated failure
scenarios.
For optimal performance and lifetime, partitions and filesystem structures
should be aligned to the erase
block
size. This size is occasionally listed on the spec sheet for the SD card.
More commonly the
preferred_erase_size
(or
discard_granularity
)
reported for the device in sysfs could be used. It is also often possible to
use flashbench
to empirically determine the erase block
size by measuring the device performance.
For the ext4 filesystem, there may be benefits to configuring the stride and/or stripe width to match the erase block size. Various methods for determining the ext4 stride and/or stripe size based on the flash media exist. I have insufficient understanding of the implications of stride and stripe size settings to know whether this is a good idea and haven’t seen any benchmarks to compare performance.
Complete Fairness Queueing
(CFQ)
has been the Linux default I/O scheduler since
2.6.18.
It is a good default, and it provides some behavior optimizations on
non-rotational
media.
However, both “eMMC/SSD File System Tuning Methodology” and Phoronix Linux
3.16: Deadline I/O Scheduler Generally Leads With A
SSD
found that both noop
and deadline
outperformed cfq
. A caveat is that
neither deadline
nor noop
support I/O prioritization (e.g.
ionice
).
If prioritization is not required, some performance can be gained by changing
the I/O scheduler. This change can be accomplished to all non-rotational
media by placing the following content in a udev rule file (e.g.
/etc/udev/rules.d/60-nonrotational-iosched.rules
):
ACTION=="add|change", KERNEL=="mmcblk[0-9]", ATTR{queue/scheduler}="deadline"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"
I recently encountered the following error while attempting to connect to a SonicWall IPsec VPN using strongSwan:
payload type ID_V1 was not encrypted
This issue has been encountered in Chromium OS and
subsequently fixed. The
fix was upstreamed to
strongSwan
and included in strongSwan 5.2.0 and later behind the
charon.accept_unencrypted_mainmode_messages
configuration option. Users
encountering the above error may want to include the following in
/etc/strongswan.conf
:
charon {
accept_unencrypted_mainmode_messages = yes
}
On Debian-based distributions this can be accomplished by editing the
appropriate line in /etc/strongswan.d/charon.conf
.
I recently ran into some difficulty enabling SQL password saving for a data source which had password saving disabled when it was created in the Power Pivot Excel Add-in. After some trial and error, I discovered that the trick is to enable password saving in the Excel workbook connection before specifying a password and enabling password saving in the Power Pivot connection. This post provides a detailed walkthrough of the process.
To enable SQL password saving (aka “Persist Security Info”) for an existing Power Pivot connection, use the following process:
Note: Excel caches the SQL password for workbook connections. During testing it is important to close and re-open the workbook before refreshing to test whether the password has been successfully saved.
While troubleshooting a graphics-related freeze on Linux I was asked whether Windows uses x2APIC. It was not immediately clear to me how to check, and my initial searching did not come up with a convenient command or WMI property to query. This post describes the method I used to read the configuration from the model-specific registers (MSRs) in hopes that it may save others the time effort of figuring it out.
The Intel(R) 64 Architecture x2APIC
Specification
says that “System software can place the local APIC in the x2APIC mode by
setting the x2APIC mode enable bit (bit 10) in the IA32_APIC_BASE
MSR at MSR
address 01BH.” Conversely, reading the IA32_APIC_BASE
MSR and checking
bit 10 will indicate whether the system is in x2APIC mode. Since the rdmsr
instruction must be executed at privilege level 0, a kernel-mode driver must
be used.
One method for reading the MSR values is to use the
msr
from the (abandoned)
Performance Inspector project:
tinstall.cmd
(as Administrator) to install the driver.msr -r APIC_BASE
The output should look something like the following:
***** msr v2.0.7 for x64 *****
CPU0 msr 0x1B = 0x00000000:FEE00900 (4276095232)
CPU1 msr 0x1B = 0x00000000:FEE00800 (4276094976)
CPU2 msr 0x1B = 0x00000000:FEE00800 (4276094976)
CPU3 msr 0x1B = 0x00000000:FEE00800 (4276094976)
Since bit 10 (0x400) is not set for any processor, it is clear that my system is not running in x2APIC mode.
It might also be possible to use the rdmsr
command
in the Debugging Tools for
Windows
to read the IA32_APIC_BASE
MSR.
Whether x2APIC is enabled or disabled is both a matter of hardware/BIOS/driver
support and a matter of policy. If the x2APIC state does not match
expectations, consider checking the Windows boot configuration using bcdedit
/enum
and
adjusting the configuration with bcdedit /set x2apicpolicy
enable
or bcdedit /set x2apicpolicy disable
as appropriate.
I recently read through Debian Bug
299007 which
resulted in the policy change to move toward /usr/local
being owned by group
root instead of group staff. The move was largely motivated by concerns
that group staff is root-equivalent (i.e. a user in group staff has all the
power of the root account) because it can create/change binaries in the root
$PATH. Although this is true, and is a good reason not to add users to group
staff, it ignores at least one good use case discussed in this post.
With /usr/local
owned by an empty staff group, you can do things like the
following:
sudo -g staff make install
Granting sudo permission for group staff to privileged user accounts allows
them to make system-wide changes after authenticating, while still providing
some protection against inadvertent changes. If the make install
script
tries to write outside of /usr/local
(e.g. due to bad configure --prefix
)
it will fail. If the user, or programs under their control, inadvertently
tries to make modifications to /usr/local
without sudo
, they will fail.
The only time they have permission to write to /usr/local
is when running
under sudo and sudo only grants /usr/local
write permission.
When used this way, the staff group provides a very basic sort of Role-Based Access Control where the user activates the staff role through sudo. It doesn’t enhance security, since the user and executing processes are still root-equivalent, but it provides some protection against unintentional misuse. For a security policy to protect against intentional misuse, a security framework such as SELinux should be used.
Note that this post isn’t arguing for keeping /usr/local
owned by group
staff by default. Since most groups are used by adding privileged users (e.g.
audio, cdrom, dialout, etc.) and there are no documented warnings or guidance
to the contrary, misuse is highly likely. I was guilty of adding users to
group staff myself before I realized the full implications. This article is
an example of how a system could be configured, as a default or non-default,
to good effect.
This post describes one way to set up Unbound as a validating, recursive, caching DNS resolver on a router running OpenWrt. The setup includes forwarding to Dnsmasq for local names.
IMPORTANT: This post is of historical interest only. OpenWrt 18.06 added
support for UCI-based configuration of Unbound and OpenWrt 21.02 added support
for the dhcp_link
option. Configuring Unbound as described in the Parallel
dnsmasq section of
README.md
should now be sufficient.
History: This post was initially written as the Unbound HOWTO on the old OpenWrt wiki for Chaos Calmer (15.05.1). It was moved to Example 2 on the Unbound services page on the new wiki and updated to work on LEDE 17.01 and OpenWrt 18.06, then subsequently removed when it became too outdated.
By default, OpenWrt uses Dnsmasq for DNS forwarding (and DHCP serving). This works well for most cases. One notable issue is that it requires a separate recursive DNS resolver, usually provided by an ISP or public DNS provider, to resolve requests. This can be a problem due to performance, hijacking, trustworthiness, misconfiguration, lack of DNSSEC support, or many other reasons. Running a recursive resolver, such as Unbound, is one solution to those problems.
The following steps assume that OpenWrt has been installed on a device and configured as desired, including the network configuration. If not, consult the Quick Start Guide for instructions.
The later steps require accessing the device using a terminal. See SSH Access for Newcomers.
The installation and configuration instructions below are written in the form of a shell script for precision and clarity to a technical audience. The script can be saved and executed, although it is recommended to run commands and make edits individually both for better understanding and because the script is written to favor readability and clarity of instruction at the cost of thorough error handling and robustness.
Note that the choice of port 53535 is arbitrary. Similar tutorials often use 5353 or 5355 (which can conflict with MDNS).
#!/bin/sh
# Steps to configure unbound on OpenWrt with dnsmasq for dynamic DNS
# Note: Clarity of instruction is favored over script speed or robustness.
# It is not idempotent.
# Show commands as executed, error out on failure or undefined variables
set -eux
# Note the local domain (Network -> DHCP & DNS -> General Settings)
lan_domain=$(uci get 'dhcp.@dnsmasq[0].domain')
# Note the LAN network address (Network -> Interfaces -> LAN -> IPv4 address)
lan_address=$(uci get network.lan.ipaddr)
# Update the package list (System -> Software -> Update lists)
opkg update
# Install unbound (System -> Software -> Find package: unbound -> Install)
opkg install unbound # Ignore error that it can't listen on port 53
# Move dnsmasq to port 53535 where it will still serve local DNS from DHCP
# Network -> DHCP & DNS -> Advanced Settings -> DNS server port to 53535
uci set 'dhcp.@dnsmasq[0].port=53535'
# Configure dnsmasq to send a DNS Server DHCP option with its LAN IP
# since it does not do this by default when port is configured.
uci add_list "dhcp.lan.dhcp_option=option:dns-server,$lan_address"
# Configure Unbound from unbound.conf, instead of generating it from UCI
# Services -> Recursive DNS -> Manual Conf
uci set 'unbound.@unbound[0].manual_conf=1'
# Save & Apply (will restart dnsmasq, DNS unreachable until unbound is up)
uci commit
# Allow unbound to query dnsmasq on the loopback address
# by adding 'do-not-query-localhost: no' to server section
sed -i '/^server:/a\ do-not-query-localhost: no' /etc/unbound/unbound.conf
# Convert the network address to a Reverse DNS domain
# https://en.wikipedia.org/wiki/Reverse_DNS_lookup
case $(uci get network.lan.netmask) in
255.255.255.0) ip_to_rdns='\3.\2.\1.in-addr.arpa' ;;
255.255.0.0) ip_to_rdns='\2.\1.in-addr.arpa' ;;
255.0.0.0) ip_to_rdns='\1.in-addr.arpa' ;;
*) echo 'More complex rDNS configuration required.' >&2 ; exit 1 ;;
esac
lan_rdns_domain=$(echo "$lan_address" | \
sed -E "s/^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$/$ip_to_rdns/")
# Check if the local addresses are in a private address range (very common)
case "$lan_address" in
0.*) ip_to_priv_rdns='0.in-addr.arpa.' ;;
10.*) ip_to_priv_rdns='10.in-addr.arpa.' ;;
169.254.*) ip_to_priv_rdns='254.169.in-addr.arpa.' ;;
172.1[6-9].*|172.2[0-9].*|172.3[0-1].*) ip_to_priv_rdns='\2.172.in-addr.arpa.' ;;
192.0.2.*) ip_to_priv_rdns='2.0.192.in-addr.arpa.' ;;
192.168.*) ip_to_priv_rdns='168.192.in-addr.arpa.' ;;
198.51.100.*) ip_to_priv_rdns='100.51.198.in-addr.arpa.' ;;
203.0.113.*) ip_to_priv_rdns='113.0.203.in-addr.arpa.' ;;
esac
if [ -n "${ip_to_priv_rdns-}" ] ; then
# Disable default "does not exist" reply for private address ranges
# by adding 'local-zone "$lan_domain" nodefault' to server section
# Note that this must be on RFC 1918/5735/5737 boundary,
# this is only equal to $lan_rdns_domain when netmask covers whole range.
lan_priv_rdns_domain=$(echo "$lan_address" | \
sed -E "s/^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$/$ip_to_priv_rdns/")
sed -i "/^server:/a\ local-zone: \"$lan_priv_rdns_domain\" nodefault" \
/etc/unbound/unbound.conf
fi
# Ignore DNSSEC chain of trust for the local domain
# by adding 'domain-insecure: "$lan_domain"' to server section
sed -i "/^server:/a\ domain-insecure: \"$lan_domain\"" /etc/unbound/unbound.conf
# Ignore DNSSEC chain of trust for the local reverse domain
# by adding 'domain-insecure: "$lan_rdns_domain"' to server section
sed -i "/^server:/a\ domain-insecure: \"$lan_rdns_domain\"" /etc/unbound/unbound.conf
# Add a forward zone for the local domain to forward requests to dnsmasq
cat >> /etc/unbound/unbound.conf <<DNS_FORWARD_ZONE
forward-zone:
name: "$lan_domain"
forward-addr: 127.0.0.1@53535
DNS_FORWARD_ZONE
# Add a forward zone for the local reverse domain to forward requests to dnsmasq
cat >> /etc/unbound/unbound.conf <<RDNS_FORWARD_ZONE
forward-zone:
name: "$lan_rdns_domain"
forward-addr: 127.0.0.1@53535
RDNS_FORWARD_ZONE
# Optionally enable DNS Rebinding protection by uncommenting private-address
# configuration and adding 'private-domain: "$lan_domain"' to server section
sed -E -i \
-e 's/(# )?private-address:/private-address:/' \
-e "/^server:/a\ private-domain: \"$lan_domain\"" \
/etc/unbound/unbound.conf
# Restart (or start) unbound (System -> Startup -> unbound -> Restart)
/etc/init.d/unbound restart
The resulting configuration (with defaults and comments removed) should look something like:
server:
do-not-query-localhost: no
domain-insecure: "0.168.192.in-addr.arpa"
domain-insecure: "example.local"
local-zone: "168.192.in-addr.arpa." nodefault
private-address: 10.0.0.0/8
private-address: 169.254.0.0/16
private-address: 172.16.0.0/12
private-address: 192.168.0.0/16
private-address: fd00::/8
private-address: fe80::/10
private-domain: "example.local"
forward-zone:
name: "example.local"
forward-addr: 127.0.0.1@53535
forward-zone:
name: "0.168.192.in-addr.arpa"
forward-addr: 127.0.0.1@53535
The above script and configuration are also available as a Gist.
After completing the above steps, DNS should be working for both local and global addresses. If it is not, here are some suggested troubleshooting steps:
Resolution can be attempted from the OpenWrt system by running nslookup
openwrt.org 127.0.0.1
and nslookup openwrt.org 127.0.0.1:53535
.
Unfortunately, the nslookup output does not distinguish between no response
and a negative response, which significantly reduces its usefulness for
debugging. A much more powerful lookup tool is DiG from the bind-dig
package. To use it run dig openwrt.org @127.0.0.1
, add -p 53535
to query
the Dnsmasq port, or add -x
with an IP in place of the domain to do a
reverse lookup.
If Unbound is not responding to any request, try restarting the service with
/etc/init.d/unbound restart
and checking the system log for
errors logread | tail
.
If the local domain or addresses result in negative responses, check that they
are resolved correctly by Dnsmasq on port 53535. If so, check that the domain
appears in domain-insecure
, local-zone
(which may be a suffix and must
match a predefined zone), and as a name
in stub-zone
.
If domains which use DNSSEC fail to resolve while other domains work, check that the system time is correct. Time skew can cause validation failures. If the time is incorrect, check the NTP client configuration.
It is relatively straightforward to extend the above configuration for IPv6. Forward resolution (from local domain to IPv6 address) does not require any additional changes to Unbound, although it may require configuration changes to Dnsmasq. See IPv6 DNS.
To configure reverse DNS for IPv6:
domain-insecure: $lan6_rdns_domain
.local-zone: $lan6_rdns_domain nodefault
if it is in a private range
(be sure to use a preconfigured
range.stub-zone
with name: "$lan6_rdns_domain"
as above.The difficulty of adding IPv6 is that Dnsmasq is compiled without DHCPv6 support and does not resolve its own name due to #17457 so the value of forwarding IPv6 reverse requests is currently rather limited. IPv6 address configuration is also more variable and more difficult to detect.
stub-zone
s to forward-zone
s for compatibility with Dnsmasq 2.79
and later which “Always return SERVFAIL
for DNS queries without the
recursion desired bit set, UNLESS acting as an authoritative DNS server.”
Since Dnsmasq auth-server
and auth-zone
are not configurable via UCI, it
can not be made authoritative without manual configuration so Unbound must
send RD queries.While helping to diagnose name resolution issues on a Windows Domain, I
discovered that Microsoft DNS Server (version 1DB10106 (6.1 build 7601))
responds to requests from the BIND DIG
tool (version 9.11) with response code 1
FORMERR
(Request format
error). This post discusses why and a workaround.
First, an example request and response, to clarify the issue:
; <<>> DiG 9.11.0 <<>> kevinlocke.name @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 59675
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 808a22be618a7750 (echoed)
;; QUESTION SECTION:
;kevinlocke.name. IN A
;; Query time: 62 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Jan 20 17:24:10 Mountain Daylight Time 2017
;; MSG SIZE rcvd: 51
DIG requested kevinlocke.name
and received FORMERR
. After some trial and
error, I determined that the issue results from DIG 9.11 sending the DNS
COOKIE option. This option was enabled
by default in BIND
9.11.
Unfortunately, adding this option causes DNS Server to treat the request as
malformed. This behavior appears to violate “Any OPTION-CODE values not
understood by a responder or requestor MUST be ignored.” from Section 6.1.2
of RFC 6891, but that is
of small consolation for a non-working system.
As a workaround, pass the +nocookie
option (or +noedns
to disable all EDNS
options) as in dig +nocookie kevinlocke.name
.
When filtering the commit history of a Git repository to contain only the history of certain files, and performance is an issue, consider the following suggestions:
--subdirectory-filter
option of
git filter-branch
, where
appropriate.--index-filter
option of
git filter-branch
and
specify the desired files as arguments.Listing the files as arguments to filter-branch
was not obvious to me, but
makes a huge difference when filtering for a small subset of the commits. As
an example, consider extracting the history for getopt from the FreeBSD src
repo:
git filter-branch --prune-empty \
--index-filter 'git ls-files -s | \
sed -n "s/\tlib\/libc\/stdlib\/getopt/\tgetopt/p" | \
GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && \
if test -f "$GIT_INDEX_FILE.new" ; then \
mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE" ; \
else \
rm "$GIT_INDEX_FILE" ; \
fi' \
HEAD -- lib/libc/stdlib/getopt*
On my laptop, without the file arguments (the -- lib/libc/stdlib/getopt*
)
after 5 minutes git estimates that the command will take about 4 more hours
(with the estimate continually increasing). With the file arguments, it
completes in about 30 seconds. By passing the file arguments, git only
applies the filter to commits which match those files. Since this can be
determined efficiently it significantly reduces the processing and resulting
run-time.
A common tactic to increase performance and decrease bandwidth is to compress
HTTP responses. This is particularly useful for text content such as the CSS,
JavaScript, and HTML that are fundamental to the web. There are several
different methods for configuring compression in Apache, but most have subtle
(or not so subtle) issues. This post continues the series of MultiViews
posts (after the earlier
XHTML and
ErrorDocuments
posts) by outlining the problems encountered in popular compression
configurations and how to avoid them using MultiViews
.
Note: Readers who are not interested in the tradeoffs or potential issues are advised to skip to the end for the working configuration.
In order to motivate the choice of MultiViews
for serving pre-compressed
content, it’s useful to consider the popular alternatives. The first is using
mod_deflate
to
compress response content on the fly. This works well for dynamic content,
but wastes resources for static content, which is needlessly recompressed for
each request.
This limitation of mod_deflate
is prominently mentioned in the
documentation, which
recommends
using mod_rewrite
to rewrite requests to their compressed alternatives when appropriate.
Although this method can work (and I recommended it to get the desired
behavior for
XHTML)
it has the major drawback that you are reimplementing content
negotiation
(which
mod_negotiation
was designed to do) and are likely to get it wrong and lack features supported
by mod_negotiation
. Some common problems and pitfalls with this approach:
Content-Encoding
header.Vary
header or setting it incorrectly (overwriting
previous values for other headers which cause the response to vary).Content-Type: application/x-gzip
instead of the underlying type.no-gzip
in the
environment to exclude the response from mod_deflate
.Accept-Encoding: gzip;q=0
header would signify that the client wants “anything but gzip”. Most
mod_rewrite
implementations would send them gzip. A more realistic
example would be a client that sends Accept-Encoding: br;q=1, gzip;q=0.5,
deflate;q=0.1
to signify that they prefer Brotli, then gzip, then deflate.
Writing mod_rewrite
rules which properly handle these sorts of expressed
preferences is extremely difficult.In addition to the above issues, this approach requires writing a redirect
rule for each supported combination of negotiated values. Supporting gzip
encoding for a few file extensions is reasonable, but if additional types and
encodings are added (much less languages or charsets) it quickly becomes
unreasonable. It also doesn’t support any of the features of Transparent
Content Negotiation which are supported
out of the box by mod_negotiation
.
As in previous posts, we will build up a solution iteratively, tackling
problems as they appear. For this to work, mod_mime
and mod_negotiation
must be enabled/loaded. On Debian and related distributions this can be done
by running a2enmod mime
and a2enmod negotiation
as root. Additionally,
mod_deflate
should not be applied to negotiated files/types. This can be
accomplished by removing its default configuration symlink at
/etc/apache2/mods-enabled/deflate.conf
or disabling the module with
a2dismod -f deflate
.
The following examples use the domain localhost
and assume the file
style.css.gz
exists in the site root.
We start by simply enabling MultiViews
and declaring the extension .gz
to
identify the gzip encoding using following configuration (which must be inside
a <Directory>
directive or .htaccess
file, as noted in the description of
the MultiViews
value for
Options
):
Options +MultiViews
AddEncoding gzip .gz
If we test the result using curl -I -H "Accept-Encoding: gzip"
http://localhost/style.css
we get something like the following:
HTTP/1.1 200 OK
Date: Sat, 21 Jan 2017 01:11:40 GMT
Server: Apache/2.4.25 (Debian)
Content-Location: style.css.gz
Vary: negotiate,accept-encoding
TCN: choice
Last-Modified: Sat, 21 Jan 2017 01:04:11 GMT
ETag: "538-5469058274adb;546906978a4c6"
Accept-Ranges: bytes
Content-Length: 1336
Content-Type: text/css
Content-Encoding: gzip
Notice that the server sent a response with the correct Content-Type
and
Content-Encoding
and as a bonus it included the
TCN headers to inform clients that the
result was negotiated and there may be other representations available.
Hurray!
Not so fast! If we add an uncompressed style.css
file to the site root, the
same request returns:
HTTP/1.1 200 OK
Date: Sat, 21 Jan 2017 01:15:34 GMT
Server: Apache/2.4.25 (Debian)
Last-Modified: Sat, 21 Jan 2017 00:05:41 GMT
ETag: "f9d-5468f86e84147"
Accept-Ranges: bytes
Content-Length: 3997
Content-Type: text/css
This response is neither negotiated or compressed! What happened?
Unfortunately, MultiViews
only negotiates requests for files which do not
exist.
After adding style.css
the request matched the uncompressed file exactly, so
the response was not negotiated and the uncompressed file was sent. This
makes what we are trying to do particularly difficult (Bug
60619).
A solution is to rename the uncompressed file with an additional extension
such as .orig
or .id
(for the identity
encoding) and include that
extension in negotiation. This could be done by adding MultiviewsMatch
Any
,
although this risks matching unexpected file types (e.g. if a type is not
assigned to .md5
, .asc
, .torrent
or other additional extensions). It
could also be done by assigning .orig
to a negotiated feature. The obvious
choice would be encoding: AddEncoding identity .orig
. Unfortunately, this
does not work as expected since Apache treats the identity
encoding
differently from an unspecified encoding with undesired results (e.g. gzip is
served for requests without Accept-Encoding
because it is smaller than the
uncompressed file). Another option would be to assign .orig
to a default
charset or language, such as AddCharset utf-8 .orig
or AddLanguage en
.orig
if all compressed files are UTF-8 or English. A third option, which I
find more appealing, is to use a no-op filter or handler such as the
default-handler
and
allow MultiViews
to match extensions assigned to handlers:
Options +MultiViews
AddEncoding gzip .gz
MultiviewsMatch Handlers
AddHandler default-handler .orig
After a bit more digging, I found François Marier has an even better
solution
of doubling the type extension. So style.css
is saved as style.css.css
on
the server and requests for /style.css
are negotiated between
style.css.css
(no encoding) and style.css.gz
(gzip encoding). This has
the added advantage of not interfering with the type-detection of any other
tools which may open the file on the server that do not recognize the .orig
file extension.
A problem with the above solution appears if we request style.css.gz
directly or request style
without an extension to negotiate the
Content-Type
.1 Consider the result for curl -I -H
"Accept-Encoding: gzip" http://localhost/style
:
HTTP/1.1 200 OK
Date: Thu, 21 Jan 2016 01:08:30 GMT
Server: Apache/2.4.18 (Debian)
Content-Location: style.css.gz
Vary: negotiate,accept,accept-encoding
TCN: choice
Last-Modified: Thu, 21 Jan 2017 01:00:51 GMT
ETag: "536-529cda2456c78;529cdb967800b"
Accept-Ranges: bytes
Content-Length: 1334
Content-Type: application/x-gzip
Content-Encoding: gzip
This is all sorts of wrong (although it is common enough that Firefox detects
it and provides a
workaround).
We wanted to send the browser a stylesheet, but instead we sent it a gzip file
(according to Content-Type
) which is gzipped (according to
Content-Encoding
). We actually sent it the same gzipped stylesheet, but
with the wrong Content-Type
. This is because Debian (and related
distributions) set AddType application/x-gzip .gz
in their default
configuration (in /etc/apache2/mods-available/mime.conf
), so for
style.css.gz
the .gz
is being interpreted as both the type and the
encoding of the file. This can be fixed using RemoveType
as follows:
Options +MultiViews
RemoveType .gz
AddEncoding gzip .gz
With this fix, the response now includes the correct headers, as in the first
example response above. Unfortunately, we’ve introduced a new problem.
Suppose we are hosting a gzipped-tarball launch-codes.tar.gz
. Requesting it
results in a response similar to the following:
HTTP/1.1 200 OK
Date: Thu, 21 Jan 2016 01:32:51 GMT
Server: Apache/2.4.18 (Debian)
Last-Modified: Thu, 21 Jan 2017 01:27:34 GMT
ETag: "3b709-529ce01dc4107"
Accept-Ranges: bytes
Content-Length: 243465
Content-Type: application/x-tar
Content-Encoding: gzip
This tells the browser that we are sending it a tar file which is compressed
for transmission. So, if the browser didn’t have workarounds for this
brokenness
too),
it would decompress the response content and save the file as
launch-codes.tar
(or worse launch-codes.tar.gz
) with uncompressed content.
What we actually wanted was to send a gzipped file with no additional content
encoding. We can achieve that by adding some further configuration to
.tar.gz
files:
Options +MultiViews
RemoveType .gz
AddEncoding gzip .gz
<FilesMatch ".+\.tar\.gz$">
RemoveEncoding .gz
AddType application/gzip .gz
</FilesMatch>
This approach can easily be extended to any other compound file extensions
that should be saved without gunzipping by altering the FilesMatch
expression. It uses the application/gzip
type of RFC
6713, which is the official type of gzip
files, but which lacks the same browser support as the legacy
application/x-gzip
type. Administrators concerned about older browsers
should use the legacy type. Also, as a matter of style, the configuration
could have used ForceType
instead of AddType
within the FilesMatch
directive.
With this configuration we have eliminated all of the previous issues and achieved the desired result. It can also be extend to include additional encodings easily, as we will demonstrate.
Now that we have found a working solution using MultiViews
, lets add support
for
Brotli
as icing on the cake.
The first question is what extension to use, since the brotli
tool does not provide one. Using .br
analogously to .gz
provokes a conflict with the ISO 639 language
code for Breton,
which is configured by default (but can be addressed by RemoveLanguage .br
).
Using .bro
as suggested in this pull
request has already been rejected
by Mozilla. So
lets use .brotli
as a neutral, if verbose, choice.
Options +MultiViews
RemoveType .gz
AddEncoding gzip .gz
AddEncoding br .brotli
<FilesMatch ".+\.tar\.gz$">
RemoveEncoding .gz
AddType application/gzip .gz
</FilesMatch>
If we then create style.css.brotli
with brotli < style.css.orig >
style.css.brotli
, a test request with curl -I -H 'Accept-Encoding:
br' http://localhost/style.css
yields:
HTTP/1.1 200 OK
Date: Sat, 21 Jan 2017 03:07:00 GMT
Server: Apache/2.4.25 (Debian)
Content-Location: style.css.brotli
Vary: negotiate,accept-encoding
TCN: choice
Last-Modified: Sat, 21 Jan 2017 03:05:02 GMT
ETag: "43b-54692084e8c35;546920f3c26ac"
Accept-Ranges: bytes
Content-Length: 1083
Content-Type: text/css
Content-Encoding: br
Hurrah!
The final configuration, which addresses all of the above issues is:
# Enable MultiViews for content negotiation
Options +MultiViews
# Treat .gz as gzip encoding, not application/gzip type
RemoveType .gz
AddEncoding gzip .gz
# Treat .brotli as br encoding
# Note: If using .br for brotli, uncomment the following line:
#RemoveLanguage .br
AddEncoding br .brotli
# As an exception, send .tar.gz files as gzip type, not gzip encoding
<FilesMatch ".+\.tar\.gz$">
RemoveEncoding .gz
# Note: Can use application/x-gzip for backwards-compatibility
AddType application/gzip .gz
# Alternatively:
#ForceType application/gzip
</FilesMatch>
This configuration requires that uncompressed files be renamed with a
double-extension (e.g. style.css.css
) unless one of the alternatives in
the Non-Negotiated Files section is used.
This configuration intentionally omits support for deflate
encoding due to
compatibility issues and no
significant use case that I am aware of, since all browsers which support
deflate support gzip. It could be easily added with AddEncoding deflate
.zlib
or similar if desired.
This configuration also does not provide a FilesMatch
for .tar.brotli
since
this format is not currently widely used. When serving tarballs that should
be saved as brotli-compressed, add a FilesMatch
directive analogous to the
one for tar.gz
. Doing so is left as an exercise for the reader.
If you encounter issues with this solution, please let me know. Otherwise, best of luck serving pre-compressed files with Apache!
Although type negotiation is not often used for stylesheets, it is currently used to negotiate WebP, XHTML, and in some REST APIs. ↩
Web developers and admins looking to tighten the security of their websites
should consider defining a Content Security
Policy for their site. For sites hosted using
Apache, a simple way to achieve this is by
sending the Content-Security-Policy
header using
mod_headers.
Unfortunately, making this simple solution robust is more difficult than it
first appears. This post describes a method for setting or modifying the
Content-Security-Policy
header in a way that won’t clobber previous values
set by earlier configuration options or returned by an application server.
A first-attempt at setting the Content-Security-Policy
header using
mod_header
may look something like this:
Header always set Content-Security-Policy "referrer origin"
For simple use cases, this is straight-forward and sufficient to get the job
done. But what happens if the site includes an application which sets its own
Content-Security-Policy
header (either via .htaccess
or from a dynamic
page or application server)? This configuration will clobber it. That’s not
ideal.
An easy solution would be to use setifempty
and require that any overriding
configuration include the entire policy. This can make configuration changes
during deployment difficult and it pushes all policy decisions up into the
application, which may be undesirable. It’s possible to do better.
A better solution would be to use append
or merge
. Unfortunately, a quick
look at the documentation reveals that this is not quite right. The major
issue is that it treats the header as a comma-separated list, while
Content-Security-Policy
is semicolon-separated, and there is no way to
change that behavior.
After pondering this problem for a bit, I realized it would not be difficult
to implement the merge
behavior using a combination of setifempty
and
edit
. Here’s how:
Header always setifempty Content-Security-Policy ""
Header always edit Content-Security-Policy "^(?!(?:.*;)?\s*referrer\s)" "referrer origin;"
It first ensures that the header exists using setifempty
(otherwise edit
will not apply), then prepends the referrer
policy only if the header does
not already contain one (by matching with a negative-lookahead). Note that it
relies on the fact that extra semicolons are permitted in both
CSP1 and
CSP2, since that will occur when
the header is empty. Alternatively, it’s easy to add another edit
command
to remove a tailing semicolon if it is not desirable for some reason.
But wait, there’s more! Using edit
provides more power than just
prepending. With a quick adjustment, the regular expression can be used to
unconditionally replace policy components. Here’s how:
Header always setifempty Content-Security-Policy ""
Header always edit Content-Security-Policy "(^(?!(?:.*;)?\s*referrer\s)|(?:.*;)?\s*referrer\s+[^;]+;?)" "referrer origin;"
Now the regular expression matches against either the beginning of the string, if it does not already contain the referrer policy, or against the existing referrer policy if it does. This way the configured policy directive is always used, regardless of any other policies present. This method can also be applied multiple times for multiple policy directives, using either the first or second variant to either preserve or overwrite the policy directives respectively:
Header always setifempty Content-Security-Policy ""
# Override the referrer policy directive, if present
Header always edit Content-Security-Policy "(^(?!(?:.*;)?\s*referrer\s)|(?:.*;)?\s*referrer\s+[^;]+;?)" "referrer origin;"
# Preserve the script-src policy directive, if present
Header always edit Content-Security-Policy "^(?!(?:.*;)?\s*script-src\s)" "script-src 'self';"
This technique allows defining directives and overriding them wherever it is most convenient, in either the application or in the Apache configuration files, while minimizing the risk of unintentionally overwriting the policy, either in whole or in part. I hope you find it useful!
For those of you who are Serving XHTML with Apache
MultiViews
you may want to be careful about how MultiViews
interacts with
ErrorDocument
.
Configuring error documents with content negotiation can lead to compound
errors in the case that the client does not accept any of the types available
for the error document. This results in both unexpected behavior and a
suboptimal user experience. This post describes how to avoid such errors
while still negotiating the returned content type.
Lets assume that you have a website with the following .htaccess
file:
Options +MultiViews
ErrorDocument 404 /404
Along with 404.html
:
<!DOCTYPE html>
<html>
<head><title>404 HTML</title></head>
<body><h1>404 HTML</h1></body>
</html>
and 404.xhtml
:
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head><title>404 XHTML</title></head>
<body><h1>404 XHTML</h1></body>
</html>
Accessing a non-existent URL from your browser will result in a 404 response
with the content from the file of the preferred content type being returned,
as expected. But what happens to a user agent which requests an unsupported
type, such as a bot collecting favicons? Lets examine the result of running
curl -i -H "Accept: image/vnd.microsoft.icon, image/x-icon"
http://localhost/favicon.ico
:
HTTP/1.1 404 Not Found
Date: Thu, 01 Oct 2015 03:57:33 GMT
Server: Apache/2.4.16 (Debian)
Alternates: {"404.html" 1 {type text/html} {length 99}}, {"404.xhtml" 1 {type application/xhtml+xml} {length 155}}
Vary: negotiate,accept
TCN: list
Content-Length: 403
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /favicon.ico was not found on this server.</p>
<p>Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache/2.4.16 (Debian) Server at localhost Port 80</address>
</body></html>
Which is accompanied by the following error in the Apache log:
[negotiation:error] [pid XXXX] [client ::1:XXXXX] AH00690: no acceptable variant: /path/to/404
The problem is that the client has requested /favicon.ico
, which doesn’t
exist, and negotiating the ErrorDocument
for /404
has failed because the
client only accepts icon types and the ErrorDocument
is only available in
HTML types. The result is a server-generated error page announcing the
compound-error. (Although it describes the second error as another 404,
rather than a 406, which is odd.) Not ideal.
Ideally Apache would provide a configuration option to specify a fallback
content type when none is accepted, analogous to what
ForceLanguagePriority
does for content language. This could then be scoped to error pages using
<Directory>
or <Files>
. Unfortunately, I could not find any way to
specify such a fallback.
The solution that I came up with is to set ErrorDocument
conditionally by
matching against the Accept
header. This is basically a poor-man’s content
negotiation, but works reasonably well when there are few types to choose
between and quality comparison isn’t required. To negotiate between the HTML
and XHTML versions of the 404 page, modify .htaccess
as follows:
Options +MultiViews
<If "%{HTTP_ACCEPT} =~ m#application/xhtml\+xml#i">
ErrorDocument 404 /404.xhtml
</If>
<Else>
ErrorDocument 404 /404.html
</Else>
This sends the XHTML version whenever application/xhtml+xml
is present in
the Accept
header (which, in practice, is only true for browsers which
support it and prefer it equally to text/html
) and otherwise send the HTML
version. The same curl command now returns:
HTTP/1.1 404 Not Found
Date: Thu, 01 Oct 2015 04:16:13 GMT
Server: Apache/2.4.16 (Debian)
Vary: Accept
Last-Modified: Thu, 01 Oct 2015 03:54:21 GMT
ETag: "63-5210300883cb3;521034eaa61f4"
Accept-Ranges: bytes
Content-Length: 99
Content-Type: text/html
<!DOCTYPE html>
<html>
<head><title>404 HTML</title></head>
<body><h1>404 HTML</h1></body>
</html>
Another drawback of this approach, which you can see from the response above, is that the response omits the Transparent Content Negotiation (TCN) headers. Although RFC 2295 does not specify 404 behavior explicitly, my reading is that since the representation of the 404 is negotiated, the headers should indicate the chosen representation and expose the negotiation process. But, as with the above limitations, it has little practical effect since the response includes the preferred type and I am not aware of any user agents that would want to dynamically negotiate error documents.
Although the above solution works well when only one dimension, with few
alternatives, is being negotiated. It quickly becomes unwieldy when multiple
dimensions (e.g. type and language) are under consideration. My attempts to
use MultiViews
to negotiate only the language have so far been unsuccessful.
The Apache documentation includes an example of a language-negotiated
ErrorDocument
configuration using a type-map in
httpd-multilang-errordoc.conf.
Although I expected that combining this technique with a type-map that only
negotiates on language (such as in Apache: The Definitive Guide Section
6.4) would achieve
the desired effect, it does not appear to. I am still seeing the same
AH00690
error as before.
Since I am currently only negotiating the content type, this is not an issue for me. However, if anyone is able to solve this problem, I would be very curious about how you did it, and more than happy to post the solution here for others to use. Until then, best of luck with the type-only solution!
This blog has been dead for the last couple of years, without a single post since the beginning of April, 2013. In that time I’ve been preoccupied with other endeavors, primarily Quantpost, and haven’t had time to document any of my minor discoveries. Luckily for readers, this blog is only mostly-dead, not all-dead, and I am planning to bring it back to life.
I have recently moved to Seattle, WA to attend courses at the University of Washington and am planning to make time available for writing new content. I have a few drafts of new posts that have been waiting to be completed since 2013 which are still relevant and useful, as well as a and a long list of topics that deserve attention which has accumulated over the past 2 years. I make no guarantees about the frequency or volume of upcoming posts, but you can expect several new posts in the next few weeks and posts on a more regular basis after that. Perhaps even some style improvements to make the blog easier to read. So, without further ado, on to the new posts!
I recently configured an additional encrypted partition mounted at boot using cryptsetup with LUKS. Doing so increased my boot time by about 5 seconds. In tracking down this minor annoyance, I learned two things about cryptsetup which may be helpful to others in a similar situation:
So, to reduce boot time place the key which is typically used in the first key slot (or specify the key slot explicitly) and, depending on the particular security requirements for the device, reduce the iteration count when creating this key slot.
Recent versions of Firefox crash on startup when /proc
is not mounted.
Although this is not a problem, per se, the fact that it crashes without
giving any indication of the reason can significantly complicate testing
alpha/beta/nightly releases. This post simply lists the errors that I have
seen in hopes that it will save others some debugging time.
Current Nightly builds crash with:
###!!! ABORT: Recursive layout module initialization: file /builds/slave/m-cen-lx-ntly-0000000000000000/build/layout/build/nsLayoutModule.cpp, line 374
###!!! ABORT: Recursive layout module initialization: file /builds/slave/m-cen-lx-ntly-0000000000000000/build/layout/build/nsLayoutModule.cpp, line 374
Current Aurora and Beta builds crash with:
###!!! ABORT: Recursive layout module initialization: file /builds/slave/m-aurora-lx-ntly-0000000000000/build/layout/build/nsLayoutModule.cpp, line 372
###!!! ABORT: Recursive layout module initialization: file /builds/slave/m-aurora-lx-ntly-0000000000000/build/layout/build/nsLayoutModule.cpp, line 372
Current Nightly and Aurora debug builds crash with:
Assertion failure: stackBase, at ../../../js/src/jsnativestack.cpp:139
Make sure /proc
is mounted!
I wouldn’t argue that Firefox should work without /proc
, but I do think it
would be preferable to inform users why Firefox can’t start, if it’s feasible.
After asking on #firefox without response,
I’m tempted to let it lie. If anyone else wants to work on a fix, count me
in.
I just finished tracking down a rather esoteric bug in a Scala application that I am writing. Understanding this bug requires some understanding of how Scala is translated to Java and how Java handles static initialization, neither of which will be explained (much) in this post. So, if you are interested in how default parameters on a constructor can cause circular static initialization resulting in a NullPointerError, read on.
In order to see the problem, consider the following program:
case class Hobbit(
name: String,
hasRing: Boolean = false
)
object Hobbit {
object Frodo extends Hobbit("Frodo", true)
object Merry extends Hobbit("Merry")
object Pippin extends Hobbit("Pippin")
object Sam extends Hobbit("Sam")
val gardenerName = Sam.name
}
object RunMe {
def main(args: Array[String]) {
println(Hobbit.Sam.name)
}
}
What will happen if RunMe is launched?
If you thought a NullPointerException
would occur, you are correct. But
why?
As a reminder, Scala objects are represented as Java classes with $
appended
to their name and their members are also accessible through static methods on
the Java class with a matching name which delegate to the $
class through a
static variable named MODULE$
on the $
class.
The critical detail behind the error is that default arguments are also stored
as methods on the object for the containing class. So the default value of
hasRing
is stored as a method (named init$default$2
, if you were curious)
in the Hobbit$
class (and a static method of the same name on the Hobbit
class). When we combine this with the Initialization Procedure defined in
the Java Language
Spec.
what happens is as follows (simplified):
Hobbit.Sam$.MODULE$.name
is accessed which starts static initialization
of Hobbit.Sam$
.Hobbit$.MODULE$.init$default$2
is accessed to get the default argument
for the Hobbit
constructor, which starts static initialization of
Hobbit$
.Hobbit.Sam$.MODULE$.name
is accessed in order to assign the name from its
value to gardenerName
. Initialization is already in progress (started in
step 1) so the current value of MODULE$
is returned and name
is called
on it. Unfortunately, the current value is still null
because the
Hobbit
constructor has not yet been called for its initialization, which
results in the NullPointerException
.Got it? If not, another example, and the bug to follow for updates on this behavior, is SI-5366.
It’s also worth pointing out that this bug can be complicated by several
factors. If any member of Hobbit
other than Sam
is accessed first, the
error will not occur. (Can you see why?) When multiple threads are competing
for the first access to this class using different members, it can make the
behavior non-deterministic. It can also be complicated by something eating
the ExceptionInInitializerError
so that only the subsequent
NoClassDefFoundError
is shown. I have still not figured out what is eating
that error in my case… but I’m still working on it.
If a delete trigger is fired on a table due to an ON DELETE CASCADE
action,
will the trigger see the rows in the parent table which triggered the cascade?
Will a trigger on the originating table see rows in the child tables? Does it
matter if the trigger is a “before” or an “after” trigger? The answer to these
questions was not immediately obvious to me, and my half-minute of searching
didn’t find a clear answer, so I have written this post to remind myself and
others what happens in PostgreSQL 9.1.
In order to test the behavior, I wrote the following test script. It simply creates a parent table, child table, results table, and a trigger which fires on both the parent and child to record whether the parent row being deleted is present in the parent table.
CREATE TABLE parents (
parent_id INTEGER NOT NULL PRIMARY KEY
);
CREATE TABLE children (
child_id INTEGER NOT NULL PRIMARY KEY,
parent_id INTEGER NOT NULL REFERENCES parents(parent_id)
ON DELETE CASCADE
ON UPDATE CASCADE
);
CREATE TABLE results (
result_id SERIAL PRIMARY KEY,
table_name VARCHAR(10) NOT NULL,
trigger_when VARCHAR(10) NOT NULL,
parent_present BOOLEAN NOT NULL,
children_present INTEGER NOT NULL
);
CREATE OR REPLACE FUNCTION report_parent_id() RETURNS TRIGGER AS $$
BEGIN
INSERT INTO results (table_name, trigger_when, parent_present, children_present)
VALUES (
TG_TABLE_NAME,
TG_WHEN,
EXISTS (SELECT 1 FROM parents WHERE parent_id = OLD.parent_id),
(SELECT COUNT(*) FROM children WHERE parent_id = OLD.parent_id)
);
RETURN OLD;
END; $$ LANGUAGE plpgsql;
CREATE TRIGGER tr_parents_bd_report_parent_id
BEFORE DELETE ON parents
FOR EACH ROW EXECUTE PROCEDURE report_parent_id();
CREATE TRIGGER tr_parents_ad_report_parent_id
AFTER DELETE ON parents
FOR EACH ROW EXECUTE PROCEDURE report_parent_id();
CREATE TRIGGER tr_children_bd_report_parent_id
BEFORE DELETE ON children
FOR EACH ROW EXECUTE PROCEDURE report_parent_id();
CREATE TRIGGER tr_children_ad_report_parent_id
AFTER DELETE ON children
FOR EACH ROW EXECUTE PROCEDURE report_parent_id();
INSERT INTO parents (parent_id) VALUES (1);
INSERT INTO children (child_id, parent_id) VALUES (1, 1);
INSERT INTO children (child_id, parent_id) VALUES (2, 1);
DELETE FROM parents;
SELECT * FROM results;
Ok, here’s the answer: The parent row is not visible from either before or after triggers on the child table. It is visible in the before trigger on the parent table but not the after trigger (as expected/documented). The child rows are visible in all before triggers (showing decreasing numbers in the child trigger as child rows are deleted, as expected/documented) and no after triggers Which changes are visible during cascading updates is left as an exercise for the interested reader.
This post is just a quick warning that Flyway (before commit 55985b, which includes version 2.0.3, the current version) disables auto-commit on its JDBC Connection. Also, BoneCP (before commit 99d50d, resulting from bug 790585, which includes version 0.7.1.RELEASE, the current version) did not apply the default auto-commit or read-only setting to recycled connections. When these behaviors are combined, connections will be returned from the connection pool which have differing auto-commit. Plan accordingly.
Another quick note, version 0.8.0-rc1 has auto-commit set to false
by
default, which differs from the JDBC behavior. I consider this a
bug.
The Lift web framework integrates the SLF4J logging framework through a set of interfaces for performing logging and a configuration mechanism. The configuration mechanism attempts to configure the logging in a manner similar to the configuration for other parts of Lift. Unfortunately, this mechanism performs differently (or not at all) when running tests than it does when running normally. This post is a quick explanation of the configuration mechanism and how to configure logging during tests.
The configuration mechanism that Lift uses is documented on the Logging page on the Lift Wiki. This post presents only a simplified overview.
Logging can be configured programmatically in lift-webkit by assigning a
configuration function to LiftRules.configureLogging
(or directly to
Logger.setup
, to which LiftRules.configureLogging
delegates) before the
logging system is initialized. During initialization, the function will be called to configure the logging backend. Initialization is performed at most once,
when the first Logger
is created. So assigning a configuration function
after this point is useless.
When using lift-webkit, configuration files ending with either .logback.xml
(when using Logback) or .log4j.xml
or
.log4j.props
(when using Log4J) are
found in the same way as Lift configuration properties files. For
example, the file src/main/resources/props/production.default.logback.xml
would be used in production mode on any server, if it existed.
This automatic configuration is accomplished by the function returned from
net.liftweb.util.LoggingAutoConfigurer.apply()
, which is the default value
of LiftRules.configureLogging
.
Unfortunately, when testing (using
Specs2 or
ScalaTest with SBT),
the Automatic Configuration method is unlikely to work. The problem is that
if an instance of LiftRules
is not created before the first Logger
is
created, LoggingAutoConfigurer
will not be assigned to Logger.setup
before
setup is completed (and therefore it is never executed).
All of the testing frameworks provide a mechanism for running code
before/after tests (or suites of tests). It’s quite possible to either create
an instance of LiftRules
, access a LiftRules
method on the LiftRules
object (which has an implicit conversion to LiftRules
), or assign
Logger.setup
directly through this mechanism. However, this requires the
most work and is therefore the worst solution (in my opinion).
Be warned, this method is very fragile. Because the order in which tests are
run is not deterministic, if any test creates a Logger
without performing
logging setup, it will prevent future configuration and cause all other tests
to log all messages in the default configuration.
In SBT, it is also possible to run code before any tests run by using
Tests.Setup and
Tests.Cleanup
in the testOptions
setting. This method is a bit awkward, since SBT project
code does not have access to the Lift classes directly (without adding a
dependency to the project code), so everything must be done via reflection.
To pass LoggingAutoConfigurer
to Logger.setup
, add the following to
build.sbt
(Note that blank lines would confuse the .sbt
parser, but would
be allowed in a .scala
file):
testOptions += Tests.Setup { loader: ClassLoader =>
// Get Logger.setup
val boxClass = loader.loadClass("net.liftweb.common.Box")
val loggerClass = loader.loadClass("net.liftweb.common.Logger$")
val logger = loggerClass.getField("MODULE$").get(null)
val loggerSetupEq = loggerClass.getMethod("setup_$eq", boxClass)
// Get function from LoggingAutoConfigurer.apply()
val configurerClass = loader.loadClass("net.liftweb.util.LoggingAutoConfigurer$")
val configurer = configurerClass.getField("MODULE$").get(null)
val configFunc = configurerClass.getMethod("apply").invoke(configurer)
// Put it in a Box
val fullClass = loader.loadClass("net.liftweb.common.Full")
val fullConstructor = fullClass.getConstructor(classOf[Object])
val configFuncBox = fullConstructor.newInstance(configFunc)
// Call Logger.setup on the Box
loggerSetupEq.invoke(logger, configFuncBox.asInstanceOf[Object])
}
If Logger.setup
has not been assigned, the logging backend will not be
configured by Lift. This does not mean that the logging backend will not be
configured at all. Conveniently, both Logback and Log4J, in their default
configurations, will search for configuration files on the classpath. Logback
will use logback-test.xml
or
logback.xml
and Log4J will
use log4j.properties
or the value of the log4j.configuration
system
property. Using
this fact, it is possible to configure Logback by creating
src/test/resources/logback.xml
(and similar for Log4J). This is by far the
easiest solution, if you don’t mind the asymmetry of the configuration file
locations.
It is possible to configure logging in Lift during tests using any of the above methods. My recommendation is to use either of the last two methods, based on whichever is more suitable for a given project.
In any case, be aware that in the default SBT configuration (fork := false
),
the JVM is shared between not only all tests, but all tasks invoked from SBT.
For this reason, re-running the test
task without exiting SBT will not
re-initialize logging. Also, running multiple tasks (such as test
and
run
) will break Lift, since the mode is determined only once.
Just a quick reminder to always flush your buffers (when appropriate) and that
the behavior of the JDK default XMLStreamWriter
(com.sun.xml.internal.stream.writers.XMLStreamWriterImpl
) differs between
UTF-8 output, which is unbuffered, and non-UTF-8 output, which is buffered
through com.sun.xml.internal.stream.writers.XMLWriter
. I just spent way too
much time figuring this out (particularly because finding the actual location
of the source file is non-trivial - Hint: It’s not in the OpenJDK source
tree). Hopefully this post will save others that time/effort.
Ruby software is commonly distributed as “gems”, packages containing Ruby
applications and/or libraries, which can be installed using the
RubyGems package manager, typically run as a command
named gem
. On Debian systems, some gems are also available as Debian
packages through the Debian package repositories. For Ruby developers on
Debian, it is almost inevitable that some gems will be installed through
RubyGems and some will be installed through the Debian package managers (and
possibly some installed through both). This post discusses some tips for
minimizing the pain of this situation.
Although the tips in this post have only been tested on Debian, they should be applicable to Debian-based distributions (such as Ubuntu and its derivatives) with only minimal changes, if any.
First, a quick digression: Why bother using both RubyGems and Debian packages? Why not choose one and stick with it? Half of the reason is simple, Debian packages only exist for a small subset of available gems. The other half is not as simple, and indeed it would be possible to avoid installing Debian packages for gems. However, using the Debian packages for gems carries the advantages of using any other Debian package: Integration-testing, compliance with Debian Policy (including the FHS), security support from the Debian Security Team, management using the Debian package managers, bug-tracking through bugs.debian.org, etc.
For the above reasons, and my personal biases, I’ll assume that if there is a Debian package available for a gem (and for the RubyGems package manager itself) that the Debian package will be used. Although, if a newer version of a gem than the one available as a Debian package is required, it’s easy enough to install that version using RubyGems.
In the default installation, RubyGems is not aware of any gems which have been
installed through the Debian package managers. Although this doesn’t cause
any serious problems, since the installation locations are different, it may
result in duplicate copies of gems being installed, causing confusion and
wasting space and bandwidth. The solution to this problem is very simple,
ensure that RubyGems is installed from a Debian package (either the
rubygems package for Ruby 1.8 or
by using the Debian package for Ruby 1.9, which includes RubyGems), then
install the
rubygems-integration
package through a Debian package manager (e.g. apt-get install
rubygems-integration
).
Once this is done, RubyGems will use already-installed Debian package gems to
satisfy dependencies and won’t install a second copy of such gems. Note
however that this will not, as far as I am aware, cause gem
to install a
Debian package to satisfy a dependency if one is available. The user must
first install the Debian package, if one is available, for it to be used.
As an administrator, I strongly prefer to keep all non-packaged software in /usr/local (or user home directories for user-specific software). This provides a clear distinction between software/files which I have to keep an eye on (to maintain, update, clean, remove, and provide security support) and those which I can defer to their Debian maintainers. However, there are valid reasons for keeping the default installation location, some of which are discussed in bug 448639, and users who find these more compelling may freely skip this section.
There are a few ways to change the installation location, called the “gem path” in the RubyGems documentation, although none of them are without drawbacks. Unfortunately, there is no supported way to change the system-wide default RubyGems installation location, that I am aware of.
The first option is simply to include
--install-dir=/usr/local/lib/ruby/gems/1.9
whenever gem
is invoked. This
could be accomplished with a shell alias, a wrapper script, or a very large
amount of discipline (for single-user systems).
GEM_PATH
Another option is to set the GEM_PATH
environment variable. This could be
set system-wide in /etc/environment
(or /etc/profile
or /etc/bash.bashrc
for bash) or in user logon scripts.
The most fool-proof option, and the one with the highest maintenance burden, is to edit the RubyGems source to change the default install location. This is relatively simple, although it does require re-editing the source after each upgrade and comes with some risks for users not familiar with Ruby.
To change the system-wide default installation directory (if the
rubygems-integration
package is installed), edit
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:4
as
follows:
def default_dir
# File.join('/', 'var', 'lib', 'gems', Gem::ConfigMap[:ruby_version])
File.join('/', 'usr', 'local', 'lib', 'ruby', 'gems', Gem::ConfigMap[:ruby_version])
end
If the rubygems-integration
package is not installed, edit
/usr/lib/ruby/1.9.1/rubygems/defaults.rb
and change the following lines at
the end of self.default_dir
as follows:
# @default_dir ||= File.join(*path)
@default_dir ||= File.join('/', 'usr', 'local', 'lib', 'ruby', 'gems', ConfigMap[:ruby_version])
With those edits made, the default installation directory will be
/usr/local/lib/ruby/gems/1.9/
. Remember to re-edit the files after an
upgrading rubygems-integration
(or libruby1.9.1
if rubygems-integration
is not installed). As a reminder, it may be useful to place a hold on the
package in the Debian package management system to require explicitly
upgrading the package (e.g. aptitude hold rubygems-integration
).
That’s it. Those minor issues aside, everything seems to work quite well. Enjoy writing Ruby on Debian!
Just a quick note: I wanted to work with fonts from Google Web Fonts offline. Unfortunately, the source code only contains TTF files and I was unable to find either a trusted converter utility for all formats or a way to download non-TTF formats. So, I wrote a simple utility to download the fonts. Perhaps you will find it useful.
Recent versions of Lift (2.2-M1 and later) provide a concise way of expressing XML transformations using a CSS-like syntax called CSS Selector Transforms. The pleasant conciseness comes with a number of unexpected/undocumented behaviors and corner-cases. One which recently caught me by surprise is the handling of attributes on XML elements. This post is a brief discussion of the behavior and how to work around it to remove attributes from elements.
Although I can not find any reference to attribute merging in the documentation, it has been discussed on the mailing list. The basic idea is that when an element is transformed, any attributes from the original element are copied to the replacement element, except where the replacement element contains an attribute with the same name (except when that attribute is “class”, in which case they are merged). This simplifies the common case, where attributes should be retained, while making the uncommon case significantly more complex.
For all of the following examples, suppose we have the following HTML stored
in a variable named html
:
<p id="notice1" class="admonition important">Be careful with CSS Selector Transforms!</p>
First, as an example of attribute merging, suppose we wanted to replace the element with another using the following code:
("#notice1" #> <p id="notice2" class="alert">Breaking News: CSS Selectors!</p>)(html)
This would produce:
<p id="notice2" class="alert admonition important">Breaking News: CSS Selectors!</p>
Notice that the id attribute has been replaced while the class attribute has been merged. With this in mind, it should be obvious why the following code does not work:
// WARNING: Does nothing!
("#notice1" #> { n =>
val e = n.asInstanceOf[Elem];
e.copy(attributes = e.attributes.remove("class"))
})(html)
Although the element returned by the transformation function doesn’t have a class attribute, attribute merging will add the class attribute from the original element causing the above transformation to have no effect (except possibly changing the attribute order).
The correct way to modify attributes is by matching against the attribute (a Lift addition to the CSS syntax) as follows:
("#notice1 [id]" #> (None: Option[String]))(html)
This would (as expected) result in:
<p class="admonition important">Breaking News: CSS Selectors!</p>
Although, interestingly, replacing None
with Nil
will result in an empty
id attribute. I haven’t fully investigated this behavior, although part of
the explanation is that None
is implicitly converted to
net.liftweb.util.IterableConst
while Nil
is implicitly converted to
scala.xml.NodeSeq
.
In 2.4-M4, there is even an addition to the syntax to allow removing a space-separated word from an attribute:
("#notice1 [class!]" #> "important")(html)
Which would result in:
<p id="notice1" class="admonition">Breaking News: CSS Selectors!</p>
Neat, huh?
Not so fast! There’s an annoying bug which prevents this from working when combined with other CSS Selector Transforms either as child transforms (as appears in the bug report) or when combined. So if we add an identity transformation function to the previous transformation as follows:
// WARNING: Does nothing!
("#notice1" #> { n => n } & "#notice1 [class!]" #> "important")(html)
The output is the same as the input (again, ignoring any attribute ordering). After a bit of digging, I found that if the attribute-modifying transforms are chained rather than combined they behave as expected. So the previous (non-functional) transformation can be changed to:
("#notice1" #> { n => n } andThen "#notice1 [class!]" #> "important")(html)
Which does behave as expected.
In Lift 2.5-M1 and later there is an addition to the CSS Selector Transform
syntax which disables attribute
merging.
Appending "!!"
to the outermost selector will disable attribute merging,
which allows our original example (or any other nested selectors) to modify
attributes at will:
// WARNING: Won't compile in 2.5-M1 or later
("#notice1 !!" #> { n => n } & "#notice1 [class!]" #> "important")(html)
Whoops! That doesn’t work, for 2 reasons. First, n
needs a declared type.
Second, the design of the new CSS Type Classes in 2.5 mean that the html
parameter is interpreted as the implicit ComputeTransformRules
parameter of
the #>
method of ToCssBindPromoter
rather than the parameter of the
apply
method of CssSel
. These problems can be fixed as follows:
// Works in 2.5-M1 and later
("#notice1 !!" #> { n: NodeSeq => n } & "#notice1 [class!]" #> "important").apply(html)
However, this syntax does not work if all of the classes are removed:
// WARNING: Doesn't work in 2.5-M1 (won't remove either class)
("#notice1 !!" #> { n: NodeSeq => n } & "#notice1 [class!]" #> List("admonition", "important")).apply(html)
// WARNING: Also doesn't work (for the same reason)
("#notice1 !!" #> { n: NodeSeq =>
val e = n.asInstanceOf[Elem];
e.copy(attributes = e.attributes.remove("class"))
}).apply(html)
"!!"
syntax.I’ve recently started using the Dispatch library for HTTP/HTTPS, which is quite a nice library, as long as you don’t need documentation. Dispatch uses the Ning/Sonatype AsyncHttpClient library, which is also quite nice, and although AsyncHttpClient is a library which I could recommend, it does have an insecure-by-default implementation of SSL. This post is a quick discussion of the AsyncHttpClient defaults and how to implement certificate verification to increase the security provided by SSL.
The information in this post is outdated. Thanks to the efforts of the Async
Http Client team, hostname validation was enabled by default in commit
3c9152e
from pull request
#510, which is
included in 2.0.0-alpha9 and later. The fix was also backported to
1.9.0-BETA1 in commit
a894583.
If you are using Async Http Client 1.9.0 or later, there is no need to use the
MyHostnameVerifier
class described in this post.
This post will assume some familiarity with SSL and the need to verify certificates. If readers are unfamiliar with either of these topics, there are many online resources available and you are encouraged to explore the topic.
For background on how SSL is implemented on the Java platform, see the Java Secure Socket Extension (JSSE) Reference Guide. Readers who are not concerned with the implementation details may feel free to skip to the end of this article for “recipes” for SSL certificate verification.
The
SSLContext
class is central to the SSL implementation in Java in general and in
AsyncHttpClient in particular. The default SSLContext
for AsyncHttpClient
is dependent on whether the javax.net.ssl.keyStore
system property is set.
If this property is set, AsyncHttpClient will create a TLS SSLContext
with a
KeyManager
based on the specified key store (and configured based on the
values of many other javax.net.ssl
properties as described in the JSEE
Reference Guide linked above). Otherwise, it will create a TLS SSLContext
with no KeyManager
and a TrustManager
which accepts everything. In
effect, if javax.net.ssl.keyStore
is unspecified, any ol’ SSL certificate
will do.
If the trusted Certificate Authorities for the application should be the same
as the trusted CAs for the operating system, it is possible to avoid the
hassles of dealing with Java key stores by using the (JRE) default
SSLContext
. Simply instantiate a new SSLContext
and initialize it with
all null
values. This offloads the burden to the JRE provider and OS vendor
and works like a charm on my test system.
Unfortunately, there does not appear to be a way to set the default
SSLContext
used by AsyncHttpClient. Instead, applications must set their
preferred SSLContext
for each connection.
Even if the SSLContext
can verify that a certificate is signed by a trusted
Certificate Authority, there is still room for problems. What happens if the
connection hostname doesn’t match the certificate hostname? Java provides the
HostnameVerifier
interface to give client code the option of providing a policy for handling
this situations. AsyncHttpClient adopts this interface for this purpose as
well. However, unlike the JDK, the default policy provided by AsyncHttpClient
is to allow all connections regardless of hostname.
Unlike SSLContext
, using the Java default
(HttpsURLConnection.getDefaultHostnameVerifier
)
is not a viable option because the default HostnameVerifier
expects to only
be called in the case that there is a mismatch (and therefore always returns
false
) while some of the AsyncHttpClient providers (e.g. Netty, the default)
call it on all
connections. To
make matters worse, the check is not trivial (consider SAN and wildcard matching) and is implemented in
sun.security.util.HostnameChecker
(a Sun internal proprietary API). This leaves the developer in the position
of either depending on an internal API or finding/copying/creating another
implementation of this functionality. For the examples in this article, I
have opted for the first option.
Unfortunately, as with SSLContext
, there does not appear to be a way to set
the default HostnameVerifier
used by AsyncHttpClient. Instead, applications
must set their preferred HostnameVerifier
for each connection.
First, a quick note: The purpose of these example implementations is to demonstrate how to verify certificates. The programs should include better exception handling, logging, and a more modular functional decomposition, but this would lengthen the examples and obscure their core purpose. Please feel free to use your better judgement when copying and expanding on these examples.
First, an example program which downloads a given URL using Dispatch in Scala:
/* An example program using Dispatch with SSL certificate verification
*
* To the extent possible under law, Kevin Locke has waived all copyright and
* related or neighboring rights to this work.
*/
import com.ning.http.client.{AsyncHttpClient, AsyncHttpClientConfig}
import dispatch._
import java.security.cert.{CertificateException, X509Certificate}
import javax.net.ssl.{HostnameVerifier, SSLPeerUnverifiedException, SSLSession}
import javax.security.auth.kerberos.KerberosPrincipal
import sun.security.util.HostnameChecker
/** HostnameVerifier implementation which implements the same policy as the
* Java built-in pre-HostnameVerifier policy.
*/
object MyHostnameVerifier extends HostnameVerifier {
/** Checks if a given hostname matches the certificate or principal of a
* given session.
*/
private def hostnameMatches(hostname: String, session: SSLSession): Boolean = {
val checker = HostnameChecker.getInstance(HostnameChecker.TYPE_TLS);
try {
session.getPeerCertificates match {
case Array(cert: X509Certificate, _*) =>
try {
checker.`match`(hostname, cert)
// Certificate matches hostname
true
} catch {
case _: CertificateException =>
// Certificate does not match hostname
false
}
case _ =>
// Peer does not have any certificates or they aren't X.509
false
}
} catch {
case _: SSLPeerUnverifiedException =>
// Not using certificates for verification, try verifying the principal
try {
session.getPeerPrincipal match {
case principal: KerberosPrincipal =>
HostnameChecker.`match`(hostname, principal)
case _ =>
// Can't verify principal, not Kerberos
false
}
} catch {
case _: SSLPeerUnverifiedException =>
// Can't verify principal, no principal
false
}
}
}
def verify(hostname: String, session: SSLSession): Boolean = {
if (hostnameMatches(hostname, session)) {
true
} else {
// TODO: Add application-specific checks for hostname/certificate match
false
}
}
}
/** Extension of Http which uses an AsyncHttpClient configured with our
* customized SSLContext and HostnameVerifier
*/
object MyHttp extends Http {
override lazy val client = new AsyncHttpClient(
new AsyncHttpClientConfig.Builder()
.setSSLContext({
val ctx = javax.net.ssl.SSLContext.getInstance("TLS")
ctx.init(null, null, null)
ctx
})
.setHostnameVerifier(MyHostnameVerifier)
.build
)
}
/** Implements the "MyDownloader" application */
object MyDownloader {
def main(args: Array[String]) {
args match {
case Array(url) =>
val request = dispatch.url(url)
Console.err.println("Downloading " + url)
MyHttp(request OK as.String).either() match {
case Left(e) =>
// Something failed
Console.err.println("Failure downloading " + url + ": " + e)
case Right(content) =>
// Success
Console.err.println("Successfully downloaded " + url)
Console.out.println(content)
}
MyHttp.shutdown()
case _ =>
Console.err.println("Usage: myhttp <URL>")
System.exit(1)
}
}
}
The above code is also available as part of a GitHub Gist.
Then, the same program using AsyncHttpClient directly from Java:
/* An example program using AsyncHttpClient with SSL certificate verification
*
* To the extent possible under law, Kevin Locke has waived all copyright and
* related or neighboring rights to this work.
* A legal description of this waiver is available in LICENSE.txt.
*/
import com.ning.http.client.AsyncHttpClient;
import com.ning.http.client.AsyncHttpClientConfig;
import com.ning.http.client.Response;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.Principal;
import java.security.cert.Certificate;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.concurrent.ExecutionException;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLPeerUnverifiedException;
import javax.net.ssl.SSLSession;
import javax.security.auth.kerberos.KerberosPrincipal;
import sun.security.util.HostnameChecker;
/** Implements the "MyDownloader" application */
public class MyDownloader {
/** HostnameVerifier implementation which implements the same policy as the
* Java built-in pre-HostnameVerifier policy.
*/
private static class MyHostnameVerifier implements HostnameVerifier {
/** Checks if a given hostname matches the certificate or principal of
* a given session.
*/
private boolean hostnameMatches(String hostname, SSLSession session) {
HostnameChecker checker =
HostnameChecker.getInstance(HostnameChecker.TYPE_TLS);
boolean validCertificate = false, validPrincipal = false;
try {
Certificate[] peerCertificates = session.getPeerCertificates();
if (peerCertificates.length > 0 &&
peerCertificates[0] instanceof X509Certificate) {
X509Certificate peerCertificate =
(X509Certificate)peerCertificates[0];
try {
checker.match(hostname, peerCertificate);
// Certificate matches hostname
validCertificate = true;
} catch (CertificateException ex) {
// Certificate does not match hostname
}
} else {
// Peer does not have any certificates or they aren't X.509
}
} catch (SSLPeerUnverifiedException ex) {
// Not using certificates for peers, try verifying the principal
try {
Principal peerPrincipal = session.getPeerPrincipal();
if (peerPrincipal instanceof KerberosPrincipal) {
validPrincipal = HostnameChecker.match(hostname,
(KerberosPrincipal)peerPrincipal);
} else {
// Can't verify principal, not Kerberos
}
} catch (SSLPeerUnverifiedException ex2) {
// Can't verify principal, no principal
}
}
return validCertificate || validPrincipal;
}
public boolean verify(String hostname, SSLSession session) {
if (hostnameMatches(hostname, session)) {
return true;
} else {
// TODO: Add application-specific checks for
// hostname/certificate match
return false;
}
}
}
public static void main(String[] args) {
if (args.length != 1) {
System.err.println("Usage: myhttp <URL>");
} else {
String url = args[0];
SSLContext context = null;
try {
context = SSLContext.getInstance("TLS");
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
return;
}
try {
context.init(null, null, null);
} catch (KeyManagementException e) {
e.printStackTrace();
return;
}
AsyncHttpClient client = new AsyncHttpClient(
new AsyncHttpClientConfig.Builder()
.setSSLContext(context)
.setHostnameVerifier(new MyHostnameVerifier())
.build()
);
Response response = null;
try {
response = client.prepareGet(url).execute().get();
} catch (InterruptedException e) {
e.printStackTrace();
return;
} catch (ExecutionException e) {
e.printStackTrace();
return;
} catch (IOException e) {
e.printStackTrace();
return;
}
if (response.getStatusCode() / 100 == 2) {
try {
String responseBody = response.getResponseBody();
System.err.println("Successfully downloaded " + url);
System.out.println(responseBody);
} catch (IOException e) {
e.printStackTrace();
return;
}
} else {
System.err.println("Failure downloading " + url +
": HTTP Status " + response.getStatusCode());
}
}
}
}
The above code is also available as part of a GitHub Gist.
Although the default initialization of SSLContext
works quite well on my
test machine, I have not found a clear specification of its behavior and it
may not be guaranteed to use/trust the operating system certificate store on
all platforms.
In my testing I found that if the invalid certificate is returned by a process
on the local machine (e.g. using mitmproxy)
AsyncHttpClient will throw a java.io.IOException: Remotely Closed
rather
than java.net.ConnectException: General SSLEngine problem
. This is probably
a bug. In either
case, users should be wary of this behavior when troubleshooting a failing
SSL/TLS connection.
I recently started using SLICK (formerly ScalaQuery) for database access in a Scala project. In the process of wrapping my head around how SLICK, I’m documenting some “recipes” for common queries. Unfortunately, this post got published before the recipes were ready. (Oops!) But, not to worry, I’ll post them here (or a link from here) once they are ready.
I recently had to make the difficult choice to replace Squeryl with SLICK (formerly ScalaQuery) much later in the development cycle than I would have liked. Although I do like some of the design and features of Squeryl, it has some very significant limitations that anyone considering using it should be aware of up-front. Also, in an effort to avoid excessive bias, I’ll include a few of the limitations of SLICK that I have encountered for comparison.
Also, my apologies in advance if this is a bit of a diatribe. I don’t mean to belittle the impressive amount of work that has been done on Squeryl at no cost to the users. I am simply frustrated that I did not find out about these limitations sooner, which is more my fault than anyone else’s.
First, a quick note that the limitations on the Squeryl website are not mentioned here. Lack of database-specific features is also not mentioned (and not generally expected). Also, any example queries are written using the SchoolDb schema. Now, with that out of the way, on to the limitations:
The biggest obstacle to using Squeryl effectively, in my opinion, is the compiler errors which are produced when there is a syntax error in the SQL DSL or a compile-time usage error (e.g. type error). I spent a significant amount of time tracking down a missing comma, incorrectly ordered clauses, type errors, and a number of other simple errors. Troubleshooting isn’t always easy, but the amount of the Squeryl implementation details which leak out into error messages necessitates learning a lot about the Squeryl internals to diagnose simple errors.
For example, consider the following query:
join(students, courseSubscriptions, courses)((s,cs,c) =>
select(s.name, c.name)
on(s.id === cs.studentId and c.id === cs.courseId)
)
Attempting to compile this results in the following error:
[error] Example.scala:32: type mismatch;
[error] found : org.squeryl.dsl.boilerplate.JoinQueryYield1[(String, String)]
[error] required: org.squeryl.dsl.boilerplate.JoinQueryYield2[?]
[error] on(s.id === cs.studentId and c.id === cs.courseId)
[error] ^
[error] one error found
The problem is that the on
method requires one argument for each table after
the first. This makes good sense, but it isn’t immediately clear from the
error message (although the number in the class name does provide a hint).
For another example (I know this query is sub-optimal, but it is meant to parallel the first example):
join(students, courseSubscriptions, courses)((s,cs,c) =>
compute(count(c.name))
groupBy(s.name)
on(s.id === cs.studentId, c.id === cs.courseId)
)
Which results in:
[error] Example.scala:32: value groupBy is not a member of org.squeryl.dsl.fsm.ComputeStateStartOrWhereState[org.squeryl.PrimitiveTypeMode.LongType]
[error] possible cause: maybe a semicolon is missing before `value groupBy'?
[error] groupBy(s.name)
[error] ^
[error] one error found
The error here is that compute
should follow groupBy
(and both should
precede on
). Also, as an exercise for the reader, try putting on
first.
I ran into issues properly typing expressions which are non-NULL (or where
NULLness is irrelevant) where the referenced column may be NULL (e.g. SELECT
column FROM table WHERE column IS NOT NULL
). Although you can call .get
on
the Option type during AST construction to get the correct non-Option type,
there are cases where this results in a NoSuchElementException
. I need to
dig through some version history to figure out which cases those are. For now,
feel free to take this limitation with a grain of salt.
To be clear, it is possible to insert, delete, and update single rows through
Squeryl using concrete values. What is not possible is to write DML queries
which may modify multiple rows (e.g. DELETE FROM table WHERE column < 5
) or
to write DML queries which use the result of a SELECT statement (e.g. INSERT
INTO table2 SELECT * FROM table1
). These queries must be done through JDBC
directly.
It’s possible that DML queries which make use of the result of a SELECT statement could be built by concatenating the SQL statement generated by Squeryl to the end of the query text. However, I am not sure if the column order is guaranteed to match the variable order, so I would approach this idea with caution.
I have been unable to find any mechanism for calling stored procedures through Squeryl. I assume it does not exist and that one must use JDBC directly (see the next limitation for a discussion of this).
This limitation was the real show-stopper for me. Although it is possible to use JDBC directly, Squeryl does not provide any mechanism that I could find which would convert the JDBC results into user datatypes, run the queries in the current session, enumerate results, or provide any of the Squeryl functionality to these ad-hoc queries.
Every library has its limitations, but whether, and how, the developer is able
to work around those limitations is critically important. When Squeryl
doesn’t support a required feature, whether that is a database-specific
feature or one of the general limitations mentioned above, it provides no
support or convenience. The developer must learn JDBC, implement
unmarshalling of result types for each result type, implement enumeration of
the result to Scala data types, and all of the other tedious details of
dealing directly with JDBC. Also, the developer must be aware of the many
pitfalls of dealing with SQL directly, particularly things like using
PreparedStatement
to avoid SQL-injection, which are so commonly done
incorrectly. No thanks!
Although I have found SLICK to be preferable to Squeryl, it is not without its own limitations.
This limit is the result of the choice of modeling tables and results as Scala
Tuple
s (which are limited to 22 elements). Although it should be possible to
link with a library which provides implementations for larger tuples, I have
not tried it.
Support for retrieving ID values for auto-incremented columns was added as a result of issue #10 in commit 09a65a8, which is in version 0.11.1 and later. This may still be important in light of the next limitation.
Unfortunately, there is no SLICK for Scala 2.9. Anyone still using Scala 2.9 will be stuck using ScalaQuery 0.10.0-M1. This is particularly frustrating because Lift is unlikely to support Scala 2.10 until it is released.
This isn’t really a limitation of SLICK, but is important for users considering whether to use SLICK with Lift. Lift Record (one of the two data-access mechanisms in Lift) does not currently support SLICK. This can be a significant limitation as many of the forms features (form construction and validation in particular) either require, or are significantly easier when using, Record.
The SQL CASE expression has two variants, often described as the “searched”
and the “simple” CASE expressions. The “searched” CASE, which behaves like
the switch
statement of many programming languages, compares each choice for
equality with the argument value and results in the value of the expression
associated with the first match. The “simple” CASE, which behaves like a
sequence of if
tests, evaluates each expression and results in the value
associated with the first expression which evaluates to true.
Although the behavior of CASE expressions in SELECT clauses of top-level queries can be easily reproduced by post-processing the result, the real value of using CASE in SQL is in sub-queries, HAVING clauses, and particularly in GROUP BY clauses.
Unfortunately, Squeryl does not support either CASE expression in any context. Support was added in 54500b90, then removed in 8784db07 and is still missing. SLICK does not appear to have, or have had, any support for CASE statements in any context.
After switching from Squeryl to SLICK, I have never looked back. I have found the design of SLICK to be exceptional and although it is possible that I may find show-stopping issues in the future, it seems unlikely. It has already taken me further than Squeryl. I have managed to re-implement everything previously done using Squeryl and a significant number of additional tasks which would not have been easily possible (requiring ad-hoc queries).
I hope this post helps clarify some of the issues to consider when choosing a Scala ORM.
Although it does not appear to be officially documented, it is possible to subscribe to a Google Group without a Google Account. There are several ways to subscribe but, as I recently found out, Google Groups tries really hard to use a GMail account, if you have one. This post explains how to subscribe to a Google Group via email and how to avoid one pitfall that may result in messages being sent to your GMail address rather than the address with which you subscribed.
Recently I have been tinkering around with the Lift web framework and decided to subscribe to their mailing list, which happens to be hosted on Google Groups. After a quick search I found several sets of instructions for subscribing to a Google Group without a Google Account. Interestingly, many of the suggestions I found link to articles on Google Groups Help, none of which actually contain the by-email instructions. It is possible this was officially supported in the past and is now deprecated, but it continues to work nonetheless.
Undaunted, I followed the instructions and sent an email to liftweb+subscribe@googlegroups.com, then replied to the confirmation email it sent in reply. I received a “welcome to the group” message and everything looked good. Piece of cake, just like the good ol’ days. Then the list mail started flowing in to my GMail address. What!?
That’s odd, I thought. So I double-checked my sent folder (in Mutt) and confirmed that all messages were sent and received from my personal email address on this domain. I checked the server logs (did I mention that I host my own email?), all messages were sent through my mail server with the envelope sender set to my personal email address. How did Google Groups even get my GMail address?
A short bit more thought and I realized that my personal address was probably set as the backup/alternate email address for GMail, in case of the need to do a password reset. I checked, and indeed it was. That explains how Google associated the two accounts.
The easiest method that I could find to prevent Google Groups from sending emails to a GMail account is to make sure that the email address which is used to subscribe is not associated with any GMail account. If it is the backup/alternate address to a GMail account, remove it, then subscribe, then re-add it. That’s the best I could find.
If anyone can find an easier way to prevent this behavior, or a even a good explanation for why this Google Groups behavior is beneficial to anyone but Google, I’d love to hear it (and post it as a follow-up). Otherwise, good luck subscribing to a Google Group with your preferred email address.
The latest issue that I’ve encountered while working with Squeryl in a Lift-based web application, is that not all transactions are being committed to the database. This post is a quick discussion of the symptoms that I was seeing and a note on how to avoid the issue.
As noted before, the setup that I am using is quite generic and based on the example configurations on the Squeryl-Record wiki page. The only portion which is relevant to this article is how transactions are handled. For now, I’m using a simple transaction-per-request strategy, implemented as follows:
S.addAround(new LoanWrapper {
override def apply[T](f: => T): T = inTransaction { f }
})
Although this code fragment appears in most of the examples on the web, it has at least one significant flaw.
The flaw with the above code fragment is that it does not handle Lift’s flow control exceptions properly. In retrospect, the behavior is very clear and easy to spot (but was not quite so obvious at the time): The transactions failed to commit to the database whenever the request which caused the database updates resulted in an HTTP redirect response. The mechanics for why this happens are equally clear in retrospect.
Squeryl’s inTransaction
method assumes that the transaction should be aborted
if the code that it is executing throws an exception and it will rollback the
transaction when this happens. However, Lift implements partial or
short-circuited responses (e.g. redirects) by throwing a
LiftFlowOfControlException
. Therefore, whenever a response is redirected
after database changes are made, those changes will be rolled back.
The solution that I am using is reasonably simple. Replace the above code fragment with:
S.addAround(new LoanWrapper {
override def apply[T](f: => T): T = {
val resultOrExcept = inTransaction {
try {
Right(f)
} catch {
case e: LiftFlowOfControlException => Left(e)
}
}
resultOrExcept match {
case Right(result) => result
case Left(except) => throw except
}
}
})
This way whenever a LiftFlowOfControlException
is thrown it will be returned
through inTransaction and the transaction will commit while any other exception
propagates out as before causing transaction rollback. Then, if an exception
was returned, it is re-thrown from the LoanWrapper
to continue propagating
up to the Lift internals.
With this in place, transactions should commit and Lift’s control flow exceptions should continue to work as expected. At least, I hope so…
Either
to follow the standard convention that Left
is failure and Right
is success (as documented in the scala.util.Either
scaladoc).In tracking down some persistent errors relating to using Squeryl with Lift, I’ve found that the latest version of BoneCP does not appear to be safe for use in this scenario. This article is a quick discussion of the symptoms that I am seeing and how to avoid them.
The setup that I am using is quite generic and based on the example configurations on the Squeryl-Record wiki page. Basically, I’m using the following code in my Lift Boot class:
Props.get("db.driver") map { Class.forName(_) }
val config = new BoneCPConfig()
config.setCloseConnectionWatch(true)
Props.get("db.jdbcUrl") map { config.setJdbcUrl(_) }
Props.get("db.username") map { config.setUsername(_) }
Props.get("db.password") map { config.setPassword(_) }
val connPool = new BoneCP(config)
val adapterName = Props.get("db.squerylAdapter").open_!
var adapterClass = Class.forName(adapterName).asInstanceOf[Class[DatabaseAdapter]]
// Note: Passed by name, called repeatedly
net.liftweb.squerylrecord.SquerylRecord.initWithSquerylSession(
Session.create(connPool.getConnection, adapterClass.newInstance)
)
S.addAround(new LoanWrapper {
override def apply[T](f: => T): T = inTransaction { f }
})
With the above code in place (and some pages which make read queries), I would see the following log messages:
[com.google.common.base.internal.Finalizer] WARN c.jolbox.bonecp.ConnectionPartition - BoneCP detected an unclosed connection and will now attempt to close it for you. You should be closing this connection in your application - enable connectionWatch for additional debugging assistance.
Which would be followed (on the next page access) by:
\[qtp3266018-33 - /path\] ERROR net.liftweb.http.LiftRules - Exception being returned to browser when processing /path: Message: org.postgresql.util.PSQLException: This connection has been closed.
So, as suggested in the warning message, I tried enabling connectionWatch, which had no effect. It appears that this is only helpful if a thread holding an open connection dies.
Next I created a proxy for Connection to observe creation/close and it appears that all connections (technically, connection handles, since BoneCP manages the real connections) are being closed.
At this point I started to suspect an issue in BoneCP, so I tried downgrading to 0.7.1.RELEASE and all of the problems vanished. Hopefully this solution will work for others in the same situation as well as it did for me.
Although the ffmpeg (and avconv) program has a relatively intuitive command-line interface, given the diversity and complexity of the functionality that it exposes, there are still many operations which can be difficult to express. I found letterboxing (and pillarboxing) to be one of those operations, so in order to save others the trouble of working out the details, this post will develop a command for doing boxing with ffmpeg/avconv.
For the unfamiliar, ffmpeg is both a command and the name of the project (more properly written FFmpeg) which developed the ffmpeg command as well as a significant amount of other multimedia software. The avconv program is a fork of ffmpeg by the Libav project. The relationship between the two projects is a bit complex (see this StackOverflow question and linked pages for some details), but all commands in this post should work with either ffmpeg or avconv. Feel free to use whichever you prefer.
The process that this post is attempting to simplify can be either letterboxing, adding horizontal bars to an image, or pillarboxing, adding vertical bars to an image. From this point forward, either process will simply be referred to as “boxing”.
Why would someone want to box video? My particular motivation is to convert video for use on mobile devices, which often require video to have particular resolutions in order to take advantage of hardware acceleration. When the aspect ratio of the video does not match the ratio of the desired resolution it’s necessary to either box or stretch the video. So, I’m boxing it. More generally, any time the aspect of the source video does not match the aspect of the desired output, boxing may be necessary.
Ffmpeg/avconv supports a number of different filters for performing video manipulation along with a formula evaluation syntax for configuring them. This allows users to write arithmetic formulas using predefined constants to specify the behavior of the filters based on values that may change based on the multimedia input.
Boxing the video will require both the scale
and pad
filters, scale
to
fit the video into the target resolution and pad
to add the bars. First,
scaling the video requires calculating the output resolution of the video
without any boxing. The most desirable output resolution in this case is one
which preserves the source aspect ratio (so the video is not stretched) and
fills the screen either horizontally or vertically. This can be done by
calculating a scale factor and applying it equally, as follows:
scale=iw*min($MAX_WIDTH/iw\,$MAX_HEIGHT/ih):ih*min($MAX_WIDTH/iw\,$MAX_HEIGHT/ih)
Note that $MAX_WIDTH
and $MAX_HEIGHT
should be replaced with the desired
output width and height, or set as variables in a shell script. Also note
that the scale factor (the min
function) is calculated twice. It could
probably be stored in a variable and reused, but I am not sure of the correct
syntax. Finally, note that the backslash before the commas is required
because commas are used to separate filters and we are using it to separate
function arguments in this case.
Now that the video has been scaled to fit the desired output resolution, the
pad
filter can be used to add the appropriate bars. pad
requires the
output width and height as well as the offset for where the input should be
placed within the defined output and, optionally, a color. To add equal-sized
pads simply requires that the offset is half of the size difference between
the output and the input in each dimension, as follows:
pad=$MAX_WIDTH:$MAX_HEIGHT:(ow-iw)/2:(oh-ih)/2
Again, $MAX_WIDTH
and $MAX_HEIGHT
should be replaced with the desired
output width and height, or set as variables in a shell script.
It’s possible that the input video is intended to be displayed at a resolution which has a different aspect ratio than the source file, called an anamorphic format. I have not had success playing anamorphic video on mobile devices (probably in part because I don’t really understand it and in part because it is rather esoteric and poorly supported), so since the video is being scaled anyway this is a great time to get rid of the anamorphism and make the pixels square. All that this requires is to take the Source Aspect Ratio (SAR) into account in the scaling calculation:
scale=iw*sar*min($MAX_WIDTH/(iw*sar)\,$MAX_HEIGHT/ih):ih*min($MAX_WIDTH/(iw*sar)\,$MAX_HEIGHT/ih)
With the filters defined above, all that is required is to put them together into a complete command. It is possible to produce H.264 video with AAC audio by using the following command:
avconv \
-i "$INPUT_FILE" \
-map 0 \
-vf "scale=iw*sar*min($MAX_WIDTH/(iw*sar)\,$MAX_HEIGHT/ih):ih*min($MAX_WIDTH/(iw*sar)\,$MAX_HEIGHT/ih),pad=$MAX_WIDTH:$MAX_HEIGHT:(ow-iw)/2:(oh-ih)/2" \
-c:v libx264 \
-vprofile baseline -level 30 \
-c:a libvo_aacenc \
"$OUTPUT_FILE"
Simply replace $MAX_WIDTH
, $MAX_HEIGHT
, $INPUT_FILE
, and $OUTPUT_FILE
(or define them as environment variables) as desired. That’s it.
I recently spent way too much time tracking down the source of an error in the Squeryl integration to the Record persistence layer in Lift. In the hopes that it may be useful to others encountering the same error, here are the details:
After reading the Squeryl-Record documentation and following the test-squerylrecord and Basic-SquerylRecord-User-Setup examples, I set out to make use of Record with Squeryl. After coding up a very simple test table and schema, I launched the website and was presented with the following exception and stack trace (excerpted):
Message: java.lang.NoSuchMethodException: net.liftweb.record.field.OptionalStringField.<init>(scala.Option)
java.lang.Class.getConstructor0(Class.java:2723)
java.lang.Class.getConstructor(Class.java:1674)
org.squeryl.internals.FieldMetaData$$anonfun$org$squeryl$internals$FieldMetaData$$_createCustomTypeFactory$1.apply(FieldMetaData.scala:511)
org.squeryl.internals.FieldMetaData$$anonfun$org$squeryl$internals$FieldMetaData$$_createCustomTypeFactory$1.apply(FieldMetaData.scala:504)
scala.Option.flatMap(Option.scala:146)
org.squeryl.internals.FieldMetaData$.org$squeryl$internals$FieldMetaData$$_createCustomTypeFactory(FieldMetaData.scala:504)
org.squeryl.internals.FieldMetaData$$anon$1.build(FieldMetaData.scala:425)
org.squeryl.internals.PosoMetaData$$anonfun$3.apply(PosoMetaData.scala:111)
org.squeryl.internals.PosoMetaData$$anonfun$3.apply(PosoMetaData.scala:80)
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:176)
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:345)
org.squeryl.internals.PosoMetaData.<init>(PosoMetaData.scala:80)
org.squeryl.View.<init>(View.scala:58)
org.squeryl.Table.<init>(Table.scala:27)
org.squeryl.Schema$class.table(Schema.scala:338)
[...]
Why the OptionalStringField class (which is part of record) is missing a constructor expected by Squeryl was beyond me. Something odd was going on.
First, I tried copying my code into the (working) examples and running it. Everything worked without issue. The model code was not the issue.
After a bit more digging, I found that
net.liftweb.squerylrecord.RecordMetaDataFactory
was not being set on
org.squeryl.internals.FieldMetaData
because the session initialization and
transaction setup (in my application’s Lift Boot class) were not being
called because Lift continues after Boot throws an exception, which was
occurring before the initialization code was reached. Ugh!
Takeaways:
If you are reading this article on the web using a modern web browser, you should be seeing an XHTML version of this page served as application/xhtml+xml. The merits of the XHTML media type, and XHTML in general, have been widely debated and I will not discus them here. Instead, here is a brief discussion of how this server is configured to serve HTML and XHTML content.
The impatient may wish to skip to the recommended configuration.
The MultiViews option, and the Apache content negotiation process in general, are well suited for serving resources represented by multiple static files with differing file extensions for each representation. File extensions may indicate language and encoding in addition to media type, but this article will focus primarily on the handling of different media types.
For resources which are not represented by multiple static files, other methods
may be better suited than MultiViews to performing content negotiation. In
particular, dynamic content is typically handled by varying the Content-Type
header returned from the content generator while static files with a single
representation which may be served under different media types (e.g. XHTML
being served as text/html) more easily by using a
RewriteCond
to match on %{HTTP:Accept}
followed by a
RewriteRule
with the T
flag to set the returned type. Neither of these techniques will
be discussed further in this article.
For each page that can be served as both HTML and XHTML, simply use the same
filename for each type with a differing file extension (.html
for text/html
and .xhtml
for XHTML) and place them in the same directory. If the two
versions are intended to be identical, it may be possible to generate the HTML
version from the XHTML version using XSLT (as is done for this website using
this XSLT).
With the content in place, simply enable the MultiViews option in the Apache
configuration (e.g. a .htaccess
file at the site root). Also, in order to
enable content negotiation for directory indexes, it is necessary to change
the search order so that it uses a resource without a file extension. This can
be done as follows:
# Enable MultiViews
Options +MultiViews
# Set the directory index to a resource named index
DirectoryIndex index
After this change is made, resources should be accessible by URLs with or without file extensions. When accessed by file extension, the file matching the requested name is returned. When accessed without a file extension, Apache uses the values from the HTTP Accept headers to determine which of the available files best satisfies the request and returns that file to the client. The exact algorithm is described in the content negotiation documentation (this will be important later).
Great! At this point, everything should be working as intended. Mission accomplished.
But wait! You may have noticed that HTML is being served to browsers which support XHTML. What’s going on?
All major browsers currently request HTML (text/html) and XHTML
(application/xhtml+xml) with equal preference (a q
value of 1
). With that
in mind, the content negotiation
algorithm
will return whichever variant has the smallest content length
(assuming they have the same language and character set). If the documents are
structurally identical, this will be HTML (because of the namespace declaration
and extra closing tags). So what do we do?
The recommended solution is to set the quality-of-source factor (used in step 1
of the content negotiation algorithm), which indicates the relative quality of
a given type from the server’s perspective. This can be done on a per-file
basis using a type
map,
or by redefining the type for the file extension to include a qs
parameter
in the Apache configuration as follows:
AddType text/html;qs=0.99 .html
AddType application/xhtml+xml .xhtml
The above configuration specifies that text/html has a slightly lower (99%) relative quality than application/xhtml+xml (with the default quality-of-source of 1, i.e. 100%) such that if the browser requests them as equal quality XHTML will be preferentially chosen.
This has two problems: The first, and most significant, is that it will serve
XHTML to browsers which do not support XHTML and do not express a preference
between HTML and all other content. This includes Internet Explorer prior to
IE9 which
expresses no preference by sending Accept: */*
. This can be avoided by
setting the quality-of-source differently when application/xhtml+xml appears
in the Accept
header:
<If "%{HTTP_ACCEPT} =~ m#application/xhtml\+xml#i">
# application/xhtml+xml is explicitly mentioned. Prefer XHTML slightly.
AddType text/html;qs=0.99 .html
AddType application/xhtml+xml .xhtml
</If>
<Else>
# application/xhtml+xml is not explicitly mentioned. Prefer HTML slightly.
AddType text/html .html
AddType application/xhtml+xml;qs=0.99 .xhtml
</Else>
The other problem is that the qs
media type parameter is also sent to the
client in the Content-Type
header. This is non-standard behavior, since the
qs
is not defined for the HTML or XHTML media type. This bug has been
reported as early as 2002 on the
http-user,
http-dev,
and
ietf-http-wg
mailing lists. I opened Bug
53595 to track the
issue, but I do not expect a fix any time soon (and I am not personally
working on one).
Although the standards require clients to ignore unrecognized media type
parameters, and I am not aware of any issues in popular browsers caused by the
qs
parameter, sending it is asking for trouble. Therefore, to avoid sending
the qs
parameter, consider removing it using
mod_headers
:
Header always edit "Content-Type" ";\s*qs=[0-9]*(?:\.[0-9]+)?\s*" ""
Before settling on the above solutions, I discovered an alternative way to
conditionally prefer XHTML during negotiation using
mod_rewrite
.
This method is more complicated and error-prone than the above solutions, but
it can also be used to influence MultiViews behavior in much more powerful
ways.
The content negotiation process occurs before the rewrite process when the
rewrite rules are in directory
context. This
allows RewriteRules to change the result of the negotiation when it results
in HTML rather than XHTML. It is made more difficult if the restriction that
HTML pages requested explicitly (with a URL that ends in .html
) should still
be served as HTML is maintained. To get the desired behavior, the request
should be changed from HTML to XHTML when all of the following are true:
qs
values were set (i.e. the browser
supports XHTML and HTML with equal quality).To test the first criterion we use the fact that %{IS_SUBREQ}
is true
when
the URL has been changed during content negotiation. This is fragile due to
the fact that if rewrite rules are added before this test it will trigger a
false positive, but I am not aware of a better alternative. The second
criterion can be tested easily by file extension. The third can be tested
using an -f
RewriteCond. The fourth and fifth can be tested by matching
against the content of the HTTP Accept header sent by the client. Rather than
compare the q
values for HTML and XHTML, this implementation takes the
conservative approach and only returns XHTML if XHTML was requested without a
q
value (which is an implicit value of 1
, the maximum). This approach can
be realized with the following addition to the Apache configuration (in
Directory
or .htaccess
context):
RewriteCond "%{IS_SUBREQ}" "=true"
RewriteCond "%{REQUEST_FILENAME}" "^(.*)\.html$"
RewriteCond "%1.xhtml" "-f"
RewriteCond "%{HTTP:Accept}" "application/xhtml\+xml\s*(?:,|$)"
RewriteRule "^(.*)\.html$" "/$1.xhtml"
This approach is almost correct with two remaining problems. First, the content-negotiation process sets the HTTP Content-Location header to inform the browser which resource was actually served. Unfortunately, the RewriteRule does not change this Content-Location. This can be done by setting an environment variable to remember that a change was made, then editing the Content-Location header in the same way. This is further complicated by some undocumented behavior of environment variables in RewriteRules. With this behavior in mind, the above configuration can be extended as follows:
RewriteCond "%{IS_SUBREQ}" "=true"
RewriteCond "%{REQUEST_FILENAME}" "^(.*)\.html$"
RewriteCond "%1.xhtml" "-f"
RewriteCond "%{HTTP:Accept}" "application/xhtml\+xml\s*(?:,|$)"
RewriteRule "^(.*)\.html$" "/$1.xhtml" [ENV=NOW_XHTML]
Header always edit "Content-Location" "\.html$" ".xhtml" env=REDIRECT_NOW_XHTML
The second issue is that when Serving Pre-Compressed Files with Apache
MultiViews
the filename may end in .html.gz
or another encoding, rather than .html
.
To address this, the above rules can be extended to match and preserve
additional extensions after .html
:
RewriteCond "%{IS_SUBREQ}" "=true"
RewriteCond "%{REQUEST_FILENAME}" "^(.*)\.html(\..+)?$"
RewriteCond "%1.xhtml%2" "-f"
RewriteCond "%{HTTP:Accept}" "application/xhtml\+xml\s*(?:,|$)"
RewriteRule "^(.*)\.html(\..+)?$" "/$1.xhtml$2" [ENV=NOW_XHTML]
Header always edit "Content-Location" "\.html(\..+)?$" ".xhtml$1" env=REDIRECT_NOW_XHTML
Due to the complexity and fragility of the RewriteRule method, my current recommendation for serving XHTML with MultiViews, and the one used on this website, is:
# Enable MultiViews
Options +MultiViews
# Set the directory index to a resource named index
DirectoryIndex index
<If "%{HTTP_ACCEPT} =~ m#application/xhtml\+xml#i">
# application/xhtml+xml is explicitly mentioned. Prefer XHTML slightly.
AddType text/html;qs=0.99 .html
AddType application/xhtml+xml .xhtml
</If>
<Else>
# application/xhtml+xml is not explicitly mentioned. Prefer HTML slightly.
AddType text/html .html
AddType application/xhtml+xml;qs=0.99 .xhtml
</Else>
# Remove qs parameter incorrectly sent by MultiViews due to
# https://bz.apache.org/bugzilla/show_bug.cgi?id=53595
Header always edit "Content-Type" ";\s*qs=[0-9]*(?:\.[0-9]+)?\s*" ""
This will serve XHTML in preference to HTML when supported and HTML otherwise, for URLs without a type extension, allowing increased flexibility and cool URLs.
qs
values.mod_headers
method for removing qs
values.qs
parameter to clients.Recently Thunderbird started opening http URLs in the wrong browser. Although you may think that the solution would be a simple configuration change, as I did at the time, it turns out that the process which Thunderbird uses to determine which browser to use is complex, poorly documented, and has changed several times between Thunderbird versions. This post outlines my understanding of the process and, most importantly, how to change the default browser in current versions of Thunderbird.
Thunderbird has a variety of methods available for determining which browser to use. The methods are attempted one at a time until a browser is found.
Update: Raman Gupta pointed out that my original determination of the order in which the methods are attempted was incorrect and after further testing it appears that mimeTypes.rdf is consulted before XDG MIME action (at least on Thunderbird 10 and later, probably on previous versions as well).
This realization also prompted me to dig into the Thunderbird sources to get a better idea of how the process works. The specific order in which the methods are tried, and which methods are available, is based on the platform and the compilation options used for the running version of Thunderbird. Interested developers can look at GetProtocolHandler in nsExternalHelperAppService for the starting point, GetProtocolHandlerInfoFromOS in subclasses of nsExternalHelperAppService for the platform-specific bits and the HandlerService implementation for the platform-agnostic bits. On Unix, the OS-specific methods are mostly handled by GIO (if available) or GnomeVFS or QDesktopServices (if compiled with QT). Although it appears to me that the platform-specific methods are attempted first, the behavior that I have observed indicates that the platform-agnostic methods are attempted first. The behavior that I have observed is that each of the following methods are attempted, one at a time, until one of them is successful. The methods that Thunderbird attempts are (in order):
The mimeTypes.rdf file contains
information for the “Helper Applications” which are used to open external
content. Existing entries can be adjusted in on the Incoming tab of the
Attachments pane of the Preferences window (Edit -> Preferences ->
Attachments -> Incoming
), or simply on the Attachments pane in earlier
versions. Unfortunately, there is no way to add new file-type
associations in the
preferences window.
To add an entry for a scheme that does not appear in the list (e.g. “http” or “https”), use the following process:
about:config
) which can be accessed
through Edit -> Preferences -> Advanced -> Config Editor...
.network.protocol-handler.warn-external.<protocol>
to true
for
each of the protocols that you wish to configure by double-clicking on
the preference. (e.g. change network.protocol-handler.warn-external.http
to true
to configure the program for http URLs).Note: Raman Gupta made a great suggestion that choosing
/usr/bin/xdg-open
as the preferred application will force Thunderbird to use
the XDG MIME action (the desktop manager’s default) if it wouldn’t otherwise
do so.
Advanced users can also view/edit the mimeTypes.rdf file directly, although
this is not recommended. The mimeTypes.rdf file is stored in the profile
directory which can
be found through Help -> Troubleshooting Information
. It is usually located
at ~/.mozilla/thunderbird/XXXXXXXX.default/mimeTypes.rdf
where Xs are
replaced by random characters and default
may be replaced by another profile
name, if named differently. The program association will be stored as an RDF
Description for urn:scheme:externalApplication:<protocol>
with an NC:path
containing the application to run.
The default browser is chosen based on the information in the XDG MIME database, as specified in the XDG MIME Applications Associations spec. Thunderbird looks up the program associated with the URL pseudo-MIME type (e.g. x-scheme-handler/http for HTTP URLs) or the attachment MIME type (e.g. text/html).
The Applications Associations database can often be changed using the
“Default/Preferred Applications” or “File Associations” GUI configuration tool
for your desktop environment. The database can also be queried and modified
using the xdg-mime
command-line tool as follows:
# Query the current default for HTTP URLs
xdg-mime query default x-scheme-handler/http
# Set default program for HTTP URLs to Firefox
xdg-mime default firefox.desktop x-scheme-handler/http
To find the .desktop file for your desired browser, look in
/usr/share/applications
(for system-wide applications) or
~/.local/share/applications
(for user applications).
If xdg-mime
is not available, the defaults can be changed by editing
mimeapps.list
as described in the XDG MIME Applications Associations
spec
and using information from /usr/share/applications/mimeinfo.cache
for
reference. For example, the default program for HTTP URLs can be set as
follows:
[Default Applications]
x-scheme-handler/http=firefox.desktop
When running in a GNOME environment (if the GNOME libraries are present), Thunderbird attempts to determine the default browser based on the preferences stored in Gconf. Thunderbird uses the following preferences for URLs of various types:
/desktop/gnome/url-handlers/http/command
/desktop/gnome/url-handlers/https/command
/desktop/gnome/url-handlers/about/command
/desktop/gnome/url-handlers/unknown/command
If the command string contains %s
, it will be substituted with the URL being
opened.
The information may be modified and queried using gconftool-2 on the command-line as follows:
# Query the current command for http
gconftool-2 --get /desktop/gnome/url-handlers/http/command
# Set the command for http to firefox
gconftool-2 --set /desktop/gnome/url-handlers/http/command \
--type string 'firefox "%s"'
The information can also be queried and modified using a graphical program
such as gconf-editor
.
In all versions of Thunderbird, the default browser may be determined based on
the settings in the Thunderbird preferences. Preferences can be edited by
opening the configuration editor (about:config) which can be accessed through
Edit -> Preferences -> Advanced -> Config Editor...
. Setting a default
protocol handler requires two preferences to be specified. First,
network.protocol-handler.external.<protocol>
must be true
to indicate that
the protocol should be handled by an external program. Next,
network.protocol-handler.app.<protocol>
must be set to the name/path of the
program which should be run to handle the URL. The following preferences may
need to be set/changed:
network.protocol-handler.app.http
network.protocol-handler.app.https
network.protocol-handler.app.ftp
network.protocol-handler.expose.http
network.protocol-handler.expose.https
network.protocol-handler.expose.ftp
Note that these preferences can be managed across multiple machines or made permanent by editing the user.js file. This is not recommended for normal situations but is mentioned here for completeness.
Update: After examining the Thunderbird sources, I am doubtful about
whether this method is still attempted in recent versions of Thunderbird.
There are no references to network.protocol-handler.app
in the Thunderbird
sources
and I didn’t find any code which looks like it accesses these preferences.
The process of changing the default browser is documented on the mozillaZine Wiki. Unfortunately, the page has not been updated since July 2010 and my requests for an account have been silently ignored for weeks. If any reader has the ability to edit that page, I highly encourage you to do so.
Alternatively, this information should probably be posted on the Thunderbird Messaging KB or Mozilla Wiki. I have not yet had the time to rework this post into a suitable format for posting in either location. If someone would like to make the changes, I’d be happy to assist.
mimeapps.list
location from
~/.local/share/applications/mimeapps.list
to
$XDG_CONFIG_HOME/mimeapps.list
since the previous location is described as
“for compatibility, deprecated” in the spec and is less frequently used.Redmine is a web-based project management system, often called a forge, built using the Ruby on Rails framework. It provides bug/issue tracking, time tracking, wiki pages, gantt charting and calendar, multiple project support, and role-based access control for users to name a few. This article will cover the process of installing Redmine on Debian Squeeze using MySQL for data storage, Thin for serving Ruby, and nginx as the outward-facing server.
This post was converted from a page I put together several years ago. It is presented here for posterity and in the hopes that it may still be useful in some way.
The Simple Directmedia Layer (SDL) library provides several methods of displaying images, many of which may be used interchangeably. In order to help developers choose which method to use in a given set of situations, this post presents performance numbers for a variety of these display methods.
Simple Directmedia Layer (SDL) is a cross-platform multimedia library which has been used in the creation of several games, emulators, and other graphically driven applications. SDL provides low-level access to the graphics system, allowing program authors to maximize the performance of their program through precise use of the available graphics hardware. However, with this power comes a certain level of complexity as well as variability from differences in the hardware and the drivers used to control it. The benchmarks/statistics on this page are intended to clear up some of the doubt about the performance characteristics of video backends under some of the different sets of configuration options available, allowing program authors to make better informed decisions about which settings to use in their applications.
When a surface is created, the program author may specify a set of flags which change the behavior of the surface created. The image flags tested were:
Along with the options described above, which apply to images, there are several options which affect the behavior of the screen (drawing surface). The ones tested here are:
SDL_Flip
will cause the updated surface to be displayed and the other surface to be
set to receive future updates. This prevents tearing effects from
appearing on the screen as different parts of the screen are updated.
Instead it allows the entire screen to be updated in a single action, creating
a more unified picture. Note: SDL_DOUBLEBUF requires a hardware surface to
work, so specifying SDL_DOUBLEBUF is equivalent to SDL_DOUBLEBUF|SDL_HWSURFACE
(the long form is used below for clarity).The test program takes an image and converts to the display format using
the specified surface flags. For surfaces with
SDL_SRCCOLORKEY
set, it converts completely transparent pixels
into color-keyed pixels. The test program then blits the image to the
screen 1000 times, as fast as it can and records the time required to complete
the blits, then calculates the framerate based on this time. The program
runs at a resolution of 800x600. For the exact details, please consult
the source to the test program.
Note that this is not necessarily a fair comparison, since in "real-world" conditions blits would be sent at greater intervals. However, for testing this would bring up problems with measurement errors for the timing of individual blits (requiring very high precision) or with the precision of sleep functions used to space out the blits. Given these caveats, the test results do seem to agree with the experienced performance under more practical conditions and none of the backends seem to be limiting themselves to the vertical refresh rate or any other constraint requiring greater spacing of the blits.
The image used for the tests was a scaled version of Klowner's See, Hear, Speak... wallpaper (the exact image used for testing is here). This image was chosen because it is similar to typical sprites found in 2D games (gradients with few colors, alpha around the edge of the images, rounded edges) and because it is just a really cute image.
The DirectFB video backend is used to display graphics on a Linux framebuffer device (the console) using Direct FB. On the test system, DirectFB was running in the console on top of the vesa frame buffer. Although the test program runs at 800x600, DirectFB was unable to resize my display from 1024x768. This left a black border around the edges of the screen which, although not really important for a test program, may be a consideration for programs where visual appeal and use of screen space is important. Also, fullscreen hardware surfaces were only available at 16 bits (which may be a result of my vesafb configuration), so the test results for this configuration should be considered with this limitation.
During the testing of this video backend several notable characteristics of the backend appeared that may be worth considering before deploying a program using this backend. When SDL_FULLSCREEN was used for the display surface it caused the image to appear significantly off-center such that it wrapped around the edge of the monitor. On top of that, the color keyed images showed (sometimes significant) blinking or tearing for the hardware display surfaces, although the framerate is recorded as being impressively high (this needs further investigation). Finally, surfaces with the SDL_SRCALPHA flag did not render when the display surface was a hardware surface, so the numbers for this configuration are somewhat meaningless.
As one other implementation note, it may be necessary to pass
video=vesafb:ywrap,mtrr
to the kernel in order to enable hardware
surfaces. DirectFB complained loudly before I added the line to the boot
configuration. See the DirectFB documentation for details.
The DirectX video backend provides graphics support using
Microsoft DirectX® on Microsoft
Windows® platforms. There were no unexpected visual effects
or strangeness during the testing, although some of the results were
unexpected. The first oddity was the
surprisingly low framerate for surfaces with the SDL_SRCALPHA
flag on a SDL_HWSURFACE
display surface. In fact, the low
framerate caused the test to run for a long enough time that there was debate
about whether or not the test had frozen. Another oddity was the low
framerate on the double-buffered display surface. My guess is that when
double-buffered the render code waits for a vblank before finishing, so the
framerate is capped at the refresh rate of the monitor (which would explain the
60 fps recorded for double-buffered output on the test machine). One
other caveat to note is that it is not possible to allocate a hardware surface
unless the display surface is fullscreen (and a hardware surface, of
course).
The FBCon video backend provides graphics support for consoles using the
Linux Framebuffer.
Although this driver does support hardware display surfaces, requesting a
display surface with SDL_HWSURFACE
caused the test machine to hard
lock and prevented gathering any test data for these surfaces. (Note that
this may be worth checking against before allocating a hardware display surface
to prevent hard locking a client's machine).
The glSDL video backend stands out in comparison to many of the other backends in that it works in concert with another backend to blit images using OpenGL. This backend may provide performance improvements on computers with hardware 3D acceleration where the work of blitting an image can be offloaded to the graphics processor through OpenGL. In this test, the glSDL backend was running on top of the X11 backend on Linux.
The state of this backend in the SDL community is also somewhat ambiguous. The original patches to SDL were the work of David Olofson and Stephane Marchesin, and are available as a patch providing the backend or as a wrapper around SDL. Although there has been much discussion of including the backend with SDL or in writing a different implementation of its features, neither solution has materialized and the backend remains largely unsupported. Even applying the patch to the latest libSDL requires quite a bit of work.
For this test, a reworked version
of the glSDL backend patch against libSDL 1.2.11 was used. However,
there is an error somewhere in the patch which results in color inversions
under many of the testing conditions (whether this was introduced in the
reworking or if it was present in the original patch is unknown). This
appears to be the result of incorrect RGBA ordering in the
DisplayFormatAlpha
function (or that only manifests when this
function is used). Hopefully, I will have the time to fix this in the
near future...
The SVGAlib video backend provides graphics support for Linux using SVGAlib. This video backend is not capable of dealing with hardware surfaces, so those results are omitted from the graph. Note that the version of SVGAlib used for testing (1.4.3) requires that the user be root. This likely provides a significant hurdle for potential end-users of a program using the SVGAlib backend and in addition it may affect the scheduling of the test program by the operating system and skew the test results (although I am unsure of this).
The WinDIB video backend provides graphics support for Microsoft Windows® platforms using Windows® GDI. This backend does not support hardware surfaces and therefore those results are omitted from the graph.
The X11 video backend provides graphics support for the X Window System. This video backend does not support hardware surfaces and therefore those results have been omitted from the graph.
Unfortunately, the test box does not support the DGA extension, which prevents testing of the DGA video backend.
Across all of the graphics backends the use of RLE acceleration showed a significant speed increase for color keyed images. Interestingly, the use of color keys alone shows a highly dramatic decrease in framerate as compared to both non-color keyed surfaces and surfaces using source alpha. For surfaces where a color key is required and the surface will not be modified often, the use of RLE acceleration seems like a must.
The glSDL backend shows particularly high performance for use with hardware surfaces. If the problems with colors can be overcome, or a new implementation of an OpenGL-based backend is created, this option seems like a winner. However, given the current limitations, it does not appear to be ready for production programs.
Although hardware and double-buffered surfaces are likely to provide the highest performance, they showed a particularly high degree of strange effects and poor performance in the test environment. I strongly suggest that if the use of hardware surfaces is being considered for an SDL application that a significant amount of testing be done on a variety of systems to ensure that none of the undesired visual effects or low performance is encountered on target systems.
Beyond those observations, the best suggestion is to try a few combinations and see what works best with your program. As always, take it all with a grain of salt and go with practical experience. Best of luck!