Archive for the 'Progress' Category beta 1 testing release

Thursday, October 9th, 2014

After a lot of work on the 3G support in the Fonera firmware, it’s finally time to share some of this work with you in the form of this beta release. As the name suggests, this release should be considered a testing release – it’ll probably not eat your data, but don’t be surprised if your internet connection breaks at some point. Having said that, we welcome you to try this release and provide feedback about any issues you encounter. You can always downgrade back to the latest stable firmware if there is a problem that is a showstopper for you.

3G improvements
As mentioned, improved 3G support is the biggest change in this release. By itself, this release does not magically add support for more 3G devices (though it’s likely that some devices that didn’t work before now do due to usb-modeswitch updates and kernel fixes), but it mostly cleans up the 3G support and makes it easier to debug issues with 3G devices.

For some more details about how the new 3G interface works and how to go about getting your device working, a separate blogpost is available. Please carefully read that post if you run into 3G problems.

Note that 3G devices no longer automatically connect when plugged in. By default, you’ll have to connect by clicking a button in the webinterface, though you can still configure the old behaviour of automatically connecting on device insertion by going to 3G -> Configure and then select “On device insertion: connect automatically”.

3G dongles and hubs
One change to point out specifically is that 3G devices in combination with USB hubs should not cause any problems anymore. Previously, using an USB hub would break connectivity for some 3G dongles. It turned out there were two unrelated bugs in the kernel USB driver that both caused problems with hubs under different circumstances. After extensive debugging, these issues have been found and fixed.

If you still find an issue with a hub (i.e., your dongle works without a hub, but doesn’t when connected through a hub) please report it.

Debug application
Another new addition in this release is the debug application. It’s not very powerful yet, but allows collecting some debugging information through the webinterface, as well as increasing the log buffer size. This should help us to collect debug information from users that are not comfortable with using the SSH interface.

To install this application, just head over to the “Applications” page on your dashboard – it should be listed there.

So, how do I upgrade?

Download the web upgrade tarball below, navigate to your Fonera’s dashboard through “Settings” and then “System” and upload the tarball there (under “Firmware update”). Note that you should not unpack the file you download (and make sure your browser doesn’t automatically unzip the file either, it should remain a .tgz file).

Note: On 2.0g, the firmware update might fail due to insufficient memory. If you upload the firmware through the webinterface, but that page keeps loading indefinitely (and you do not get the updating countdown), you’re probably seeing this issue as well. As a workaround, you can disable windows file sharing (“Settings” -> “Fileserver” -> “Windows Network Shares”) and/or the public Fonspot (“Settings” -> “FONSpot”) to free up some memory. This should allow the 2.0g to successfully unpack the firmware and start flashing it.

This firmware is available for the following hardware. Be sure to pick the version for your hardware. If you pick the wrong one, the firmware update will silently bail out and no upgrade will happen (no error will appear either).

Since this is a beta release, only developer’s versions are available. For the .img version (which can be installed through SSH), you can look in the directory on the download server.

If the upgrade somehow fails and you need to recover, the FAQ section has recovery instructions.

Have fun!

3G support progress

Thursday, March 14th, 2013

As soon as the Paco release was out last christmas, we’ve started working on the next firmware release. The focus of this release will be on improving the 3G experience, which is the topic of this post. I’ll detail a few of the things we’ve be working on, to give you an idea what’s coming up and to get an idea of all the different components that are involved in getting a 3G connection up and running.

Changing the connection manager daemon
The current firmware uses “umtsd.lua”, which is a simple connection manager written in lua. It was a first attempt at something back when the 2.0g and 2.0n were being released, and that shows. Its internals are a bit messy and it is hard to debug any 3G problems due to limited feedback.

To solve this, a new daemon was written, named “udiald” (originally it was called “umtsd2″). This daemon has been released by Fon under an open source license and its sources can be found on github. This new daemon is written in C, has a cleaner structure and makes it a lot easier to change device-specific configuration and get debug output. We expect we can support more 3G devices in the future because of udiald, though in the short run some devices might stop working when switching to udiald (because they just worked “by accident” in umtsd.lua and now need some specific tweaking applied, for example).

At some point, this udiald (or probably something based off it) might also be integrated into the main OpenWRT repository, to improve 3G support in general OpenWRT as well.

Rewriting the 3G webinterface
Managing the 3G connection is a bit of a challenge in the current firmware. When you plug in a device, it automatically connects and having two 3G dongles connected at the same time is asking for trouble. If you’re running into problems, most of the time you’ll only see that the Fonera is stuck “Dialing…”, but there is no indication of any error message.

We’ve revamped this webinterface now, making it easier to see what 3G dongle is connected and what its status is. We’re making automatically connecting on plugin optional (and disabling it by default), so you have more control over how your Fonera connects to the internet.

2.0n USB driver improvements
One of the long-standing issues with 3G devices on the Fonera 2.0n is that most of them would not properly operate when connected through an USB hub. Devices that work correctly when directly connected to the Fonera fail when connected through an USB hub, even when no other devices are connected.

We’ve been diving into this issue for the last month or so and slowly started to unravel the causes for this issue. When communicating with the Fonera, the 3G devices offer virtual “serial ports”, which can send and receive data. When the Fonera is receiving data from the 3G device, it often occurs that the device has no data to send. In this case, the 3G device will send a USB “NAK” message back to the Fonera to let it know there is no data. The (USB controller in the) Fonera can then knows there is no data yet and it should just keep trying until there is data. These NAK messags were the root cause of the problem, but the actual problem differed depending on the USB version of the 3G device:

  1. For USB 1.1 3G devices, which use so-called “split transactions” to translate from USB 2.0 to 1.1, every NAK message received caused the Fonera to receive an interrupt, process the NAK message and resubmit the original message for a retry. Apparently, these NAK messages would occur so often that the Fonera’s CPU would be busy only with processing these interrupts, leaving no time for any other useful work and eventually causing a reboot.

    We managed to fix this issue by only retrying these NAK’d messages once per USB frame (millisecond), a fix that was inspired by a similar fix found in the Raspberry Pi Linux version (which uses a very similar USB controller). This makes the USB 1.1 3G devices we tested work properly through an USB hub.

  2. For USB 2.0 3G devices, the USB hardware takes care of all these retries, so the interrupt overload issue does not occur. However, here we were running into a hardware limitation: The USB hardware uses “Host Channels” (HC) to communicate with USB devices. The Fonera 2.0n has four of these HCs available, and a host channel can be used to send a message to one specific USB “endpoint”, meaning we can send message to four different endpoints at the same time (but once a message is sent, a HC can be reconfigured to send a message to another endpoint if needed).

    Now, for each “periodic endpoint” (isochronous and interrupt), a dedicated HC is needed. These periodic endpoints need to have messages sent at regular intervals, and once the time for such a periodic message comes, it must be certain that a HC is availalbe, so the driver reserves one HC for each of these periodic endpoints. This allows up to three HCs for periodic transfers, leaving at least one HC available for non-periodic transfers.

    This is what happens when you connect a 3G device through an USB hub. The USB hub and the 3G device both have an interrupt endpoint, which takes up two of the four HCs. Now udiald connects to two of these “virtual serial ports”, one to transfer control messages and one to transfer data. Receiving data from these virtual serial ports takes up two more HCs. These latter two are expected to be used only shortly, but because the 3G device often does not have data to send for some time (a few to a few dozen seconds, typically), all of the host channels are taken up. Now, when udiald wants to send something to the device (either some network data or a control command), it needs one more HC (since input and output are different endpoints in the USB protocol). The outgoing data is queued, waiting for a host channel to become available, which doesn’t happen, so the connection times out before it even started.

    To fix this, we’re trying a workaround where these “blocking” HCs (which keep getting NAK replies from the device and keep getting retried by the hardware) are interrupted when the driver runs out of available host channels, so these host channels can be used for other transfers for a while and then resume retrying the blocking transfers. This is now working fairly well in our development environment, though there might be one or two corner cases left to tackle.

In summary: We’ve managed to improve this third-party USB driver in order to better support blocking transfers typically used by 3G devices. This helps to get the 3G devices we have here working, but of course it is no guarantee that all devices will start working with USB hubs: there might be other problems we simply did not observe yet. However, the problems we solved are fairly fundamental, so they will have affected most if not all 3G devices out there.

If you’re interested in the details about this USB driver problem, I can recommend the USB made simple and USB in a nutshell article series, which properly explain the terms and logic of the USB protocol.

Remaining USB limitations
Even though the above should make 3G devices work with USB hubs, it is expected that the performance is somewhat reduced by this approach, because of the way a channel must now be time-shared between multiple transfers.

Furthermore, this approach does not mean there is no limited number of host channels all of the sudden. Now, multiple (blocking) non-periodic transfers can succesfully share a single host channel, but there is still only room for three different periodic endpoints. Fortunately, USB storage devices typically do not use a periodic endpoint, so this limit should be sufficient in most cases (e.g., you can add one hub, a 3G stick and an USB sound card, each of which typically has one periodic endpoint and a number of USB disks or flash drives without problems).

Backporting parts of the serial drivers
When a 3G device is plugged in, its virtual serial ports are handled by an usb serial driver in the Linux kernel. This driver needs to know which devices it should handle (usually based on the usb “vidpid”, vendor id and product id, as advertised by the device) and for some devices, which workarounds or special quirks to apply. The driver most commonly used for 3G devices is called the “option” driver, which we’ve improved by taking the list of USB devices and some of the workarounds from a newer kernel version and backporting them to the Fonera kernel versions. This should allow more 3G devices to be supported by both Foneras, since now udiald can actually talk to these devices.

This change should remove the need to mess with the new_id files in /sys, which was sometimes needed to get a 3G device recognized by its driver.

Updating usb-modeswitch
usb-modeswitch is the software responsible for getting 3G devices into “modem mode”. A lot of 3G devices start out in “storage mode”, where they pretend to be a read-only CD drive or card reader, offering the Windows drivers for the device on a virtual disk. To get these devices to offer their virtual serial ports used for the 3G connection, they need to be sent a special message (which is normally done by Windows driver installed from the virtual disk) to switch them to modem mode.

By updating usb-modeswitch and the associated database of 3G devices and the messages they need to be sent to make them switch, more 3G devices can be supported that were previously not accessible to udiald.

Updating pppd
pppd is the daemon that is responsible for setting up the actual IP connection once all the dialing and configuration is complete. It handles negotiations about IP addresses and DNS servers, using standardized PPP (Point-to-Point Protocol). Even though PPP is an old and standardized protocol, we were still running into some problems with 3G devices that would confuse pppd with particular (IIRC not entirely standards-compliant) negotiation sequences, causing the connection to fail. Newer pppd version turn out to handle these devices more gracefully, making this particular problem (and possibly others) go away.

All of the work above is ongoing in our development trees. We’ll start pushing out these changes into the fon-ng Subversion repository over the coming weeks and once we’re satisfied that this produces a firmware image suitable for testing, we’ll be releasing the first beta release. The goal of this release will be mostly to gather information about the support for different 3G devices: which devices work right away, which need some configuration tweaks, are there still more fundamental problems with devices?

Fonera big disk support, testers needed

Monday, August 27th, 2012

The current SVN version of the Fonera 2.0n and 2.0g firmware should support big disks (more than 2TiB / 2.20 TB) connected through USB, which is great combined with the Transmission torrent client.. But since we can only test with so many different types of disks, we’d like to get some feedback about whether this support works with your disk.

In addition to supporting bigger disks, the mountd sofware (which takes care of making external USB disks available on your Fonera), has received a number of updates. It should be a bit more stable now and it offers more information about the different partitions on your disk through the WebGUI (in particular, information about partitions with an unsupported filesystem or partitions that could not be enabled for whatever reason is now shown).

So, if you have a big disk lying around and/or use the USB disk support a lot, you can help by upgrading to the latest SVN build and see if your disks (still) work with your Fonera. If they do, or don’t, or you’re having other disk-related problems, just leave a comment here (preferably including the disk brand and size).

How to install?
To test this SVN version, you can download it from our auto-builder here (pick fonera2 for 2.0g and fonera2n for 2.0n):

The easiest way to install this firmware version is to download the tgz file and simply upload it through your WebGUI (you should be running beta1 or above already). You can always downgrade to a stable or beta firmware again if you want.

Why aren’t these big disks supported by default, anyway
These limits come from the fact that 32-bit numbers were originally used to describe disk and partition sizes and addresses. Because, given the sizes of modern disks, 32-bit numbers cannot count very far (“only” up to 4.2 billion), we run into size limits when using them. To fix this, all parts of the Linux kernel as well as the partition table should be using 64-bit values, which can count over a billion times farther. Where the disk size limit lies exactly, depends a bit on the partition table and sector size in use.

Each disk uses a partition table to describe the partitions that are present on the disk. The standard partition table format is MSDOS/MBR, which uses 32-bit sector numbers, limiting the usable disk size to 2 TiB (232 * 512). To solve this, the GUID partition table (GPT/EFI) was created, which uses 64-bit sector numbers, practically removing the disk size limit altogether.

As a separate development, hard disks are switching from using 512-byte sectors to 4096-byte sectors. A sector is the smallest piece of the disk that can be separately addressed and using bigger sectors allows disks to more efficiently store and retrieve data. Using bigger sectors has the side effect that partition tables (which count sectors, not bytes) can address bigger disks.

Note that there are also disks that internally use 4096-byte sectors, but appear to use 512-byte sectors through some internal translation mechanism. The Fonera treats these disks as normal 512-byte disks.

To find out the sector size of your disk, look in the file “/var/state/mountd” on your Fonera or run the command “wmic DISKDRIVE get bytespersector,caption” on a command promt on a Windows machine.

Combining the partition table and sector size in use, we get:

  • Disks with normal sized sectors (512 bytes), using a standard (MSDOS/MBR) type partition table support only disks up to 2 TiB (2.20 TB, possibly more if all partitions are 2 TiB or smaller and start before the 2 TiB limit, but I haven’t tested this). These disks were supported on previous firmware releases as well.
  • Disks with normal sized sectors, using a GUID (GPT/EFI) type partition table, support disks over a billion TB in size. These disks should be supported on 2.0n since beta2, and on 2.0g in the current SVN version.
  • Disks with big sectors (4096 bytes), using a standard (MSDOS/MBR) type partition support disks up to 16 TiB (17.6 TB). These disks should be supported in the current SVN version.
  • Disks with big sectors (4096 bytes), using a GUID (GPT/EFI) type partition support disks over a trillian TB in size. These disks should be supported in the current SVN version.

Note: In this post TiB (Tebibyte) is used to mean 1024 * 1024 * 1024 * 1024 bytes, which is the conventoin used by computers to display disk sizes (and usually displayed as “TB” as well). Disk manufacturers usually use TB (Terabyte) to describe 1000,000,000,0000 bytes, which is why disks appear smaller than the manufacturer specifies them.

Last post!

Wednesday, March 31st, 2010

Hi all,

This is the last post on the fonosfera… before the next release!

We are right now (while I write these lines here), fixing the last bugs in the list for this release. They all seem to be under control. This has been possible thanks to the testers Alexandre, Marco, Juan Manuel, Cristian and others. And of course all the others that have been reporting bugs. Thanks a lot to all.

So, please, stay tuned, we will very very very soon have a new firmware for you guys. Only remember, we’ve been focusing on connectivity: making the router work as a router. Other bugs will be fixed in coming releases.