3G support progress

As soon as the Paco release was out last christmas, we’ve started working on the next firmware release. The focus of this release will be on improving the 3G experience, which is the topic of this post. I’ll detail a few of the things we’ve be working on, to give you an idea what’s coming up and to get an idea of all the different components that are involved in getting a 3G connection up and running.

Changing the connection manager daemon
The current firmware uses “umtsd.lua”, which is a simple connection manager written in lua. It was a first attempt at something back when the 2.0g and 2.0n were being released, and that shows. Its internals are a bit messy and it is hard to debug any 3G problems due to limited feedback.

To solve this, a new daemon was written, named “udiald” (originally it was called “umtsd2″). This daemon has been released by Fon under an open source license and its sources can be found on github. This new daemon is written in C, has a cleaner structure and makes it a lot easier to change device-specific configuration and get debug output. We expect we can support more 3G devices in the future because of udiald, though in the short run some devices might stop working when switching to udiald (because they just worked “by accident” in umtsd.lua and now need some specific tweaking applied, for example).

At some point, this udiald (or probably something based off it) might also be integrated into the main OpenWRT repository, to improve 3G support in general OpenWRT as well.

Rewriting the 3G webinterface
Managing the 3G connection is a bit of a challenge in the current firmware. When you plug in a device, it automatically connects and having two 3G dongles connected at the same time is asking for trouble. If you’re running into problems, most of the time you’ll only see that the Fonera is stuck “Dialing…”, but there is no indication of any error message.

We’ve revamped this webinterface now, making it easier to see what 3G dongle is connected and what its status is. We’re making automatically connecting on plugin optional (and disabling it by default), so you have more control over how your Fonera connects to the internet.

2.0n USB driver improvements
One of the long-standing issues with 3G devices on the Fonera 2.0n is that most of them would not properly operate when connected through an USB hub. Devices that work correctly when directly connected to the Fonera fail when connected through an USB hub, even when no other devices are connected.

We’ve been diving into this issue for the last month or so and slowly started to unravel the causes for this issue. When communicating with the Fonera, the 3G devices offer virtual “serial ports”, which can send and receive data. When the Fonera is receiving data from the 3G device, it often occurs that the device has no data to send. In this case, the 3G device will send a USB “NAK” message back to the Fonera to let it know there is no data. The (USB controller in the) Fonera can then knows there is no data yet and it should just keep trying until there is data. These NAK messags were the root cause of the problem, but the actual problem differed depending on the USB version of the 3G device:

  1. For USB 1.1 3G devices, which use so-called “split transactions” to translate from USB 2.0 to 1.1, every NAK message received caused the Fonera to receive an interrupt, process the NAK message and resubmit the original message for a retry. Apparently, these NAK messages would occur so often that the Fonera’s CPU would be busy only with processing these interrupts, leaving no time for any other useful work and eventually causing a reboot.

    We managed to fix this issue by only retrying these NAK’d messages once per USB frame (millisecond), a fix that was inspired by a similar fix found in the Raspberry Pi Linux version (which uses a very similar USB controller). This makes the USB 1.1 3G devices we tested work properly through an USB hub.

  2. For USB 2.0 3G devices, the USB hardware takes care of all these retries, so the interrupt overload issue does not occur. However, here we were running into a hardware limitation: The USB hardware uses “Host Channels” (HC) to communicate with USB devices. The Fonera 2.0n has four of these HCs available, and a host channel can be used to send a message to one specific USB “endpoint”, meaning we can send message to four different endpoints at the same time (but once a message is sent, a HC can be reconfigured to send a message to another endpoint if needed).

    Now, for each “periodic endpoint” (isochronous and interrupt), a dedicated HC is needed. These periodic endpoints need to have messages sent at regular intervals, and once the time for such a periodic message comes, it must be certain that a HC is availalbe, so the driver reserves one HC for each of these periodic endpoints. This allows up to three HCs for periodic transfers, leaving at least one HC available for non-periodic transfers.

    This is what happens when you connect a 3G device through an USB hub. The USB hub and the 3G device both have an interrupt endpoint, which takes up two of the four HCs. Now udiald connects to two of these “virtual serial ports”, one to transfer control messages and one to transfer data. Receiving data from these virtual serial ports takes up two more HCs. These latter two are expected to be used only shortly, but because the 3G device often does not have data to send for some time (a few to a few dozen seconds, typically), all of the host channels are taken up. Now, when udiald wants to send something to the device (either some network data or a control command), it needs one more HC (since input and output are different endpoints in the USB protocol). The outgoing data is queued, waiting for a host channel to become available, which doesn’t happen, so the connection times out before it even started.

    To fix this, we’re trying a workaround where these “blocking” HCs (which keep getting NAK replies from the device and keep getting retried by the hardware) are interrupted when the driver runs out of available host channels, so these host channels can be used for other transfers for a while and then resume retrying the blocking transfers. This is now working fairly well in our development environment, though there might be one or two corner cases left to tackle.

In summary: We’ve managed to improve this third-party USB driver in order to better support blocking transfers typically used by 3G devices. This helps to get the 3G devices we have here working, but of course it is no guarantee that all devices will start working with USB hubs: there might be other problems we simply did not observe yet. However, the problems we solved are fairly fundamental, so they will have affected most if not all 3G devices out there.

If you’re interested in the details about this USB driver problem, I can recommend the USB made simple and USB in a nutshell article series, which properly explain the terms and logic of the USB protocol.

Remaining USB limitations
Even though the above should make 3G devices work with USB hubs, it is expected that the performance is somewhat reduced by this approach, because of the way a channel must now be time-shared between multiple transfers.

Furthermore, this approach does not mean there is no limited number of host channels all of the sudden. Now, multiple (blocking) non-periodic transfers can succesfully share a single host channel, but there is still only room for three different periodic endpoints. Fortunately, USB storage devices typically do not use a periodic endpoint, so this limit should be sufficient in most cases (e.g., you can add one hub, a 3G stick and an USB sound card, each of which typically has one periodic endpoint and a number of USB disks or flash drives without problems).

Backporting parts of the serial drivers
When a 3G device is plugged in, its virtual serial ports are handled by an usb serial driver in the Linux kernel. This driver needs to know which devices it should handle (usually based on the usb “vidpid”, vendor id and product id, as advertised by the device) and for some devices, which workarounds or special quirks to apply. The driver most commonly used for 3G devices is called the “option” driver, which we’ve improved by taking the list of USB devices and some of the workarounds from a newer kernel version and backporting them to the Fonera kernel versions. This should allow more 3G devices to be supported by both Foneras, since now udiald can actually talk to these devices.

This change should remove the need to mess with the new_id files in /sys, which was sometimes needed to get a 3G device recognized by its driver.

Updating usb-modeswitch
usb-modeswitch is the software responsible for getting 3G devices into “modem mode”. A lot of 3G devices start out in “storage mode”, where they pretend to be a read-only CD drive or card reader, offering the Windows drivers for the device on a virtual disk. To get these devices to offer their virtual serial ports used for the 3G connection, they need to be sent a special message (which is normally done by Windows driver installed from the virtual disk) to switch them to modem mode.

By updating usb-modeswitch and the associated database of 3G devices and the messages they need to be sent to make them switch, more 3G devices can be supported that were previously not accessible to udiald.

Updating pppd
pppd is the daemon that is responsible for setting up the actual IP connection once all the dialing and configuration is complete. It handles negotiations about IP addresses and DNS servers, using standardized PPP (Point-to-Point Protocol). Even though PPP is an old and standardized protocol, we were still running into some problems with 3G devices that would confuse pppd with particular (IIRC not entirely standards-compliant) negotiation sequences, causing the connection to fail. Newer pppd version turn out to handle these devices more gracefully, making this particular problem (and possibly others) go away.

All of the work above is ongoing in our development trees. We’ll start pushing out these changes into the fon-ng Subversion repository over the coming weeks and once we’re satisfied that this produces a firmware image suitable for testing, we’ll be releasing the first 2.3.8.0 beta release. The goal of this release will be mostly to gather information about the support for different 3G devices: which devices work right away, which need some configuration tweaks, are there still more fundamental problems with devices?

27 Responses to “3G support progress”

  1. Nuno Says:

    These are great news !
    Currently I have one of the newer 4G / LTE usb modems.
    Will it also be supported by fon ?

    for your reference this is the model I’m using: HUAWEI E392u-12

  2. Alex Says:

    Please, support Huawei E173 :)

  3. Tamas Kalman Says:

    Hey,

    Is there a chance for 4G Wimax USB modem support?
    It would be incredible amazing if you could add Clear 4G USB modems (USA).

  4. Matthijs Says:

    @Nuno & Tamas, If the 4G modems also behave like 3G modems and present a serial interface that responds to AT modem commands and can setup a PPP connection, then it should be possible to support these modems (they might even already work with the right settings). However, if they provide a different interface at the USB level, then things might be more complicated. Frankly, I’ve never worked with these devices yet.

    To make this happen, perhaps you could each open up a a ticket on the trac for the device you have and include the output of the “cat /proc/bus/usb/devices” command as run on your Fonera with the modem plugged in?

    @Alex, The E173 should be well supported, since that’s one of the devices I’ve been using during development.

    @All, most of the 3G-related changes are in SVN now, so if you’re curious, give the latest svn auto-build a spin! (see this post)

  5. Alex Says:

    @Matthijs I can’t believe it! Thank you!!!! :D:D

  6. TheUntouchable Says:

    Hi Matthijs,

    installed the latest build r2295 / 2.0n dev with the firmware-fonera2n.tgz method. But after that, its not possible to choose any plugin to install, the list is empty =/

  7. Matthijs Says:

    @TheUntouchable, that is because the version number for SVN builds was already bumped to 2.3.8.0. I’ve now add some configuration to the download server such that plugins should be available again, could you see if this fixes the problem for you?

  8. TheUntouchable Says:

    Yes thank you, the plugins are back again :)

  9. Reloweb Says:

    In the next firmware I wish it could be possible configure NFS sharing easily from the GUI interface. I know this is off topic but i want you to know!

    Thanks for all the work you will make! :-D

  10. Cedric Bou Says:

    Hi,

    That a good idea to work on this subject.
    And I am the same as Nuno, I would love to have compatibility with the HUAWEI E392u-12 because 4G is coming in France and it would be nice to be able to have this. And you can find some HUAWEI E392u-12 quite easily for a good price.
    I was looking for a modem that would be able to handle my HUAWEI E392u-12 because my fonera 2n isn’t able to do it for now. And I have read that TL-MR3420 was able to support it but only in Portugal. You can find the bin file they used to make this possible (if it can helps…).

    And by the way, I love the new 2.3.7.0 firmware ;-)
    Good job, thank you!

  11. Alex Says:

    Will we have the update soon? :)

  12. peter Says:

    good! I prefer when go in Bridge Mode continued to controlled the fonera from 192.168.0.1, but dosent work in bridge mode. plz support this feature.
    good job

  13. Matthijs Says:

    @Reloweb, sounds like a useful feature, though I doubt will implement it soon. Still, it could be useful if you would file an enhancement report on the trac?

    @Cedric, as I asked Nuno, perhaps you could open up a ticket on the trac? Seems none has been opened yet so far.

    @Alex, development got sidetracked a bit with work on the USB driver, but we’re still slowly progressing.

    @Peter, in bridge mode your Fonera is in the same IP range as your upstream router (e.g, the router or modem/router that you connect your Fonera to). So it’s likely that the upstream router already uses the 192.168.0.1 address, and thus the Fonera cannot also use it. You should either pick a different address for your Fonera, or configure the upstream router to use a different address.

  14. Hans Vos Says:

    Any news about when we will be able to test the new 3g functionality ?

  15. Matthijs Says:

    @Hans, I’ve been sidetracked a bit working on the 2.0n USB driver which is now getting into the mainline kernel (which we can hopefully use for the 2.0n in the longer term). This week, I’ve also done some more work on the 3G support for the next firmware version. The basic support is finished and I just committed a small application that should help for debugging.

    What’s left now is to do a bit more testing (and fix the things I find) and then wrap up for the beta release. It’s always hard to give an exact timeframe, but it’ll probably be somewhere within the next month.

  16. Gustav Says:

    Thank you fonosfera.

  17. The Effiov Says:

    Are we going to get an early Xmas prezzie or even just an Xmas prezzie in the form of a beta?
    Thanks for the good work.

  18. Ton Says:

    I miss Matthijs here!

  19. Matthijs Says:

    Good to hear I’m appreciated ;-)

    I’ve been quite distracted with some other (non Fon-related) work in the past few weeks, so not much progress here. However, it seems like a good idea to do another release before christmas. I should at least manage the 2.3.7.1-rc1 release and hopefully also be able to wrap up the 2.3.8.0-beta1 release… Now, back to work! :-)

  20. Effiov Says:

    Yes, you are very appreciated! Great to hear that you will be delivering a firmware for us (me) to play with before Xmas. I have 4 different dongles here to try out!
    Effiov

  21. Effiov Says:

    I still have my fingers crossed for something before Xmas but not so hopeful now with just a few days to go. Oh well, hope Santa brings me something on my list anyway :-)

  22. Matthijs Says:

    Unfortunately, we’ve ran into an issue with the updated wifi driver (for fixing a WPS problem in 2.3.7.1), which delayed both the 2.3.7.1 and 2.3.8.0 beta release… After a few days of debugging, I haven’t foudn the exact cause of the issue yet, but traced it back to some changes in the wifi driver that we don’t strictly need. I’ll be proceeding with the 2.3.7.1 release with a driver without those changes. But, since Christmas is upon us already, the actual release will be the 26th at the earliest.

    For the 2.3.8.0 beta release, I need to have a look at (and fix) this issue as well as do some overall testing to make sure that everything works as expected.

  23. Effiov Says:

    Mathijs and all, have a good Xmas!
    Thanks, indeed, for the update on progress.

  24. fonosfera » Blog Archive » Paco Jr: small bugfix release Says:

    […] we keep working on revamed 3G support for the next feature release, we also found some small but important bugs or regressions. To make […]

  25. Aggiornamento firmware: piccola versione bugfix - MAGICLIFE WIFI Says:

    […] i gestori delle fonere continuano a lavorare su supporto 3G revamed,  per la prossima feature release, abbiamo anche trovato alcuni piccoli ma importanti bug o […]

  26. Ridjal Says:

    please,…Sierra 312U modem support by Fon ?

  27. Matthijs Says:

    @Ridjal, I don’t have a 312U to test, so I can’t be sure if it will work. However, a quick google shows that it is a regular PPP-based 3G modem that works out of the box on Ubuntu, so chances are it will also work on the Fonera.

Leave a Reply

Please read the comment policy before posting a comment.