[isf-wifidog] Ok, the real message: RADIUS, MySQL, Postgres, reimplementations & the Sherbrooke meeting and other things

Helene Gauthier webmaster at zapbas-saint-laurent.org
Jeu 28 Mai 14:20:44 EDT 2009



De l’avis général, MySQL est la meilleure alternative


La majorité des intervenants s’entende pour dire que c’est difficile … voire même impossible de faire de la réplication avec PostGreSQL


Québec n’a pas réussi à mettre la réplication en place malgré de nombreux tests


Je te proposerais de tester préalablement PostgreSQL… que tu arrives ou non à une impasse, tu testeras ensuite avec MySQL


Tu pourras ainsi donner ton avis sur :


quelle base de données est la plus conviviale ? et pourquoi ?


est-il réalisable de penser faire la réplication d’ici septembre 2009 sur l’ensemble des CSF avec PostgreSQL et pourquoi ? avec  MySQL et pourquoi ?


Helene Gauthier







De : wifidog-bounces at listes.ilesansfil.org [mailto:wifidog-bounces at listes.ilesansfil.org] De la part de Jimmy Godbout
Envoyé : 28 mai 2009 12:54
À : WiFiDog Captive Portal
Objet : Re: [isf-wifidog] Ok, the real message: RADIUS, MySQL, Postgres, reimplementations & the Sherbrooke meeting and other things



Quel serait la database idéale ? Je vais faire des tests avec différentes versions de BD.



-----Original Message-----
From: benoitg at coeus.ca
Sent: Tue, 26 May 2009 19:51:53 -0400
To: wifidog at listes.ilesansfil.org
Subject: [isf-wifidog] Ok, the real message: RADIUS, MySQL, Postgres, reimplementations & the Sherbrooke meeting and other things

On May 26, 2009, Marc Blanchet wrote:
> wlanmac a écrit :
> > The question becomes, where should it be optional?
> >
> > Optional in the way it is now where it is used by the portal to be
> > compatible on the roaming side,
> >
> > AP/WiFiDog ---> <wifidog protocol> ---> WiFiDog Portal ---> RADIUS
> >
> > or optional for access provisioning to the router, for device / access
> > controller compatibility?
> >
> > AP/NAS ---> <RADIUS> ---> <something?> ---> WiFiDog Portal
> >
> > (Note: "NAS" for Network Access Server is a more generic term for access
> > controller in RADIUS documentation).
> >
> > A concept I have been thinking of would be something along the lines of
> > what Marc suggested. A proxy to map the AAA features of a NAS with the
> > WiFiDog HTTP protocol.
> >
> > Such that you might have:
> >
> > AP/NAS/Proxy ---> <wifidog> ---> WiFiDog Portal (to support old-school
> > sites)
> >
> > AP/NAS ---> <RADIUS> ---> Proxy/WiFiDog Portal (to support standard
> > access controllers)
> >
> > AP/NAS ---> <RADIUS> ---> <FreeRADIUS/SuperProxy> ---> Multiple WiFiDog
> > Portals based on Realm (the future)
> >
> > It's just an idea... but, could be a way of achieving the best of all
> > worlds, RADIUS-wise.
> you were just 5 minutes faster than me to write almost exact same
> drawings... a few minor edits below:
> this is my view of v2:
> "v1" AP -> wifidog-http -> v2 wifidog portal -> radius backend
> "v2" AP -> radius -> radius backend

You just essentially described CoovaChilli, but even with Coova there is still a piece of software to run on the leaf node, and a piece of software other than the RADIUS server to run on the server for full full functionnality. But if this is what you want, continuing with wifidog is just plain silly (assuming you use it) Coova is really good at what it does, and I believe it's main developer has exactly the same RADIUS-centric vision as you do.

It is funny however to see people suggest using RADIUS would make everything 
simpler to understand and implement. Unless someone spends most of his 
workdays working with RADIUS and writing FreeRADIUS plugins, I find that well beyond hard to believe. The mere LIST of distinct RFCs specific to RADIUS 
(http://en.wikipedia.org/wiki/RADIUS#RFCs) is half as long as the current wifidog protocol documentation: http://dev.wifidog.org/wiki/doc/developer/WiFiDogProtocol_V1

"Using RADIUS" is just about as precise in the wifidog project context as "Using http". There is the RADIUS UDP transport protocol then there are the various extensions, the RADIUS servers themselves, and all the modules and backends to get them to help you manage networks.

But when people say "let's use RADIUS" they probably have a much more precise meaning in mind. I think the reasoning goes like this: 
1- RADIUS is designed to manage users
2- RADIUS databases are frequently replicated
3- RADIUS has provisions for roaming between servers
4- Several switches and access points have built-in RADIUS support
5- RADIUS is used to manage large networks, including wireless ones.
Therefore, RADIUS probably does everything wifidog does, and does it better.

The reasoning is logical. But indicates either a misunderstanding of what wifidog is, or of what RADIUS provides.

1- User login handling is a relatively minor aspect of WiFiDog. It's central to RADIUS. Minor for wifidog but with a rather unique twist: the focus managing hundreds of thousand users that are allowed to self-subscribe and freely use the network, and then manage abuse of any single user without human intervention. Furthermore, the system has to track displaying content taking while taking into account which content the user has previously seen and where. Basically manage users as a big set, not individually (for the longest time, there wasn't even a facility to delete them, because in general it's undesirable to do so). If user's can't self-subscribe WiFiDog just plain doesn't care where the username-password pairs come from. RADIUS, LDAP, whatever. But even if we did rely entirely on a mandatory RADIUS backend, we would have had to write a user management backend for said RADIUS server to provide all these services. Relying on a RADIUS server and the use it as a dumb user/password store just doesn't make sense.
2- Quite true. However (unlike LDAP) database replication isn't part of the scope of the RADIUS specs. A typical radius server with a MySQL backend will use the mysql replication facilities to replicate. A radius server with a LDAP backend will use LDAP facilities to replicate. A RADIUS server with a file backend will use rsync. You get the idea. In any event, one cannot replicate two radius servers running different RADIUS software together simply because they are RADIUS servers.
3- It does, but it doesn't make figuring out the realm automatically (which is the real UI problem) any simpler or harder. That being said, using RADIUS for inter-auth server communication would make perfect sense, if it didn't suddenly add the dependency of an operational freeradius server, and of writing a freeradius module to authenticate against the wifidog database (although this would potentially make WPA encryption available at the same time)
4- They do, but that doesn't make them support dynamic walled garden, equitable bandwidth share between users, centralizing gateway configuration and firewall settings automatic or even any easier. If they did, there would hardly be any need for CoovaChili or the wifidog gateway either. 
5- That's a valid point. I personally believe the techniques applied by the ISPs (which are the main force behind RADIUS) are frequently insufficient and conter-productive. But that is very much open for debate.

The RADIUS protocol is a UDP protocol designed to transmit network credentials and usage statistics between an authentication source (typically a database of some kind) and a network access controler (a modem, a switch, these days a wifi router). As such it's applicability may seem obvious. Various extentions define additional attributes (but have very uneven level of support between implementations). 

------- The gateway ------

Indeed at the genesis of the wifidog project (End of 2003), Using the RADIUS protocol for gw-auth server communication was considered, and rejected. The protocol is simple (more so than http in many ways). But It clashed with points 1 and 4 of the original gateway requirements which were:

1- Be embeddable in even the smallest systems. Indeed we got the gateway down to 16kB.
2- Work behind any proxy or firewall, and be completely unaffected by NAT
3- Offload as much intelligence and processing to the auth server.
4- Only require a web server to implement an auth server.
5- No dependencies on crypto libs if at all possible.

Indeed the several alternate auth server implementations for specific needs have been written for the WiFiDog 1.0 protocol.

While this is not the focus of the current sprints by the Quebec wireless groups that agreed to put their efforts in common a few weeks ago, there is a clear need to have a WiFiDog 2.0 protocol, to support the following features:

1- Have all the nodes run identical images, INCLUDING configuration
2- Dynamic walled garden (For OpenID support, mesh backhaul nodes whitelisting, allowing web mashups on the login page)
3- Allow storing static and dynamic node-specific firewall rules on the auth server if for no other reason than to finally get rid of the TrustedMacList gateway config parameter)
4- Dynamic bandwidth management (as pecentage of available bandwidth, not just an absolute number of bytes/sec for a class of user)
5- Allow the auth server to tell the gateway when to contact it next.

This COULD all be implemented as RADIUS protocol extensions, but once again RADIUS wouldn't make it any simpler. 

------------ The auth server ---------------

Now what was the original focus of the auth server?
1- Allow people to self-subscribe, and self-verify their email address
2- Present location specific content to the user, using content stored OUTSIDE the auth server as much as possible.
3- Be fully multilingual
4- Manage abuse

Since then it's grown in scope, and we didn't fill these four original goals to anyone's complete satisfaction. Explaining exactly what we were trying to build in early 2004, and the successes and failures we had could fill a book. Yet it can be illustrated simply with today's technology:

We were trying to build a fully location and view history aware Liferay.


But even in 2009 we haven't yet consolidated on a standardized, language neutral portlet specifications with widely available open server implementations. But it is (excruciatingly) slowly getting there.

There was a decision in Sherbrooke to write a specification and API to completely decouple the content serving from the auth server. This will be very, very interesting work. It will also be very, very difficult work to make it a reality. I've done standard writing work before. Avoiding groupthink and allowing implementations that are so lowest common denominator as to make using the standard pointless, while making it easy to implement and popular is never easy. 


It will not make the auth server simpler either (on the contrary). But it must be done if we are to have any kind of meaningful l impact on helping making delivering truly location specific content to end users a reality. It's the only MAJOR technical undertaking agreed on in Sherbrooke, but it's an important one to the future of the project.

----- PostGres vs MySql ------

As for PostGres vs MySql, I won't get into that debate again. It's pointless.


I just give some usefull information for anyone who intends to revisit the issue:
-MySql used to be supported, but support for it was dropped because we couldn't find ANY developer to maintain it.
- http://bugs.mysql.com/bug.php?id=2364 would make automatic schema upgrade of live servers like we have now excruciatingly slow once you have a fairly large connetion history. Not to mention that database modification statements are not transactional in MySql, so we'd have to do a lot more testing, so we couldn't do schema updates as often as we do now.
- The inability to determine which row is kept in in GROUP BY statements in mysql will require architecting the statistics subsystem very differently if one wants to maintain comparable performance. 
- If I were to start from scratch today, I'd probably investigate the feasibility of basing the authserver on something like CouchDB (real support for asynchronous multi-master replication, map-reduce, keep real ACID support at the cost of being non-relational). 

In any event I'm very much open to a project to re-add MySql support IF it brings new developers along (i know it WILL bring new users along, but it's a disservice to everyone if we can't support them like last time we supported MySql). 

Teaching both PostGres ans MySql classes and doing data-mining on both, I must say that if someone manages to make all the database queries MySql and Postgres compliant while keeping dynamic schema upgrade and near realtime stats on a 1gb + wifidog dataset I'll be quite favorably impressed ;) But it IS doable, with some compromises.

------ Why not start over ------

I've said most of this while in Sherbrooke, and a consensus was reached. 


Yes, the wifidog auth server is very big.

Writing a basic auth server that displays a different web age to every user is a < 1week project in a modern dynamic language (Python, Ruby and even PHP) for a competent programmer. In fact that's the time it took one person to write the original implementation of the auth server.

Writing a generic auth server that is easy to install, and meets the needs of disparate communities and is structured to accept outside contributions is something completely different.

Most of the complexity in wifidog isn't in user management, or gateway interaction, or any of the things I see people debate when they say that it would be easier to start from scratch. Most of the complexity is in:

-Location-aware and history-aware content delivery and management
-User permissions
-Web based installer, dependency install and automatic schema upgrade
-The statistics subsystem
-Pervasive multilingual support
-Making everything generic enough that we don't end up with spaghetti code.


Off course, wifidog support many, many other features, but code-wise they are comparably very small.

If you have specific needs, ask how we think they could be integrated in wifidog without causing problems to anyone else. If you can't do it, by all means do roll your own. With any luck, we'll have some good ideas to steal from ;) 

------ What will happen in the next few months -----


In the meantime, I expect very good things to come from the meeting in Sherbrooke. I've seen open minded people with actual resources to commit who agreed on cooperation and realistic and achievable goals. The patches are already rolling in. 


I'll do my best to help the new contributors, and integrate the patches. But very soon, other people will do most of that job as well.


If I become "just another contributor" I'll consider that a major victory for the project. 



Benoit Grégoire


Free Online Photosharing - Share your photos online with your friends and family!
Visit http://www.inbox.com/photosharing to find out more!

-------------- section suivante --------------
Une pièce jointe HTML a été nettoyée...
URL: http://listes.ilesansfil.org/pipermail/wifidog/attachments/20090528/283e2a6a/attachment-0001.htm 

Plus d'informations sur la liste de diffusion WiFiDog