[isf-wifidog] Ok, the real message: RADIUS, MySQL, Postgres, reimplementations & the Sherbrooke meeting and other things

[Michael Lenczner]

I think that was a very helpful email for people that didn't know the
origins and goals of the project and - like me - are trying to think
though Radius but don't know enough about it.  Thanks!

The one thing I would want to add - or correct under the response to
"> 1- RADIUS is designed to manage users"

Benoit says:
"> 1- User login handling is a relatively minor aspect of WiFiDog. It's central
> to RADIUS. Minor for wifidog but with a rather unique twist: the focus
> managing hundreds of thousand users that are allowed to self-subscribe and
> freely use the network, and then manage abuse of any single user without
> human intervention. Furthermore, the system has to track displaying content
> taking while taking into account which content the user has previously seen
> and where. Basically manage users as a big set, not individually (for the
> longest time, there wasn't even a facility to delete them, because in
> general it's undesirable to do so). If user's can't self-subscribe WiFiDog
> just plain doesn't care where the username-password pairs come from. RADIUS,
> LDAP, whatever. But even if we did rely entirely on a mandatory RADIUS
> backend, we would have had to write a user management backend for said
> RADIUS server to provide all these services. Relying on a RADIUS server and
> the use it as a dumb user/password store just doesn't make sense."

There are three points here.

The first is that users can self-subscribe.  I'm curious to know if
and how Radius allows for this.

Secondly, managing users as a big set, not as individually. I don't
know how Wifidog currently accomplishes this, but I know Benoit well
enough to know that he's referring to something specific.  Any help?

Thirdly - and most importantly - Content history. I don't think that
we should assume that people are interested in retaining content
history (when and where a user has seen something).   I *love* that
feature and have seen one awesome example where it was used. I also
know that we used it for showing an updated TOS (which could have been
accomplished in other ways).  However, I don't care if it's in the
next version.  I would rather not have it if it adds complexity.  In
my opinion it's a far-off feature that may or may never really become
useful and I would rather concentrate on allowing much simpler
user--content interactions.  And personally, I have never heard anyone
from any other group request that functionality.

So I would not agree that we would need to recreate it through Radius
extensions.  That point is referred to again at the bottom of Benoit's
email as part of the original goals of Wifidog.  I'm doubtful that it
should be retained as a necessary feature of ZapStack / Wifidog2

Similarly, I'm curious to know if all of the groups believe that the
multilingual support needs to be as powerful as it was built in
wifidog if the trade of is increased complexity in the code (which I
don't know if it is) and increased complexity in the UI / UX.  I can
picture us lowering the bar for the ZapStack / Wifidog2 by not
requiring the same richness to that feature - ie - not allowing every
individual piece of content to have multiple versions according to
language. Very un-montreal of me, but I'm definitely ready to give up
on the pie-in-the-sky version of an awesomely powerful _and_ easily
usable Wifidog in exchange for something a lot more friendly and
admittedly less cool. :-)

Maybe minor points, but I'm enjoying the process of everyone becoming
more clear and more knowledgable about the decisions / options we have
in front of us.  :-)


2009/5/26 Benoit Grégoire <benoitg at coeus.ca>:
> On May 26, 2009, Marc Blanchet wrote:
>> wlanmac a écrit :
>> > The question becomes, where should it be optional?
>> >
>> > Optional in the way it is now where it is used by the portal to be
>> > compatible on the roaming side,
>> >
>> > AP/WiFiDog ---> <wifidog protocol> ---> WiFiDog Portal ---> RADIUS
>> >
>> > or optional for access provisioning to the router, for device / access
>> > controller compatibility?
>> >
>> > AP/NAS ---> <RADIUS> ---> <something?> ---> WiFiDog Portal
>> >
>> > (Note: "NAS" for Network Access Server is a more generic term for access
>> > controller in RADIUS documentation).
>> >
>> > A concept I have been thinking of would be something along the lines of
>> > what Marc suggested. A proxy to map the AAA features of a NAS with the
>> > WiFiDog HTTP protocol.
>> >
>> > Such that you might have:
>> >
>> > AP/NAS/Proxy ---> <wifidog> ---> WiFiDog Portal (to support old-school
>> > sites)
>> >
>> > AP/NAS ---> <RADIUS> ---> Proxy/WiFiDog Portal (to support standard
>> > access controllers)
>> >
>> > AP/NAS ---> <RADIUS> ---> <FreeRADIUS/SuperProxy> ---> Multiple WiFiDog
>> > Portals based on Realm (the future)
>> >
>> > It's just an idea... but, could be a way of achieving the best of all
>> > worlds, RADIUS-wise.
>>
>> you were just 5 minutes faster than me to write almost exact same
>> drawings... a few minor edits below:
>>
>> this is my view of v2:
>>
>> "v1" AP -> wifidog-http -> v2 wifidog portal -> radius backend
>> "v2" AP -> radius -> radius backend
>
> You just essentially described CoovaChilli, but even with Coova there is
> still a piece of software to run on the leaf node, and a piece of software
> other than the RADIUS server to run on the server for full full
> functionnality. But if this is what you want, continuing with wifidog is
> just plain silly (assuming you use it) Coova is really good at what it does,
> and I believe it's main developer has exactly the same RADIUS-centric vision
> as you do.
>
> It is funny however to see people suggest using RADIUS would make everything
> simpler to understand and implement. Unless someone spends most of his
> workdays working with RADIUS and writing FreeRADIUS plugins, I find that
> well beyond hard to believe. The mere LIST of distinct RFCs specific to
> RADIUS
> (http://en.wikipedia.org/wiki/RADIUS#RFCs) is half as long as the current
> wifidog protocol documentation:
> http://dev.wifidog.org/wiki/doc/developer/WiFiDogProtocol_V1
>
> "Using RADIUS" is just about as precise in the wifidog project context as
> "Using http". There is the RADIUS UDP transport protocol then there are the
> various extensions, the RADIUS servers themselves, and all the modules and
> backends to get them to help you manage networks.
>
> But when people say "let's use RADIUS" they probably have a much more
> precise meaning in mind. I think the reasoning goes like this:
> 1- RADIUS is designed to manage users
> 2- RADIUS databases are frequently replicated
> 3- RADIUS has provisions for roaming between servers
> 4- Several switches and access points have built-in RADIUS support
> 5- RADIUS is used to manage large networks, including wireless ones.
> Therefore, RADIUS probably does everything wifidog does, and does it better.
>
> The reasoning is logical. But indicates either a misunderstanding of what
> wifidog is, or of what RADIUS provides.
>
> 1- User login handling is a relatively minor aspect of WiFiDog. It's central
> to RADIUS. Minor for wifidog but with a rather unique twist: the focus
> managing hundreds of thousand users that are allowed to self-subscribe and
> freely use the network, and then manage abuse of any single user without
> human intervention. Furthermore, the system has to track displaying content
> taking while taking into account which content the user has previously seen
> and where. Basically manage users as a big set, not individually (for the
> longest time, there wasn't even a facility to delete them, because in
> general it's undesirable to do so). If user's can't self-subscribe WiFiDog
> just plain doesn't care where the username-password pairs come from. RADIUS,
> LDAP, whatever. But even if we did rely entirely on a mandatory RADIUS
> backend, we would have had to write a user management backend for said
> RADIUS server to provide all these services. Relying on a RADIUS server and
> the use it as a dumb user/password store just doesn't make sense.
> 2- Quite true. However (unlike LDAP) database replication isn't part of the
> scope of the RADIUS specs. A typical radius server with a MySQL backend will
> use the mysql replication facilities to replicate. A radius server with a
> LDAP backend will use LDAP facilities to replicate. A RADIUS server with a
> file backend will use rsync. You get the idea. In any event, one cannot
> replicate two radius servers running different RADIUS software together
> simply because they are RADIUS servers.
> 3- It does, but it doesn't make figuring out the realm automatically (which
> is the real UI problem) any simpler or harder. That being said, using RADIUS
> for inter-auth server communication would make perfect sense, if it didn't
> suddenly add the dependency of an operational freeradius server, and of
> writing a freeradius module to authenticate against the wifidog database
> (although this would potentially make WPA encryption available at the same
> time)
> 4- They do, but that doesn't make them support dynamic walled garden,
> equitable bandwidth share between users, centralizing gateway configuration
> and firewall settings automatic or even any easier. If they did, there would
> hardly be any need for CoovaChili or the wifidog gateway either.
> 5- That's a valid point. I personally believe the techniques applied by the
> ISPs (which are the main force behind RADIUS) are frequently insufficient
> and conter-productive. But that is very much open for debate.
>
> The RADIUS protocol is a UDP protocol designed to transmit network
> credentials and usage statistics between an authentication source (typically
> a database of some kind) and a network access controler (a modem, a switch,
> these days a wifi router). As such it's applicability may seem obvious.
> Various extentions define additional attributes (but have very uneven level
> of support between implementations).
>
> ------- The gateway ------
>
> Indeed at the genesis of the wifidog project (End of 2003), Using the RADIUS
> protocol for gw-auth server communication was considered, and rejected. The
> protocol is simple (more so than http in many ways). But It clashed with
> points 1 and 4 of the original gateway requirements which were:
>
> 1- Be embeddable in even the smallest systems. Indeed we got the gateway
> down to 16kB.
> 2- Work behind any proxy or firewall, and be completely unaffected by NAT
> 3- Offload as much intelligence and processing to the auth server.
> 4- Only require a web server to implement an auth server.
> 5- No dependencies on crypto libs if at all possible.
>
> Indeed the several alternate auth server implementations for specific needs
> have been written for the WiFiDog 1.0 protocol.
>
> While this is not the focus of the current sprints by the Quebec wireless
> groups that agreed to put their efforts in common a few weeks ago, there is
> a clear need to have a WiFiDog 2.0 protocol, to support the following
> features:
>
> 1- Have all the nodes run identical images, INCLUDING configuration
> 2- Dynamic walled garden (For OpenID support, mesh backhaul nodes
> whitelisting, allowing web mashups on the login page)
> 3- Allow storing static and dynamic node-specific firewall rules on the auth
> server if for no other reason than to finally get rid of the TrustedMacList
> gateway config parameter)
> 4- Dynamic bandwidth management (as pecentage of available bandwidth, not
> just an absolute number of bytes/sec for a class of user)
> 5- Allow the auth server to tell the gateway when to contact it next.
>
> This COULD all be implemented as RADIUS protocol extensions, but once again
> RADIUS wouldn't make it any simpler.
>
> ------------ The auth server ---------------
>
> Now what was the original focus of the auth server?
> 1- Allow people to self-subscribe, and self-verify their email address
> 2- Present location specific content to the user, using content stored
> OUTSIDE the auth server as much as possible.
> 3- Be fully multilingual
> 4- Manage abuse
>
> Since then it's grown in scope, and we didn't fill these four original goals
> to anyone's complete satisfaction. Explaining exactly what we were trying to
> build in early 2004, and the successes and failures we had could fill a
> book. Yet it can be illustrated simply with today's technology:
>
> We were trying to build a fully location and view history aware Liferay.
>
> But even in 2009 we haven't yet consolidated on a standardized, language
> neutral portlet specifications with widely available open server
> implementations. But it is (excruciatingly) slowly getting there.
>
> There was a decision in Sherbrooke to write a specification and API to
> completely decouple the content serving from the auth server. This will be
> very, very interesting work. It will also be very, very difficult work to
> make it a reality. I've done standard writing work before. Avoiding
> groupthink and allowing implementations that are so lowest common
> denominator as to make using the standard pointless, while making it easy to
> implement and popular is never easy.
>
> It will not make the auth server simpler either (on the contrary). But it
> must be done if we are to have any kind of meaningful l impact on helping
> making delivering truly location specific content to end users a reality.
> It's the only MAJOR technical undertaking agreed on in Sherbrooke, but it's
> an important one to the future of the project.
>
> ----- PostGres vs MySql ------
>
> As for PostGres vs MySql, I won't get into that debate again. It's
> pointless.
>
> I just give some usefull information for anyone who intends to revisit the
> issue:
> -MySql used to be supported, but support for it was dropped because we
> couldn't find ANY developer to maintain it.
> - http://bugs.mysql.com/bug.php?id=2364 would make automatic schema upgrade
> of live servers like we have now excruciatingly slow once you have a fairly
> large connetion history. Not to mention that database modification
> statements are not transactional in MySql, so we'd have to do a lot more
> testing, so we couldn't do schema updates as often as we do now.
> - The inability to determine which row is kept in in GROUP BY statements in
> mysql will require architecting the statistics subsystem very differently if
> one wants to maintain comparable performance.
> - If I were to start from scratch today, I'd probably investigate the
> feasibility of basing the authserver on something like CouchDB (real support
> for asynchronous multi-master replication, map-reduce, keep real ACID
> support at the cost of being non-relational).
>
> In any event I'm very much open to a project to re-add MySql support IF it
> brings new developers along (i know it WILL bring new users along, but it's
> a disservice to everyone if we can't support them like last time we
> supported MySql).
>
> Teaching both PostGres ans MySql classes and doing data-mining on both, I
> must say that if someone manages to make all the database queries MySql and
> Postgres compliant while keeping dynamic schema upgrade and near realtime
> stats on a 1gb + wifidog dataset I'll be quite favorably impressed ;) But it
> IS doable, with some compromises.
>
> ------ Why not start over ------
>
> I've said most of this while in Sherbrooke, and a consensus was reached.
>
> Yes, the wifidog auth server is very big.
>
> Writing a basic auth server that displays a different web age to every user
> is a < 1week project in a modern dynamic language (Python, Ruby and even
> PHP) for a competent programmer. In fact that's the time it took one person
> to write the original implementation of the auth server.
>
> Writing a generic auth server that is easy to install, and meets the needs
> of disparate communities and is structured to accept outside contributions
> is something completely different.
>
> Most of the complexity in wifidog isn't in user management, or gateway
> interaction, or any of the things I see people debate when they say that it
> would be easier to start from scratch. Most of the complexity is in:
>
> -Location-aware and history-aware content delivery and management
> -User permissions
> -Web based installer, dependency install and automatic schema upgrade
> -The statistics subsystem
> -Pervasive multilingual support
> -Making everything generic enough that we don't end up with spaghetti code.
>
> Off course, wifidog support many, many other features, but code-wise they
> are comparably very small.
>
> If you have specific needs, ask how we think they could be integrated in
> wifidog without causing problems to anyone else. If you can't do it, by all
> means do roll your own. With any luck, we'll have some good ideas to steal
> from ;)
>
> ------ What will happen in the next few months -----
>
> In the meantime, I expect very good things to come from the meeting in
> Sherbrooke. I've seen open minded people with actual resources to commit who
> agreed on cooperation and realistic and achievable goals. The patches are
> already rolling in.
>
> I'll do my best to help the new contributors, and integrate the patches. But
> very soon, other people will do most of that job as well.
>
> If I become "just another contributor" I'll consider that a major victory
> for the project.
>
>
> --
> Benoit Grégoire
>
>
>
> _______________________________________________
> WiFiDog mailing list
> WiFiDog at listes.ilesansfil.org
> http://listes.ilesansfil.org/cgi-bin/mailman/listinfo/wifidog
>