[TAG] [Fwd:] In RI federal court -- Harvard vs. the RIAA

[Rick Moen]

Quoting Ben Okopnik (ben at linuxgazette.net):

> Sorry, I obviously didn't make myself clear: that's far too broad an
> interpretation of "carry", and one I didn't expect you to make. Perhaps
> I should have said "allow". What I meant was that pulling down, say,
> half a dozen NGs a day went from a few hundred KB to hundreds of MB -
> and a dozen people reading their favorite froups now constituted
> significant traffic.

You're still not making a whole lot of sense.  People reading netnews
from a remote news server via NNTP don't "pull down newsgroups"; they
merely remotely read individual articles fetched upon demand into the
news reader's buffer for reading and possible reply, that are then
deallocated.   That is, in the usual access model, there isn't a local
news spool at all; the news spool is on the remote news server.

When I do "tin -r -g nntp2.tsoft.net" to read from SuperNews, for
example, the spool is on nntp2.tsoft.net AKA the SuperNews news server.

It is _possible_ to have a non-default configuration where you (or your
local sysadmin) does out of your way to have some sort of local news
spool:  The most common way is using leafnode, a "news proxy" that
watches which newsgroups local users are fetching from the remote spool
and then institutes periodic fetching of a local "cache" spool via NNTP
of just the groups that have been recently read, ceasing to do those
fetches, per-group, whenever local users have ceased to read them.

If what you say were true, given that there are several Usenet users on
my pathetically overburdened 500MHz PIII server that is perennially
running out of disk space and lives on the end of a really slow aDSL
line, the linuxmafia.com box would have long ago collapsed.  Which
further illustrates that you really don't know this subject -- which we
already knew from earlier posts, plus such howlers as:

> comp.os.linux.misc reputedly has 5325 members; comp.os.linux.setup,
> 2164.

WTF?  I'm sorry, but it's sadly apparent that you really have no idea
_why_ these statements inherently make no sense whatsoever.  It's
difficult to know even where I should start.
start?

I guess I have to start with the fact that Usenet (and similar
newsgroup-sharing systems) is radically decentralised.  Some machine to
which you have local or NNTP posting access has a spool managed by a
news daemon, which accepts posted articles, each of which gets an
article ID in its header that is designed to be globally unique across
any and all hosts that might carry the group.  The news server caches
your article in the local spool and also broadcasts it out to all the
other news servers with which it peers, which examine the article ID,
make sure it hasn't already been received, and broadcasts it onwards to
_its_ peers, with the result that the article "floods" across the globe
among news hosts carrying the group.  Each local server has a local,
idiosyncratic-to-that-news-host expiration policy determining how long
and according to what criteria articles are retained in the local cache.

The point is that there is no physical point at which it is functionally
possible to deterministically measure the "members" of a newsgroup.
It's conceivable you might be able to analyse logfiles of a single news
server, to attempt to guess how many purportely unique local + remote
(NNTP) users had _posted_ to the news group during the period covered by
its spool and logfiles.  Similarly, you could probably in theory
determine how many purportedly unique local users had _read_ articles in
the local spool of that group within a specific time span -- but it
would not be possible to be even that accurate regarding remote
_readers_ of the news group during a time span, since you have only
their connecting IPs.

However, even at that, the news machine has absolutely no idea what
other users are reading and posting on other news machines, regardless
of time period, because it has no idea about host-to-host peering
arrangements beyond those it, itself, has with its immediate neighbours.  
Its knowledge is limited to what's in its current logs, and its spool of
(unexpired) articles.

I suspect that what you did, to come up with your number of how many
"members" two newsgroups have, is to use some (unspecified) front-end
software to some single (unspecified) news host's spool of
(not-yet-expired) posted articles, querying for how many avowedly unique
posters wrote the still-spooled (not yet expired out) articles in the
local spool of that group.  Which is of course nothing at all like the
group's "members" in any meaningful sense.

Or you could have meant something else -- and failed to say what you
meant -- but it really doesn't matter because, since no physical place
on the Internet exists where "members" of such a radically distributed
messaging medium can be meaningfully tallied, the numbers cited
inherently cannot mean anything useful.


I just went through several paragraphs of patient explanation to explain
how netnews articles propagate, because in this case the person
confidently spouting gibberish is my fried Ben.  Normally, I'd just
smile indulgently, nod noncommitally, and rely on the ignorant and
misinformed person walking away and becoming Somebody Else's Problem.

 
> Not at all; see my point above. To clarify even further, I'm not
> talking about carrying a feed: I'm talking about the situation that's
> common in most of the world outside the US, in which downloading
> several hundred MB per day is a Big Deal and negatively affects
> everyone else at the end of that pipe.

You're repeating the same gibberish.  

An NNTP connection to, e.g., nntp2.tsoft.net does not "download"
newsgroups; it pulls down current headers in the remote spool, removes
from the local display the ones your newsreader believes you have
already "read" on that particular remote spool, and shows you the
headers of the rest.  One at a time, until you're tired of reading
netnews, you can now spend your time pulling down via NNTP and
buffering/reading individual articles, one at a time, moving from one to
the next or by threads of interest or other criteria you choose.

If you were correct, and any of my news-reading users including myself
were "downloading several hundred MB per day", that would probably put
my overburdened PIII and aDSL line over the edge and DoS my Internet
presence to death.  It of course doesn't happen, because (thankfully)
your understanding of the technology is all wrong.

Of course, my poor little aDSL _could_ be endangered by a lot more
worrisome threats, e.g., if I had users who were pulling down big file
collections, or e-mailing people PowerPoint presentations.  Luckily I
don't -- but practically all businesses do.

 
> > > Ubuntu, for example. 
> > 
> > Bad example:  The lion's share of their communication is on their
> > (very stereotypically awful) Web forums -- not their mailing lists.
> 
> No, good example: it's not being done on USENET, as it would have been
> in the old days. Whether awful or not, their Web forums have beaten
> out USENET by a huge margin.

As to _quantity_.  Ergo, you can get wrong answers in friendly pastel
colours and with avatar icons, in vast numbers.  Practically all of the
people I encounter who've screwed up their Ubuntu system with badly
installed proprietary A/V software, accellerated video drivers, wireless
drivers, and codecs installed from downloaded tarballs or via the buggy
EasyUbuntu or Automatix scripts were cheerfully advised to do that via
Ubuntu's Web forums.

To their credit, the Ubuntu community leaders are aware of this brain
damage among the general run of people who post to its Web forums, which
is why one of the FAQ pages for those forums tries forlornly to
compensate:

    Warning Regarding Alternative Installation Methods

    WARNING: EasyUbuntu and Automatix are third-party utilities for
    installing the most commonly requested applications in some
    Debian-based distributions. THEY ARE NOT SUPPORTED OR RECOMMENDED BY
    UBUNTU. While they work well for many users, they have also been
    known to corrupt systems and leave them in a state where they cannot
    be upgraded to a later Ubuntu release.

Practically anybody's advise on the comp.os.linux.* newsgroups (except
.advocacy) is, by contrast, likely to actually _help_ -- as opposed to
crippling your system and compromising its security.

 
> > However, I'd speculate that you're still slightly more likely to get
> > a _good_ answer to an Ubuntu question on comp.os.linux.misc than you
> > are in either of those other two places.
>  
> When I search the Web for an answer to an Ubuntu problem, those
> "awful" Web forums are the first thing that pop up - and they usually
> have a pretty reasonable answer, or set of answers, for what I'm
> looking for.

I've seen those answers, and have lost a great deal of time dealing with
the damage caused by them.  I therefore call bullshit on your assertion.

> > Yahoo Groups are a "success" in pretty much the same way fruitflies
> > are.  OK as to _quantity_.
>  
> Heh. Lovely metaphor there, Rick. How is USENET different, again?
> Except for being _less_ successful than fruitflies?

I have not yet seen a Yahoo Group that has information worth reading. 
And yes, they're bountiful and buzz about energetically.


> Rick, I'm sure that you do a great job of the above. Now, with that
> granted: how many USENET feeds are administered by sysadmins of your
> quality and experience?  Another point, one that intersects with this
> one: it takes a lot of skill to do that job well. It takes little to
> no skill to manage a Yahoo group. Which one is going to be more
> popular?

The minimum number of competent news server administrators that Usenet
requires to function well enough is approximately four:  one each for
UsenetServer.com, Supernews, Giganews, and Usenet.com.  Each of those
has a couple of large machines running the INN daemon, which requires
(for hosting a full Usenet feed) a seriously beefy machine and
significant care and maintenance.  

Fortunately, the rest of us don't have to administer any news servers
whatsoever, any more than we or our users need to "download several
hundred MB per day", or any of the rest of it.  The problem is that --
oh, yeah -- you apparently don't understand the technology at all, and
at this point I don't expect that you're making a serious attempt to
listen to my explanation, either.


My patience for explaining _even_ to good friends who've gotten deep
into a spouting-bullshit presentation mode is strictly limited, so I'm
going to cut both of us off, at this point, and close with a brief
description of _other_ aspects of netnews technology (that I hadn't 
detailed earlier) that underlay my characterisation of "the message
semantics and architecture of NNTP newsgroups" as "the best we've yet
achieved" (and by "we" I mean the entire world of online discourse):

1.  Reply being distinct from followup.  All of those stupid mailing
list flamewars you see about dumb measures such as Reply-To munging were
completely unnecessary on netnews media (Usenet _and other_), because
the technology design provides in advance for two correctly routed,
conceptually distinct response modes:  "reply" generates an
off-newsgroup communication back to a poster via e-mail.  "followup"
generates an on-newsgroup response to (by default) the same set of
groups as the earlier post.

There are never any problems with confusing the two operations, no
haplessly held postings from CC'd individuals who sent a copy to the
discussion forum by accident, no accidentally disclosed private
communications sent into public by mistake to the correspondent's
horror, no accidentally getting sent two copies, and so on.  This is a
difference that can be fully appreciated probably only by mailing list
administrators, who have to endure and witness the effects of its being
missing, all the time.

2.  A properly integrated, robust mechanism for handling crossposts and
to recommending particular groups for any subsequent follow-ups.  Again,
this is something whose _absence_ in all mailing list software causes
immense problems, particularly for us mailing list administrators,
especially when -- as happens continually -- some idiot decides to
crosspost among several separate mailing lists with few-to-no common
members and all having subscriber-address-only posting policies, and
then various of the recipient on the several mailing lists attempt to
respond.  This situation, which we listadmins must deal with regularly,
results at best in large amounts of held mail and at worst in multi-list
flamewars.

3.  The concept of a spool subject to a local expiration policy.  
Unlike the case with mailing list postings where the post you as a
subscriber have finally decided to reply to is inevitably the one you've
recently deleted, with a newsgroup, any posting from any date remains
available for correctly formatted, correctly quoted followups as long as
it hasn't yet been expired from the spool.  (If you're a frequent
newsgroup user, you end up setting up something like leafnode or GNUS on
your own workstation, and then you get to have your own spool of just
the groups you must care about, subject to your _own_ expiration
policy.)

Which reminds me:

4. Decentralised flooding of posts to an arbitrary number of spools on
peered, cooperating news servers (and copies of leafnode & similar).
This advantage means that the newsgroup is immune to collapse if a
single central host goes down, because there is no single point of
failure and complete sets of everything needed to run and populate and
archive the group exist all over the place.  (The correctly designed
header for globally unique article IDs, to facilitate the decentralised
operation, means it's essentially impossible to suffer sending loops, as
mailing lists occasionally do.)


Which reminds me:

There once was a time, starting somewhere around 1997, when Nicholas
Petreley ran the very best, most vibrant, most literate and clueful
set of online forums for discussion of Linux that's yet existed.  He did
it using the open source INN daemon on host forum.linuxworld.com, i.e.,
a set of private newsgroups -- and was utterly brilliant for years,
until IDG summarily discontinued Petreley's incarnation of
LinuxWorld.com (the first and still best one), laid off Petreley, and
converted the whole lot over to a set of stillborn (but
advertising-friendly) Web forums in 2001.

If the rest of us (participants) had been a bit more wary, we could
easily have kept all postings in separate spools of our own, and
kept the forums alive independently on other computing resources.
Instead, the community established on them evaporated immediately and
never came back.


5.  Real threading and (variously) killfiles or scoring systems, in
newsreader client software.  Again, the standard for how to do these
things was so well established that everyone gets to do them right,
unlike the hacks in many MUAs and the utter failure to even try to do
those things in Web forums.

6.  Inability to festoon everything with advertising (which might be
part of why IDG shut the forum.linuxworld.com forums down).

I'm hardly the only person who's held these views.  Off the top of my
head, I can spot Randall Schwartz and Brian Aker doing so:  
http://krow.livejournal.com/396948.html?thread=1594260

And that's not counting the degree to which the _quality_ of what's
posted has in general been horribly degraded in the move towards mailing
lists and (much worse) Web forums.  It's a tragedy.

Frankly, the extent to which we all lose by _not_ using that technology
-- newsgroups, I mean, regardless of whether they're on Usenet -- 
and I count both the technological loss and the debasement (as a 
general rule) of content is staggering and past dispute.

So, go ahead and dispute it, but you're simply and massively wrong, and
I absolutely do not, and will not in the future, respect that view in
the slightest:  I've seen how much better we can do, and have a good
grasp on _why_ it's better.