About 6 months ago now I whipped up bowsah, a cute little app to scrape my web browser’s history file into a database and a surrounding rails app to munge the data.
So far it’s come in handy a few times for me searching for something I knew I’d seen on the web but couldn’t find, but mostly serves to measure how much time I actually spend on the internet. Here’s the past 24 weeks, equal to 164 days or ~5.5 months.
A few stats:
Is this too much? It may well be – but I don’t know. I’ve always used the computer lots, whether it was playing games when I was younger or reading lots of shit on the internet nowadays. I’m glad that I don’t spend all day watching TV or playing World of Warcraft, but could probably stand to see the honest-to-god light of day a bit more often or maybe even spend more time with actual people…
My browsing habits for the past 2 months. Pages per day is a crude metric, but good enough to get an idea of how many I look at per day.

I wrote a little script that scrapes ~/Library/Safari/History.plist and throws the the page title and url into a 150 line rails app which spits out basic info and lets me search through to try and find pages I remember but otherwise would have a hard time getting at.

The sqlite3 database is at 2MB after 2 months, I don’t think that 1MB/month is an unreasonable size to keep track of the pages I’ve been too. It might start getting unreasonable a few years from now, but nothing that can’t be handled then.
I don’t know why I wanted to have all the pages I go to logged into a system, even just for private use. Something about leaving a trail, having instant recall. I probably read more stuff on the internet than I do out of books, and I could remember for the most part what books I read, but there’s no way I could keep track of the thousands of webpages I go to every week.
It also just puts things in scale, I mean holy shit. In the past 7 days I’ve been to ~2300 pages. Not all different pages. In the whole 2 month period this thing’s been going, I’ve been to ~1900 different sites (judging by top level domain). That’s a lot of browsing.
The big idea that I feel is behind all this: when I’m 40 I’ll be able to come back and sit down and take a look at all the data I’ve left behind. Maybe figure out what my life had been like at some point in time. What websites did I visit 20 years ago today? Where did my interests lay, what kind of stuff did I like? It might end up being a dry, boring as hell legacy; but there’s potential for it to be interesting as well.
[root@ambrose:~] % cat >> /etc/hosts
127.0.0.1 bloglines.com
127.0.0.1 forum.textdrive.com
127.0.0.1 my.reddit.com
^C
Beat that internet.
Fiber to rural areas, living in the boonies with a fat pipe. May be my kind of life.
Albert László Barabási
How everything is connected to everything else and what it means for business, science, and everyday life.
A nice look at networks and how they’re manifesting themselves in the world today. I read the book for class, it was a good class, and I think that all the talking about it we did there will only detract from the mulling over it gets here.
I’m not sure I’m a huge fan of the pop science genre – you can tell that the guy would just love to pull out his formulas and really get down to things; but he holds back and the whole book has a sort of wishy washy tone to it. An interesting read, but nothing really popped out at me.
Tim Berners Lee (inventor of the www) on net neutrality:
The Internet is increasingly becoming the dominant medium binding us. The neutral communications medium is essential to our society. It is the basis of a fair competitive market economy. It is the basis of democracy, by which a community should decide what to do. It is the basis of science, by which humankind should decide what is true.
(A letter to my representatives in government)
Network Neutrality: “in order to promote innovation, network service providers such as telephone and cable internet companies should not be permitted to dictate how those networks are used (i.e., not permitted to ban certain types of programs, to ban certain types of devices connecting to the network, or to favor carriage of traffic to certain web sites over others).” (from wikipedia)
If the above was a bit much for you, and I’m indeed hoping that this isn’t being read by some intern, watch a quick introductory movie on the issue. Here’s a link which is likely to be much more coherent then my letter; and here’s a whole collection of material on the issue.
The crux of things is that:
Right now there are laws against this. There’s a bill surfacing in the House, sponsored by Rep. Joe Barton, which passed through committee a week or two ago, and is headed for a vote in the house.
This centralization entirely contradicts the entire notion upon which the internet was founded and has flourished. In less then 20 years, the internet has left a bigger mark on our world than any other technology of recent invention. Allowed to continue to evolve and better itself, as it has done ever since it’s creation, only God knows what will come of it in the years ahead. Who could have foreseen just 10 years ago what it would be today?
This is bad, bad, bad. The only reason the internet has evolved to the point it has is that it’s a decentralized system which allows equal access to everyone with a computer and connection. The web as we know it has emerged from nothing but a conglomeration of research universities and government organizations, into a massive engine for learning, communicating, and commerce. Above all the internet embodies equality and liberty, and who would entrust such important values to any corporation?
Delightful Neal Stephenson article on the laying of undersea cable, written about 10 years ago.
In 1870, a new cable was laid between England and France, and Napoleon III used it to send a congratulatory message to Queen Victoria. Hours later, a French fisherman hauled the cable up into his boat, identified it as either the tail of a sea monster or a new species of gold-bearing seaweed, and cut off a chunk to take home.
The rule of thumb for calculating revenue loss works like this: for every penny per minute that the long distance market will bear on a particular route, the loss of revenue, should FLAG be severed on that route, is about $3,000 a minute. So if calls on that route are a dime a minute, the damage is $30,000 a minute, and if calls are a dollar a minute, the damage is almost a third of a million dollars for every minute the cable is down. Upcoming advances in fiber bandwidth may push this figure, for some cables, past the million-dollar-a-minute mark.
It’s when a society plunders its ability to look over the horizon and into the future in order to get short-term gain – sometimes illusory gain – that it begins a long slide nearly impossible to reverse.
The collapse of the lighthouse must have been astonishing, like watching the World Trade Center fall over. But it took only a few seconds, and if you were looking the other way when it happened, you might have missed it entirely – you’d see nothing but blue breakers rolling in from the Mediterranean, hiding a field of ruins, quickly forgotten.
They [Alexander Graham Bell, et al] electrified the reeds in such a way that they generated not only acoustical vibrations but corresponding electrical ones. They sought to combine the electrical vibrations of all these reeds into one complicated waveform and feed it into one end of a cable. At the far end of the cable, they would feed the signal into an identical set of reeds. Each reed would vibrate in sympathy only with its counterpart on the other end of the wire, and by recording the pattern of vibrations exhibited by that reed, one could extract a Morse code message independent of the other messages being transmitted on the other reeds. For the price of one wire, you could send many simultaneous coded messages and have them all sort themselves out on the other end.
The world has actually been wired together by digital communications systems for a century and a half. Nothing that has happened during that time compares in its impact to the first exchange of messages between Queen Victoria and President Buchanan in 1858. That was so impressive that a mob of celebrants poured into the streets of New York and set fire to City Hall.
This has gone way too far: it’s playing on the radio right now. (1/7, 7:26pm). And again. Now it’s three times too far.
Last.fm bumped me to subscriber for the rest of the month, and man is this place cool. I really like that you can see who’s taken a look at your profile, and they will generate you an image of your weekly tracks:
Please, dear god, don’t let my generation be dubbed thusly.
But real interesting article about how the internet works socially for kids these days.
I really wish I could get into video games – they can be really fun.
But games today suck. My roomate and a friend are here playing midnight racer all the time, and I can’t believe they’re still at it. It’s the same goddamn thing over and over, the same five songs repeating, just you racing around cities at night.
I think Beattie is on to something here. I played a mud pretty obsessively for a year or two, and got really into it before I just quit one day and never went back. Around the same time I’d mastered Madden 2000 and could beat anyone I played. Now I can hardly play a game without getting bored.
Computer games of today are the same as they were five years ago. Better graphics, but the graphics still aren’t that good. In fact, the closer virtual humans come to real humans, the worse they get.
There needs to be something more compelling to gaming. A kid on my floor plays WoW all the time, and apart from me being all RPG’d out, I’d give it a try.
I think that the same thing that happened with social software, the enabling of user participation, is that usable content balloons so significantly when users are enabled to participate at the highest level. Skinnable computer games enable way more creativity then static console games. So they enjoy a greater popularity for a longer time. How many of you really kept playing Tony Hawk after beating it?
The thing I loved about that mud was that not only is it open ended, but because it’s text based you can write scripts to capture the environment of the game and do whatever you want with it.
Games today are at the communication level of phone calls – bi (maybe tri) directional. As games begin to approach the level the internet has reached, as they begin to take on multiple nodes of simultaneous communication, I think they’ll really start to kick.
Blog about how to boost your blog traffic.
Earlier today I was explaining blogs and aggregation to a friend as casting a net out that will bring you back the stuff you want. Blogs filter out all the stuff I don’t care about, and in addition to providing me with interesting content, they provide me with other sources of more similar and interesting content.
I’ve read blogs in some capacity for two years now, and never come close to my threshold. But lately I’ve been a little trigger happy with
Streamline the finding and obtaining of tv show torrents. Not that it was ever all that hard…
Verizon is starting to bring fiber right into the home to build it’s internet tv system, with 30-50mb/s!
Bigger then 2000. Will growth in the internet predict global economic improvement? The converse?
Tim O’Reilly looks like a pretty cool guy.
A study earlier this year by a group of scholars called the OpenNet Initiative revealed what no one had thought possible: that the Chinese government is succeeding in censoring the net.
In 1993 Rupert Murdoch boasted that satellite television was “an unambiguous threat to totalitarian regimes everywhere”. The Economist had already made the same claim on its cover: “Dictators beware!” The Chinese went berserk, and Murdoch, in response, ensured that the threat did not materialise.
So instead of democracy we get Baywatch. They are not the same thing.
So I’m still ambivalent about college. I don’t know how much it’s helping me learn. I’ve always been a fiercely individual learner, and I’m not yet sure whether or not college is the right place for me. I decided to attend college more by default, I didn’t quite have anything else to do that would satisfy me entirely. I thought I should at least give it a try. But I’m still only lukewarm at best about it’s value, and what the fuck am I doing here when I don’t think it’s the best thing for me?
I played Medievia while I was still in middle school, but it just got boring, and I’ve hardly plated a video game since.
I’ve been looking (more like thinking about looking) for a really sweet RPG to play since I read Snow Crash, because the metaverse absolutely kicks ass. Anyone know of a virtual world with not so much an achievable goal, and one that incorporates creating your own bits and pieces? World of Warcraft hasn’t ever appealed to me, I don’t know why (my aversion to video games, maybe).
I guess you could use this as your game engine…
Kevin Kelley on what the web is going to become. This guy has the real scoop on Web 2.0.
What happens when the data flow is asymmetrical – but in favor of creators? What happens when everyone is uploading far more than they download? If everyone is busy making, altering, mixing, and mashing, who will have time to sit back and veg out? Who will be a consumer? 4
What will most surprise us is how dependent we will be on what the Machine knows – about us and about what we want to know. We already find it easier to Google something a second or third time rather than remember it ourselves. The more we teach this megacomputer, the more it will assume responsibility for our knowing. It will become our memory. Then it will become our identity. In 2015 many people, when divorced from the Machine, won’t feel like themselves – as if they’d had a lobotomy. 5
It’s on.
Google’s related: is a fascinating way to web surf. Also, google cheat sheet
kottke wants internet in the middle of nowhere – I hope to be able to move out into the middle of nowhere sometime in the (near) future and get by with some sort of internet connection – wimax ought to be big by then. But I’ve thought about starting a wireless internet co-op and soliciting funding through grants to provide high speed internet to rural folks (myself included) which might generate me a meager salary on the side.
At least almost.
<a target="_blank" href="http://211.189.88.203/www.paypal.com/us/cgi-bin/webscr=cmdxpt/cps/clickthru2/Billing-Verification=CookieId=4801de10f2194572779a171135820269/">https://www.paypal.com/cgi-bin/webscr?cmd=_login-run</a>
You criminals are going to have to work harder then that to get my paypal login out of me.
Impressive linux box running broadcasting a wireless signal with a verizon cell data card to give an always connected link, even in the middle of nowhere.
The weekend of April 15 two of us took the system along during an 800mi drive through the high deserts of Southern California (Mojave and areas near). I’m quite surprised how far 1xRTT covers! We had continuous data for >90% of the trip. If the connection was re-established within 2 minutes (not hard when driving highway speeds) established TCP sessions resumed without worry. The only time we had extended data losses were when we stopped behind blocking geologic features (hills, ridges, etc) that kept us from seeing the nearest cel tower.
Also with gps and an ultra cool web app to google map the car’s exact location at any time… sweet. Also a camera to show where you are.
This should be good for security situations. If you point a camera inside at the driver’s position (from, say, the headliner or dashboard) you can make the video router start recording upon motion-sense. It would constantly upload the results to an offsite server along with timestamp and GPS coordinates. Even if the thief ripped the system out it would get a few good frames of him/her and store them offsite. If the equipment is hidden correctly it’ll give live GPS tracking and video to help with retrieval of the car.
Man, this stuff is cool.
Reveals, not releases or anything cool like that.
Sean Maloney, head of Intel’s mobility unit, said many trials were under way, but the actual deployment of mobile Wimax would not start this year.
They say the end of 2006, so a year and a half before it gets out into the market. WiMax looks sweet, I hope I can use it to get internet up at my cabin within the next few years, so that I can live there and farm/compute for a living.
I’m convinced that products of the Internet generation have an unusually neurotic, borderline obsessive personality which makes it difficult to try to characterize or rationalize their erratic behavior according to common standards.
...either the bulk of the people applying to business school are morally bankrupt, or some shared impetus compelled them to deviate. I think it’s the latter, and I think the impetus is the unparalleled, competitive-to-the-point-of-sickening nature of modern college admissions.
I just applied to colleges, and am in the minority in terms of not really caring at all about it, having one foot in the college boat and one in the screw going to college boat. But I have to agree here – the entire admissions process is sickening and obnoxious and tense and fairly badly conceived – I hated it.
It’s interesting, this internet generation. Yessir. Are we all entirely compulsive? Can we not get along without our information?
They may want the money, but they do not need the money the way they need to know their admissions decision, the way they need to check and recheck their status page like it’s some kind of nervous tick or insatiable addiction.
Are students just inherently unethical, or does their addiction spur unethical behavior? I want to understand now why today’s admissions climate makes students crave the decision, what’s unique about my generation that so affords this craving, and how we can make progress toward curing both.
A great interview between a few internet mavericks – Greg Storey and jason Kottke.
Airbag: How does it feel knowing you can never go back to being an amateur, that you can’t participate in Olympic competition?
Kottke: I’m sorry, I can’t hear you over the sound of my money counter.
It’ll be interesting to see how this turns out for kottke – I don’t think I’ll ever be able to hold down a job, maybe I should turn pro and blog all day (but I don’t think that would float my boat either).
A nice wireless antenna modification, focusing and increasing your range.
I don’t know what all the fuss over googles new rel="nofollow" has been, but Ben Hammeresly does a good job at bunking it:
There’s no incentive for me to spam those sites for the sake of getting Pagerank, that is true, but there’s even less incentive for me not*to. Why would I bother even testing the site for rel=”nofollow”? I might as well just hit it and leave. It’s less work for me, for exactly the same gain (some) and exactly the same loss (none).
All it does is shift the problem from the high pagerank blogs we here might have, with rel=”nofollow”, custom sanitize settings, and mt-blacklist in full effect, all the way over to the less technically adept. And that is one enormous customer service problem heading towards Blogger, 6A and the rest.
Technorati will have to choose if it’s a site that measures raw interconnectivity, or some curious High School metric of look-at-that-person-but-don’t-pay-her-any-attention that the selective use of the rel=”nofollow” attribute will produce. For many purposes, this would mean the results are totally debased and close to useless.
The web has always been build on links – and yes, when it was build there wasn’t this marauding problem of comment spam, referrer spam, and really, there wasn’t spam at all – but I don’t think that the solution to spam is to fundamentally corrupt what has become the primary presence of the web – the interconnection and relation between various nodes in the larger network.