Choosing a web development framework and database server

September 26, 2003

The current state of the art from a server capacity perspective is cooperative multitasking and queues. The basic notion is that you should not have to pay for all the context switches in the operating system and the CPU when you don’t want/need them. If you design your application to yield everytime it is waiting for something to return then you gain MASSIVE performance improvement both in terms of speed and in terms of not requring huge amounts of memory per simultaneous client. See e.g. The C10K Problem.

In python, your options are async core (aka Medusa) or Twisted.. Medusa is being fecklessly maintained while twisted is in active development so the choice is obviously twisted. If I were working in Java, I might choose Seda.

So having chosen twisted you want to create a src directory. In the src directoy, create a directory with your application name (this is going to be a module name directory or package directory if you are in java). Add the src directory to your compiler or script language loadpath.

In that directory, create a main.py that takes a config and runs your site given your framwork. For Twisted, this looks something like this:


    #!/bin/env python
    from twisted.internet import app
    from twisted.web import static, server,script,vhost

    def start(config):
    root = static.File(config.fileBase)

    root.indices=['index.rpy']
    root.processors = {'.rpy': script.ResourceScript}
    root.ignoreExt(".rpy")

    rootroot = NameVirtualHost()
    rootroot.addHost(config.host,root)

    application = app.Application('web')
    application.listenTCP(config.port, server.Site(rootroot))
    application.run()

    import sys
    if __name__=="__main__":
    configPath=sys.argv[1]
    execfile(configPath)
    main(__name__)

This script allows you to start the site either with config.py as a commandline argument or from another script by passing a config as an argument to start. It also runs the site as a virtual host so that unless someone knows the name of the file your dev site is inaccessible to nosy people on your LAN even if you are exposing the port to the world. Remember to edit your hosts file to get the local name to resolve correctly.


Introducing My Secret Project

September 26, 2003

Some friends of mine may know what I am actually working on, but the details are irrelevant here. What this topic is about is how I am developing this project. I spend a few hours on the #twisted irc channel getting a lot of help and want to give back to the community so here goes. How to make a massively cool web site on a tight budget.

1. make a directory for your project on your development computer

2. Build a prototype in HTML that has all the functional options you may want to provide. Show it to some friends and get positive feedback about all the power you can provide.

3. Start thinking about the complexity of specifying and using all those features and shove all those files into a subdirectory of your project called proto

4. check the directory into revision control. I am using subversion on my latptop to a reposiotry also on my laptop.

5. make a directory called site and make the entire HTML for your site again thinking carefullly about how much work it will take for the user to understand what you are offering and whether anyone really cares to have each bit of functionality in version 1. You want to trim to the bare minimum at this stage.

6. Make a directory called src and a directory called test.

7. choose python to develop your application because python programmers are more likely to be smart and saavy and culturally in sync with you than most other languages (because it is a little obscure and a little mainstream and very powerful).

8. Put a config.py file in your test directory. It wll be used by both your test runner and your site runner. config.py should start off looking like this:


    fileBase="../site"
    scheme="http"
    host="myhost"
    port=8080

Update your hosts file so that myhost points to localhost.

9. Create test.py and start.py in your tests direcotry which both import config.py. start.py starts the website and test.py runs tests against the website. Here is my start.py:


    import sys
    sys.path.append("../src")
    import cospend.main as main
    import config
    main.start(config)

10. Using pyunit (the unittest module), write tests of the most basic functionality that you intend to implement (e.g. authorization) in test.py. test.py should end with unittest.main(). You may want to create an HTTP client module to automate some of the HTTP stuff inherent in all of these tests. (I did.)

Next up. Choosing a site development model to satisfy the tests and other constraints.


Religious people are more wealthy

September 18, 2003

In answer to Islamist concern about maintaing religion in the face of economic growth comes a new study that finds that people who attend religious services regularly build more wealth than those who fail to do so and that their are wide disparaties in wealth production accross religions.

The study suggests that different religions train their adherent in different wealth building behavior and that attending religions services furnishes greater opportunities for social networking. I would add that religions endow their adherents with additional social capital that they can use to cooperate and achieve public goods more efficiently than they could as unaffiliated individuals. For more on why this social capital is important see the end of the enlightenment..


Wireless Deregulation vs New Chip Fabs

September 17, 2003

After the Accelerating Change Conference, John Smart and I were debating the merits of spending on better bandwidth vs spending on more CPU power. He noted that bandwidth efficiency is growing more slowly than compute power or storage and argued it was natural. I said that the money spent on the next chip fab would be better spent buying out TV broadcasters and using their spectrum for bits. There are regulatory reasons why that is difficult, but Kevin Werbach shows the value of doing so and how to get there.


Social Networking Is a Business

September 17, 2003

Went to this last night. No discussion of how many different social networking variants there actually are or what the opportunities are in the space. No discussion of tacit vs articulated social networks and the benefits of each. No discussion comparing friendster to blogrolls or about the distinction between articulated meta-data and extended personality as expressed through one’s blog content. Oh well. better luck next time.


Bits vs. Atoms: Embodiment overtaking Enlightenment

September 17, 2003

Kevin Werbach recommends looking at Me++ The Cyborg Self and the Networked City. From its web site:

Thus, Mitchell proposes, the “trial separation” of bits (the elementary unit of information) and atoms (the elementary unit of matter) is over. With increasing frequency, events in physical space reflect events in cyberspace, and vice versa; digital information can, for example, direct the movement of an aircraft or a robot arm. In Me++ Mitchell examines the effects of wireless linkage, global interconnection, miniaturization, and portability on our bodies, our clothing, our architecture, our cities, and our uses of space and time. Computer viruses, cascading power outages, terrorist infiltration of transportation networks, and cellphone conversations in the streets are symptoms of a dramatic new urban condition–that of ubiquitous, inescapable network interconnectivity. He argues that a world governed less and less by boundaries and more and more by connections requires us to reimagine and reconstruct our environment and to reconsider the ethical foundations of design, engineering, and planning practice.

Thats right. The experiment of the enlightenment that our minds are separate from our bodies has come to an end.
The enlightenment is failing on all fronts these days. We need to come up with a replacement fast.


What They’re Fighting For (Embodiment and not Enlightenment)

September 17, 2003

Vinod does a great job describing what the Islamists are fightingfor. He quotes a recent Al Queada published book saying:

…The goal of democracy, according to Al-Ayyeri, is to “make Muslims love this world, forget the next world and abandon jihad.” If established in any Muslim country for a reasonably long time, democracy could lead to economic prosperity, which, in turn, would make Muslims “reluctant to die in martyrdom” in defense of their faith.

Question for western folks: Is Al Queada correct that economic prosperity and democracy cause secularism?

Given that the most secular cultures are the most infertile cultures (see last post), does this place limits on the overall rate of economic growth for human cultures?

Is economic growth sustainable in the face of declining population?
Some people suggest we can solve the declining growth problem with immigration, but we then face either the cost of acculturating and educating the immigrants and integrating them into our culture or we face an alien population that does not really replace the children lost to infertility.

Can we find ways to grow fast that don’t cause secularism and infertility?

Vinod then quotes Islamist author Sajjad Khan as saying:

…The Western viewpoint in life divorces societal actions in this life from the hereafter and therefore ignores any accountability after death. Therefore Western politicians judge actions they do in this life according to their own criteria usually the pursuit of material benefit, i.e. utilitarianism.

In this Khan is correct. So far, the best mechanism for internalizing public good producing behavior that we know of is religion. Given that child production is a public good, are the Islamists right to be alarmed about the threat Western Culture poses to their future generations?