Field Notes Inside an Integrated Communications Agency

Note: This little how-to works great for me and the way I work. I use a Mac laptop to develop Django sites which have a life as a production site somewhere else. If you work another way (like in Windows, or with Apache for development instead of Django's dev server, or if you work straight off the production server), this might not help directly, but I hope it helps get you started on finding a way to solve this little annoyance on your own. 

So I do a lot of Django development on my laptop. I have a bunch of Django sites which I built over the course of the past two years or so - most of which I have to go back and work on a little every now and then. Because they've been developed at different times, most go back to before the new QuerySet refactor merge, and a few go back to before the Autoescape revision. I'm also looking at building my next site with the newforms-admin branch.

 I'm using subversion to check out the latest versions of Django. I usually check it out into /usr/local and then make a symbolic link to it in Python's site-packages folder. Because some of my sites raise errors when I run them with versions of Django released after they were written, I find myself changing the revision number of my laptop's Django occasionally (and by occasionally, I mean way too often).

To get around that, I figured out that I can do this little trick. Now, in my /usr/local I have all the different versions of Django I'm likely to need. Like so:

django/
django-newforms-admin/
django-pre-qsr/
django-pre-autoescape/
django-myclient/

The one called just django/ is the latest trunk revision... the one I usually use. The others are the revisions (or close approximations of the revisions) that I used when I made some of the older Django sites on my laptop. Each of these folders contains a folder called "django".

In the manage.py files for each of my sites where I can't or don't want to use the latest trunk revision, I just drop in a couple of lines right at the top:

import sys
sys.path.insert(0, "/usr/local/django-myclient/")

This drops the path to my preferred Django rev onto the start of the sys.path list, and Python uses the first django module it finds in it's path. Now whenever I use manage.py's shell, dbshell or runserver commands, I'm using the Django rev number I need without having to fool around with svn update each time I need to work on a different site.

This article describes the process we use at Capstrat to set up Django applications on WebFaction shared hosting. The instructions are specific to Mac OSX or Linux, but I've included a few additional notes for Windows users. It is assumed that you are comfortable working with the command line, that you have Python and Subversion client installed on your machine and that you have checked out and worked with Django before.

1. Create a new WebFaction Shared hosting account (Shared-1 through Shared-4, according to your project's needs).
   1. You can do this at http://www.webfaction.com/signup/
2. Create a new Django application on your local machine.
   1. I recommend keeping your projects in an easily accessible place, like /src/python/ (C:\src\python\ on Windows).
   2. At the command line, navigate to the directory where you keep your Web projects.
   3. Run:
      python <path-to-django>/bin/django-admin.py startproject <project-name>
      to create a new project.
      1. Note: <path-to-django> refers to the place where the Django source code exists on your machine.
         Additionally, <project-name> will be the name of your new project.
         
      2. Concretely, I would do this on my machine:
         cd /src/python
python /opt/local/lib/python2.5/django/bin/django-admin.py startproject mikesproject

3. Set up your project's static assets.
   1. I store my project's static assets in a directory named "elements." Create this directory.
      cd /src/python/<project-name>
mkdir elements
mkdir elements/css      # store all CSS here
mkdir elements/img      # store all HTML design images here
mkdir elements/js       # store JavaScript files here
mkdir elements/swf      # store Flash files here
mkdir elements/media    # store user-contributed media here

   2. To simplify local development and deployment, you can use Django's static media server to serve out the elements folder. Note: when the project is deployed to WebFaction, you must be certain that these files are served by Apache and not by the Django media server.
      1. In settings.py, add these new global settings variables (anywhere you want):
         DEVELOPMENT = True
PROJECT_DIRECTORY = '/src/python/<project-name>/'
ELEMENTS = PROJECT_DIRECTORY + 'elements/'

      2. In the global urls.py, add this code at the bottom of the file:
         # Catch elements URLs for working in development
if settings.DEVELOPMENT:
   urlpatterns += patterns('',
      (r'^elements/(?P<path>.*)$', 'django.views.static.serve', { 'document_root': settings.ELEMENTS, 'show_indexes': True }),
   )
      3. You will also need to add this to the top of the urls.py file to avoid a run-time error:
         import settings

   3. If you want to use Django's admin interface, set ADMIN_MEDIA_PREFIX in settings.py like so:
      ADMIN_MEDIA_PREFIX = '/elements/admin/'
4. Set up your project's database.
   1. To ease development, we typically use sqlite3. The sqlite3 database is stored in a single file and can be easily passed around without the need for a dump/load procedure.
   2. In settings.py, set the database settings like so:
      DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = PROJECT_DIRECTORY + 'db.sqlite3'

   3. Leave the other database settings blank.
   4. If you want to use Django's admin interface, add this line to the end of the INSTALLED_APPS tuple in settings.py:
      'django.contrib.admin',

   5. Finally, create the database file. On the command line (in your project directory), type:
      python manage.py syncdb

      1. You will need to enter your new administrator username, email and password as part of the initial sync.
5. Create a settings file for WebFaction.
   1. settings.py will be different on WebFaction than on your local machine. Copy your local settings.py to a new file called wf-settings.py. In your project directory, type:
      cp settings.py wf-settings.py

   2. Edit the new wf-settings.py file.
      1. Set the DEVELOPMENT global to False. This will disable the static media server set up in step 3.2.
      2. Modify the PROJECT_DIRECTORY setting to the directory where your project will exist on WebFaction. Assuming that you set up your account as a Django account, it should look like:
         PROJECT_DIRECTORY = '/home/<account-name>/webapps/django/<project-name>/

   3. Maintaining a separate settings file for WebFaction is a reasonably efficient way to manage differences between your local machine and the WebFaction server; however, note that you will need to manually keep the two files in sync. Anything you add to your local settings.py will need to be added to wf-settings.py as well.
6. Set up Subversion on WebFaction.
   1. Log in to the WebFaction control panel at http://panel.webfaction.com with the username and password provided for your new account by WebFaction.
   2. In the navigation menu, under "Domains/Websites" click on the "Applications" link.
      1. Click on the "Add" button in the bottom right to create a new application.
      2. Type "subversion" into the name field.
      3. Choose "Subversion" from the App type drop-down.
      4. Click the "Create" button.
   3. Now, create a subdomain for Subversion. In the navigation menu, under "Domain/Websites" click on the "Domains" link.
      1. Click on the "Edit" button in the row for "<account-name>.webfactional.com"
      2. Click on the "Add" button in the bottom right of the "Subdomains" section.
      3. Enter "svn" in the new "Prefix" field.
      4. Click the "Update" button.
   4. Create a Website to tie the subdomain to the application. In the navigation menu, under "Domains/Websites" click on the "Websites" link.
      1. Click the "Add" button in the bottom right to create a new Web site.
      2. Enter "subversion" in the name field.
      3. Select "svn.<account-name>.webfactional.com" in the "Subdomains" field.
      4. Click the "Add" button under "Site apps".
      5. Choose "subversion" from the "App" drop-down, and enter "/" in the "URL path" field.
      6. Click the "Create" button.
      7. Now, the Subversion repository is set up and running at http://svn.<account.name>.webfactional.com/
   5. Next, you need to ssh into your WebFaction account to configure the Subversion repository permissions.
      1. At the command line, type:
         ssh <account-name>@<account-name>.webfactional.com

      2. Enter your password to log in.
      3. Navigate to the Subversion directory and set up usernames and passwords to control access to the repository:
         cd ~/webapps/subversion

         1. View the .htpasswd file to see what usernames have been set up:
            less .htpasswd

         2. Delete the "test" username from .htpasswd
            htpasswd -D .htpasswd test

         3. Create a username and password for all the people who will have access to the repository (the example here uses md5 hashes to store the passwords):
            htpasswd -m .htpasswd <username>
(Enter the new password twice at the prompt.)
            
         4. Review the list of usernames to verify that you've added all desired users:
            less .htpasswd

   6. Congratulations! Subversion is ready to use.
7. Check your Django project into Subversion.
   1. First, create the directories that you'll need in the repository, according to Subversion best practices. At your local command prompt, from your project folder type:
      svn mkdir -m "initial project directory structure for <project-name>" http://svn.<account-name>.webfactional.com/<project-name> http://svn.<account-name>.webfactional.com/<project-name>/trunk http://svn.<account-name>.webfactional.com/<project-name>/tags http://svn.<account-name>.webfactional.com/<project-name>/branches

   2. Now, verify that your local subversion settings are correct before you check in any code. I recommend not storing .pyc files in the repository and also not checking in the settings.py file.
      1. Open this file for editing: ~/.subversion/config ("C:\Documents and Settings\<windows-username>\Application Data\Subversion\config" on Windows)
      2. In the [miscellany] section, uncomment the "global-ignores" line and add any filename patterns that you want Subversion to ignore. On my machine, where I'm primarily performing Django development using TextMate as my editor, my global-ignores settings looks like this:
         global-ignores = *.o *.lo *.la *.pyc *.tmproj settings.py .*

      3. The most imporant entries here are "*.pyc" and "settings.py" - these prevent Subversion from checking in and managing binary compiled Python (.pyc) and Django settings.py files.
   3. Now, you can import your new project into the Subversion trunk you set up in step 7.1. In your project's directory, type:
      svn import . http://svn.<account-name>.webfactional.com/<project-name>/trunk/<project-name>

      1. Note: This will import all files and directories from your project with the exception of settings.py (because of your "global-ignores" setting).
   4. At this point, your code is stored in the repository, but your local files are not yet managed by your Subversion client. You need to check out the project you just imported to get a managed local copy. Starting in your project directory, perform these commands.
      cd ..                                        # go up one directory
mv <project-name> <project-name>_tmp         # move the project to a temp directory
svn co http://svn.<account-name>.webfactional.com/<project-name>/trunk/<project-name> <project-name> # check out the project
cp <project-name>_tmp/settings.py <project>  # copy the original setting.py file into the checked out project.
rm -rf <project_name>_tmp                    # remove the temporary project folder

   5. Now you have a local working copy of the code stored in the Subversion repository.
8. Check out the project on the WebFaction server.
   1. You must have a Django application set up on the server before you can perform this step. If you chose Django as the application type when you created your WebFaction account, this is already done. Otherwise, perform these steps:
      1. In the WebFaction control panel's navigation menu, under "Domains/Websites" click on the "Applications" link.
      2. Click the "Add" button in the bottom right.
      3. Enter "django" in the "Name" field.
      4. Choose a version of Django from the "App type" drop-down. At the time of this writing, I'm using "Django 0.96.2/mod_python 3.3.1/Python 2.5"
      5. Click the "Create" button.
   2. SSH into WebFaction. Navigate into your Django project directory and check out the project:
      cd ~/webapps/django
svn co http://svn.<account-name>.webfactional.com/<project-name>/trunk/<project-name> <project-name>

   3. Create the setings.py file for the server by copying wf-settings.py
      cd ~/webapps/django/<project-name>
cp wf-settings.py settings.py

9. Create a new application to serve static media in your project's elements folder.
   1. In the WebFaction control panel's navigation menu, under "Domains/Websites" click on the "Applicatins" link.
   2. Click the "Add" button in the bottom right.
   3. Enter "django_elements" in the "Name" field.
   4. Choose "Symbolic Link" from the "App type" drop-down.
   5. Enter the absolute path to the elements folder in the "Extra info" field. It should look like this:
      /home/<account-name>/webapps/django/<project-name>/elements

   6. Click the "Create" button.
10. Set up Apache to serve your Django application.
    1. SSH into the your WebFaction account. Navigate to the Apache configuration folder:
       cd ~/webapps/django/apache2/conf

    2. Edit the httpd.conf file. Modify this line:
       SetEnv DJANGO_SETTINGS_MODULE myproject.settings
to read:
       SetEnv DJANGO_SETTINGS_MODULE <project-name>.settings

11. Finish the remaining tasks in the WebFaction control panel.
    1. In the WebFaction control panel's navigation menu, under "Domains/Websites" click on the "Websites" link.
    2. If you chose Django as the application type when you created your WebFaction account, there should already be a Web site called <account-name>. If not, create one by following these steps:
       1. Click the "Add" button in the bottom right.
       2. Enter "<account-name>" in the "Name" field.
       3. Choose the desired subdomain(s) (something other than svn.<account-name>.webfactional.com).
       4. Under "Site apps," click the "Add" button.
       5. Choose "django" from the "App" drop-down. Enter "/" in the "URL Path" field.
       6. Click the "Create" button.
    3. Now, edit the Web site by clicking the edit button in the "<account-name>" row.
       1. Under "Site apps," click the "Add" button.
       2. Choose "django_elements" from the "App" drop-down. Enter "/elements" in the "URL Path" field.
       3. Now, your site is set up to serve out the elements files with Apache and to handle all other URLs with your Django application.
12. Tie it all together.
    1. SSH into your WebFaction account.
    2. Check out the version of Django you require for your application:
       cd ~/lib/python2.5
You may need to remove an existing copy of django: rm -rf django
svn co http://code.djangoproject.com/svn/django/trunk/django django
If you need a different revision than the latest trunk, do this: svn update -R <desired-revision> django
       
    3. Set up a symbolic link to Django for you project:
       cd ~/webapps/django/lib/python2.5
rm -rf django                       # remove the WebFaction-provided copy of Django
ln -s ~/lib/python2.5/django django # create a symbolic link to your revision of Django
Note: Here, we have modified the project-specific Python lib folder that Apache uses when it runs Django. If you need to run different revisions of Django for two different applications on the server, you could alternatively check out the required revision in the project-specific lib folder, rather than using a symbolic link to the global copy of Django.
       
    4. Create a symbolic link for serving Django's admin media files out of your elements folder:
       cd ~/webapps/django/<project-name>/elements
ln -s ~/lib/python2.5/django/contrib/admin/media admin
Note: the symbolic link name "admin" corresponds to the ADMIN_MEDIA_PREFIX in settings.py from step 3.3.
       
    5. Make Subversion ignore the new symbolic link. From the same directory, type:
       svn propset svn:ignore admin .

    6. Configure the Subversion client on WebFaction. Just like in step 7.2, you'll need to edit the Subversion config file:
       1. Open this file for editing: ~/.subversion/config
       2. In the [miscellany] section, uncomment the "global-ignores" line and change it to look like this:
          global-ignores = *.o *.lo *.la *.pyc *.tmproj settings.py .*

       3. Again, be sure to add "settings.py" to the list so that you don't clobber your local machine's version of this file.
    7. Check in the working copy from the server:
       cd ~/webapps/django/<project-name>
svn ci -m "working copy is now set up on WebFaction"

    8. Restart Apache:
       cd ~/webapps/django/<project-name>
../apache2/bin/stop
../apache2/bin/start

    9. You should now be able to view your project running in a Web browser at the subdomain you specified in step 11.2.
    10. Finally, you'll probably need to set up an alias to Python 2.5 if you intend to work in the Python interactive shell on WebFaction.
        1. SSH into your WebFaction account and edit the file ~/.bashrc
        2. At the bottom of the file, add this line:
           alias python=/usr/local/bin/python2.5

        3. Save the file and run the command source .bashrc to make the changes take effect. Now, running python will invoke Python 2.5, so Django will be available for import.
13. Develop.
    1. Write your Django application incrementally in your local working copy.
    2. When you have created and tested a new feature, check it in to subversion.
    3. SSH into WebFaction and update the working copy to apply the checked-in changes.
       cd ~/webapps/django/<project-name>
svn update

    4. If you have modified any Python files (*.py files), restart Apache so that mod_python reloads your files:
       ../apache2/bin/stop
../apache2/bin/start


In the current development version of Django, model field validation in the admin has one difficult shortcoming: validator functions don't know if the fields they are validating are for an object that is being created or edited. In the common case, this isn't a problem, but certain problems cannot be solved without this information. In particular, these problems can't be solved with Django validators:

• Automatically creating unique URL slugs from other fields in the model (e.g. the "title" CharField)
• Preventing self-referencing foreign key infinite loops

Consider this partial implementation of a Blog Article model that has URLs like this "/2008-3-11/my-article/" which allows articles to have the same title (actually, the same slug) so long as they are published on different dates:

This model has a "slug" field that gets populated from the "title" field in the "save()" method. This seems okay at first glance, but consider what would happen if a user entered these two articles in the admin:

Despite the change in letter case, these articles will have the same slug and the same date, so the produced URLs won't be unique. Unfortunately, the "unique_together" constraint will not help us here in the admin, because "slug" is not calculated until "save" is called, which happens after all validators have run. This situation will produce a database exception, rather than a validation error, resulting in an ugly and confusing user experience. This little experiment also shows that adding an additional "unique_together" constraint for ('title', 'published',) would also be ineffective, because we could still get non-unique URLs from two similar titles with letter case changes or other subtle changes.

To solve this properly, we'll need to write a validator for the "title" field that checks whether the slugified version of the title is actually unique for the given date. It should look something like this:

This looks almost right. It will work perfectly when we're creating a new BlogArticle, but this will prove to be a miserable bug when editing an existing BlogArticle, because we'll be forced to change the blog title or publish date to get the validation to work. To correct this logic, the validator needs access to the BlogArticle object being edited. Suppose the validator has access to this object, call it "original_object," which will be set to None when we are creating an object. Then the validator would look like this:

This will accomplish the validation we need, but there's a big problem here: how do we figure out the "original_object?" We could try to query for an existing object based on some data in "all_data," but that would be a fallacy, because "all_data" is being edited by the user and may no longer match an existing object in the database. We know that the admin view code must be aware of whether an object is being added or edited. If you take a quick look in the admin view functions (django.contrib.admin.views.main), you'll see an "add_stage" and a "change_stage" function. These functions instantiate an object of your model class' AddManipulator or ChangeManipulator, respectively, to accomplish the adding or editing process. The ChangeManipulator object stores the object being edited in its "original_object" attribute, and the AddManipulator has no such attribute. We need to find a point to hook into the admin / field / manipulator code to gain access to this "original_object" so we can pass it to our validator.

The validation occurs after POST when the manipulator object's "get_validation_errors" function is called. This function calls "get_validation_errors" for each oldforms.FormField object in the manipulator (the manipulator's "fields" are all objects that subclass django.oldforms.FormField). The oldforms.FormField's "get_validation_errors" function calls each validator function in its "validator_list" attribute, passing each function "field_data" and "all_data," the value of the current field being processed and all the POSTed values, respectively. This is a clean mechanism for handling validation, but the oldforms.FormFields' "get_validation_errors" functions are still hopelessly unaware of the current object being edited.

The trick that I've discovered is to hook into the code at the point where the manipulator's oldforms.FormField objects are created. At this point in the code, the "validator_list" can be modified with more powerful validators. When the admin views construct an AddManipulator or ChangeManipulator object, the manipulator constructor creates an oldform.FormField object for each field in your model class (for your reference, the model class fields inherit from django.db.models.fields.Field). The manipulator constructor accomplishes this by calling each model fields' "get_manipulator_fields" function, passing itself to each function as a parameter. The "get_manipulator_fields" function creates the oldforms.FormField object and, of course, sets its "validator_list" from the corresponding attribute stored in the model field object. At this point, we can hook in and modify the "validator_list" before the framework creates the oldforms.FormFields. I suggest doing this by subclassing a model field class. For django.db.models.fields.CharField, it looks like this:

Now, in our BlogArticle model, we'll change the "title" field to this:

Note the new parameter "context_validators" that accepts our modified validate_title validator function. Now, slugs will be safely created from the title field and all blog articles will have unique URLs.

Now, what about preventing infinite loops in self-referencing foreign keys? As a concrete example of this problem, consider this: We want to build a model representing a simple website's navigation system. We'll create a NavigationNode model that stores each node's parent NavigationNode (with top-level nodes having no parent) and information about the URL and page to display. A stripped-down version of the model might look like this:

Here, we've used a new field subclass called "ContextValidatedForeignKey" that works similarly to the "ContextValidatedCharField" described above. Its definition looks like this:

Finally, we need to implement the validator:

And that's it. An attempt to solve a similar problem was written up on djangosnippets. The author wrote a standard validator, but the code falls short of solving the problem, because it assumes that the slug field does not change. The code is still prone to producing an infinite loop.

Clearly, this "ContextValidatedField" method is a bit of a hack. A better solution might be to modify the Django framework to pass the original object data to all validator functions, preferably in a backwards-compatible way. One obvious problem with that approach is that validator functions could do perverse things like delete the original object. Maybe a better solution would be to have the framework pass another dictionary called "old_data" or "original_data" as a third parameter to each validator function that would contain all of the original object's data but would provide no mechanism to alter the original object.

I wasn't able to find any TRAC tickets or other information with a quick search of djangoproject.com, and I think this problem may merit opening a ticket. I welcome any comments or ideas on how to solve this problem more cleanly or easily in Django.

If you've ever needed to use the Django database API or templating system from a standalone Python script, you've probably written a quick workaround or ugly hack to get the Django environment working properly. Specifically, this problem has to be solved in order to run Django scripts as cron jobs. Some possible solutions to this problem were written up by James Bennet and Scott Newman. I prefer the clean approach in this method documented by Jared Kuolt.

Jared uses an undocumented function in django.core.management called setup_environ to prepare a script so that Django can be used normally, just like you would use it in any of your website's view modules. It goes like this:

The setup_environ function simply takes your project's settings module as a parameter and sets up Django accordingly. Search for this function in django/core/management.py if you want to see the details of how Django sets itself up. Though this method is simple, elegant, and decently abstracted, it has two main limitations. It may only be a three-line incantation, but it's a pain to memorize these sort of things, and it would really be ideal if this could be accomplished in one line. More importantly, in order for this method to work, your script's file must live in the root of your project, so that "settings.py" is its neighbor. I prefer to keep my cron job scripts or data importing scripts in an appropriately named subdirectory, such as "scripts." From a subdirectory, import settings will not work.

I've written a script to solve this problem so that you only need to write one line of code that's easy to memorize. You'll need to copy and unzip this script, setup_django.py, into the directory where your script lives. Write this as the first line of your script:

And voila! That's all you need to do. You can now use the Django database API just as you'd expect. The code borrows some ideas directly from the setup_environ function in django.core.management. It goes like this:

When you import this module, it runs prepare_environment. First, the function calculates its own path. It then takes this path and moves upward one directory at a time, attempting to import a module named "settings." At each step, it checks if it has reached the root directory using the is_top_level function, which employs a simple regular expression for detecting the root directory on Unix or Windows. If the root has been reached, an exception is raised; otherwise, when the settings module is found, it is passed to Django's setup_environ function.

Some may not find this method to be as nice as others, because you'll need to copy the "setup_django.py" script into all of your script directories, but the one-line import setup_django solution is very nice. The main caveat to consider is that if you have a file named "settings.py" other than the project's main "settings.py" in the directories above your script's path, this method will not work. This was only tested in Python 2.5.x, but there shouldn't be any problems running it in slightly older versions of Python.

Enjoy!

When clients come to our firm for interactive recommendations they generally have a lot of ideas. The interactive world is scary. There are a lot of acronyms. There are nerds that keep information shrouded in secrecy so that you will have to continue to rely on them. Or worse, there are nerds that complicate things by overexplaining things you don’t need to know.

As a result, clients try to get up to speed on current Internet trends or hide behind their IT person. 

The first thing they will try to tackle and understand is the development language. Clients will often agonize over the language used to develop their site, but this is the wrong focus. Technology is going to change. Developers are smart and they will figure out a way to make new technology work with what you have established. If you build a site and continue to pay attention to the backend as well as the front end, you are going to make a long lasting Web site that will have a longer shelf life. If you build a site and don’t touch it again for 18 months, you are going to have to rebuild it the next time you touch it. So why agonize over the language? Trust the experts.

As a client, your focus should be...

The most important thing you can do as a client is decide what function your Web site will perform. Is it a brochure site? Is it an application? Is it core to your company's business?

There are plenty of companies out there that don’t need to create anything more than a brochure site. We don’t all need social networking sites.

The clients that are most successful are focused on the results, not the way we get there.

Instead of blowing all of your money on a redesign and then doing it again 18-48 months later, spend your money wisely. Invest as you go along. Pay attention to your analytics. Find an expert that is going to keep you up on technology without boring you with the details. You don’t care if Django or Python is the best solution, but you do care that users are finding the right information on the site.

This approach is not always applicable to large companies. When your Web site has to fit into a large-scale enterprise solution, it is very important to perform the due-dilligence to ensure the technologies you invest in will be supported later by your in-house technology team.

As an interactive professional for the past 10 years, I have seen a lot of languages and development theories come and go. Don’t agonize about the language, you’ll always be able to find someone to support it. Instead, focus on the content and the user experience.

Marketing is all about relating to humans. One way to truly be a step ahead is to take the budget you have for your redesign, double it and spend the second half on committed maintenance on your site. It will guarantee a longer lifespan and will help you focus on customers- the reason you made the site.