What's wrong with Interchange

(and a proposal to fix it)

by Mark Stosberg
Last modified: Sat Apr 14 11:26:48 EST 2001

Background and Introduction

Before I get started on the reasons I'm dumping Interchange, I want to give you some background on why I adopted it and kept using it. I first found Interchange, then Minivend, in the summer of 1998. It was the only solution that satisfied all the buzzwords I was looking for:

worked with Perl
ran on FreeBSD
could work with Postgres
connected to Authorize.Net and Cybercash
could work with PGP or GnuPG
provided session management

I was in the early stages of learning Perl at the time, and the complexity of all the elements involved in e-commerce scared me. I had yet to explore on my own the difficulties of setting up session management, encryption, interfacing with a credit card processor or managing a complex database project. Minivend appeared to offer an incredible amount of out-of-the-box functionality, including a simple (looking) tag language and a huge amount of documentation.

Three years later, I've worked with Minivend off and on for several projects and am actively maintaining it for one (for a little while longer). My programming experience has expanded dramatically. I've written my own code for complex database-driven sites that use session management, encryption, and interface with credit card processors. It is with this experience and hindsight that I offer this review of Interchange, explaining why I'm migrating away from it, and how Interchange could possibly evolve to be the right solution for me again in the future.

My critique will focus on developing with Interchange (IC) rather than actually using it. I continue to find IC's default store setups that it ships with incredibly powerful and functional with minimal out-of-the-box configuration needed. For someone who is happy to use IC's default functionality, or something very close to it, I could still recommend it for that. If you plan to develop much custom functionality for IC using ITL and Perl, expect a significant amount of time or money spent doing so-- you may be better off with another solution.

The value of using Interchange Markup Language (or not)

One of my most frequent frustrations with Minivend occurs when I know how to solve the problem using custom Perl and SQL, but I need to rework the solution to work in Interchange Tag Language (ITL). Often this has taken longer that it would do to without the extra "help".

I believe ITL is fundamentally flawed. It was originally developed as MML, it was intended to be a Mere Mortals Language (MML actually stood for Minivend Markup Language). The idea was to make developing an e-commerce site possible for non-technical people. Perhaps it was at one time with Minivend. Then feature creep happened. Now it's called ITL, perhaps because you need to be an Information Technology Leader to understand it. The markup language is huge and complex and examples are often interwoven with Object Oriented Perl. So to get the most out of it, you need a thorough understanding of this complex language that's used only for Interchange on top of a strong understanding of Perl programming. Ironically, many of the ITL constructs take about the same amount time to write out in Perl as they do in the tag language. So I get stuck in this position when I need strong Perl skills to work with Interchange, but then I need to implement solutions in ITL.

So the tag language fails because it's no longer easily used by either non-technical people, or experienced Perl Programmers who are new to Interchange. In my experience, I can write a solution in Perl faster than I can learn how to do the same thing in the Interchange "short cut" language.

Encumbered Perl Access

One of the attractions of Minivend was it's integration with Perl. It had the appearance that anywhere Minivend couldn't directly solve my problem, I just call some Perl to solve the problem. Well, in theory, yes. In practice, solving problems this way has been harder for a number of reasons:

Communication between Perl and IC -- To usefully integrate, Perl and IC have to communicate. Doing this means understanding how the structure of IC's perl objects maps to to the structure of the ITL tags. So in addition to learning ITL, you know have to learn another syntax, the IC Perl syntax in order to solve your problem.
[perl], [calc] and interpolation -- To make the situation more complex, IC offers two similar tags for embedding perl: [perl] and [calc]. They are similar enough that are easily confused. Additionally you have option to interpolate IC tags within the Perl tags. This adds some extra flexibility, but it makes the Perl less clear-- the results are not always what you might expect
Safe Mode -- Since IC doesn't trust you to know what you are doing with Perl, it sticks you in "Safe" mode. That seems like a good idea at first, providing extra security (which it does). Unfortunately, I found I couldn't do something as basic as creating and using a database handle through DBI. So I faced two problems here: I was limited in what I could do with Perl under Safe mode, and if I wanted to developed in this environment effectively, I had to learn the details of the Safe module, as well the details of the options that IC invoked it with.
Debugging -- Before I learned Perl, I quickly learned (and forgot) C. I remember often getting feedback when my C programs ran like "Segmentation fault" and "bus error". Perl is so much kinder and informative. It gives me hints about what went wrong, and tells me just which line of which file to look at. Programming with IC, I have the painful experiences debugging that remind of my days using C. Frequently there is no feedback at all when something fails, leaving me to wonder whether it really failed, (or it successfully did the wrong thing). When there is an error, it's frequently reported originating from some core IC module, several abstraction layers away from the page that caused the error. It's usually fruitless for me to try trace this vague errors back to their origin. So even at the times that I've felt like I've been programming in IC with the same competency that I have in Perl, the process takes much longer because of the difficult debugging process involved.

Suggestion: expose and document the Perl API

So my first suggestion for Interchange is to expose, document and provide complete examples for working with the Perl API directly. This is already happening to some degree, but I'd like to see Interchange running entirely without ITL. I see a number of benefits to this, such as:

Faster learning Curve -- The documentation for this interface could slimed down dramatically because it could assume some amount of Perl knowledge and focus on the core functionality of IC.
Attract Developers -- A project written in Pure Perl would be accessible to more developers. From this I would expect submissions both of bug patches and new features and extensions.
Faster debugging -- By removing this layer of abstraction, debugging should be faster since there is one less layer for a bug to occur in, and less distance between your code and the error that was thrown.

Separation of Design, Content, and Code

Getting rid of ITL leaves the question, "well what do you put in the templates?". It's true of course, that you need to have some sort of tag language to run the templating system. However, this should be a minimal amount, so that there is a strict separation between the logic, design and content of the site. HTML::Template would provide an excellent solution here. It provides a minimal amount of logic in the templates, providing designers with some readily understandable concepts (if and loop) without asking them to master a new language.

What about multiple language support? Interchange has the ability to support multiple languages in a single page template, using a custom system. This is another area where IC has the opportunity to adopt a standard that is used elsewhere and more widely understood. In this case, it's XML. Apache::Pagekit is supports a number of the core features of Interchange using standard Perl technologies, including templating with HTML::Template, content abstraction with XML and session management with Apache::Session. ( disclaimer: I haven't built any projects with Pagekit at this point. )

Data modeling standards

When Minivend was young, it targeted flat-file systems as it's primary database type. As it grew up, it maintained a database abstraction layer, keeping support for flat-file systems and adding SQL support. Since that time, IC has yet to adopt some standard database design practices. For example, looking through the data model for the "construct" store of IC 4.5.7, it looks like Perl data structures are still being stored in BLOB and TEXT fields. Forget trying to do a "join" on this embedded data with a standard SQL table outside of IC.

To relate a product to a category, it appears the name of the category is stored in "category" field in the products table. Th standard way to handle this relationship in data model would be to relate the product based on a unique code for the category. Then, if the name of the category changes, you only have to change it in one table instead of two.

Likewise, it appears that the data model allows you to have the product listed in multiple categories, but the convention of having a table like category_product_map, that maps the category id's and the product id's is not used here either. Using standard data model design techniques would attracted more database-oriented people to the project, as well as allowing for better integration with existing projects.

Database abstraction layer

Having a good database abstraction layer is a sort of holy grail of database-backed applications. Few applications have them, and of those, very few have good ones that work well across several databases. I classify database abstraction layers into two main types:

pass-through abstraction -- In pass-through, the abstraction layer has little or no knowledge of each individual database, and the databases are expected to act in a uniform manner. For this to work, your application has to rely on the lowest common denominator of functionality available across all the databases. This design has the advantage of keeping you out of the business of knowing and caring about most of the details of each database, and allowing you to add a new database very easily. I believe this currently what IC uses.
lookup abstraction -- In lookup abstraction, you lookup the particular SQL you in need in a configuration file and then send the correct version off to the database. This system has the advantage of allowing you extract maximum power from the databases since the system is able to adjust to nuances of each database. The downside to this system is that now you are in the business of understanding every database that you support somewhat closely, including the differences between different versions of the same database. DBIx::Abstract is one module that helps you design an abstraction layer in this style.

Both schemes have their advantages. Here's my case for the latter. From what I can tell currently, IC has strong support for flat-file and MySQL databases, Postgres works most of time, and your mileage may vary with any other database. Say for example that I'm using Postgres and some function is failing that I know works on MySQL. I could lookup the function in the database lookup files to compare how it worked in MySQL and Postgres. I could tweak the Postgres version so that it worked on my database, and the MySQL command would work the same as before. With many people working on the project, someone could easily translate the commands that were known to work on MySQL to the variants that worked on other databases.

Under pass-through abstraction, if you can't find a lowest common denominator between the two bases, making a special tweak to fix one database endangers breaking all the others. I also think it would be quite appropriate for the official IC position to support just 1 or 2 databases and leave contributors, or paying clients, to cover the translations for the remaining databases.

Conclusions and Alternatives

If there's one thing I've learned from my experience with e-commerce it's this: E-commerce is complex. What I look for in a good e-commerce toolkit is enough functionality to address all the complexities, while adding a minimum of it's own. Interchange does a great job of addresses many of the complexities of e-commerce but adds so much complexity of it's own that it's more difficult than need be to learn and extend.

So, if IC is not the best alternative for a for a Perl/SQL e-commerce solution, then what else is?
At Summersault right now, we are developing custom solutions for clients using Perl, Postgres, several standard Perl modules including DBI, CGI, HTML::Template, HTML::FormValidator, and DBIx::Abstract, plus some of own modules and routines we find handy. This toolkit aspires to be ia publicly available open-source project, but in the meantime is not an option for developers looking to download a solution rather than hire one.

There continues to be limited competition for Interchange in the Perl/SQL E-commerce market. Zelerate is out there, but the last I checked, it had weak Postgres support [ update 06/13/02: website URL no longer works ]. The one e-commerce toolkit I have been most impressed with lately has been the Arsdigita Community System. [ update 06/13/02: the original Arsdigita fell apart and the remainder was sold to Red Hat ]. It's written in tcl (and now with some Java, I think) and runs on a Oracle. A spin-off, OpenACS, runs on Postgres.

This project minimized abstraction and kept the database central the system. Although the tcl is not reusable for me, I still look to the data models used in this ACS for inspiration when developing my own.

In the future, it's hard to say what might evolve to fill this niche. I could see a project based on CGI::Application or Apache::Pagekit stepping up to plate, perhaps a future version of Summersault's toolkit, or perhaps Interchange itself will evolve (or fork) to become the e-commerce toolkit I'm looking for.

Comments

I'm currently doing quite a bit of IC and share many of your frustrations. Some of your examples are a little exaggerated; however, I like your conclusion, "Interchange does a great job of addressing many of the complexities of e-commerce but adds so much complexity of its own that it's more difficult than need be to learn and extend."
I also share your ideas about things being easier to do without ITL (assuming the IC->Perl API were better documented). In fact, you're writeup has given me a good focus to take to my development. I think I'm going to try to see how much I can do without ITL. I don't know how familiar your are with the [mvasp][/mvasp] container, but it is a pretty good way to go ITLless.
The other main point I agree with is that the IC->Perl API should be much better documented. You are absolutely correct that this would speed up learning time for experienced perl developers, and therefore attract developers to help with IC.
One point I feel like you've left out is the development environment. The biggest frustration I'm facing now (after 9 months of learning ITL and the Perl interface) is the lack of editors that can syntax highlight and properly indent pages developed with ITL. Being used to how nice xemacs is for c/c++/perl/html development, it is totally frustrating to have to manually indent all my files.
Anyway, I'm working on getting an ic-mode for xemacs. I think that will go a long way to improving my development. Also, I will try to get mike to do more documentation of the API.
-- Doug Alcorn (mailto:doug@lathi.net http://www.lathi.net)

Mark Stosberg