SuperSatellite

A Day with dotCMS 1.6.5

// December 2nd, 2008 // 5 Comments » // Software, Tech, Web

It was long coming and well worth waiting for.  On Halloween dotCMS released version 1.6.5, and feeling confident in the code base release, we finally tested and rolled out the upgrade on our production servers (I try not to jump on updates right away, just in case there are any surprise bugs that need to be worked out).  Naturally, we tested things out on our dev box first, were satisfied, and just before Thanksgiving we finalized the upgrade on our live server.  I can say with confidence that this upgrade has proven to be a nice kick in the butt, without a single hitch so far.  Even the improved edit mode interface hasn’t drawn any complaints for being “different.”

A peak at the new dotCMS 1.6.5 edit mode

I’m among the people who think that 1.6.5 is really more like a 1.7 in that it’s not just a minor patch of the 1.6 codebase – there’s a lot of new crap.  Among things to look for:

• WebDav is all new
• Edit Mode is all new
• Widgets have been added
• Updated TinyMCE
• Caching is now in memory instead of disk
• Lucene index has been moved
• PDF page output macro
• A whole crap ton of other stuff

All in all, it’s like 322 different changes, fixes, and additions. Nothing to sneeze at.  And it all matters.  First off, compared to 1.6.0.10, it’s substantially faster on both the front and the back end thanks to the improved caching.  And it flat out blows away 1.5, so if you’re still lingering on that version, I suggest an upgrade pronto.  The speed is coupled with a less noisy log as well, making it easier to tell actual errors from random chatter from the system.  They went through and completely refactored the WebDav interface, so if you use it for bulk adding files or live editing, you should notice a substantial boost in speed there, as well as the ability to autopublish files uploaded through it (great for photos for photo galleries, or live editing things like CSS (Cascading Style Sheets) or VTL files).  The upgrade process itself didn’t feel overly painful either.  One thing that has been a constant complaint was that the system was a little overly painful to actually upgrade, especially from much older revisions.  What I can say is that going from 1.6.0.10 was about the fastest and smoothest upgrade I’ve done so far (that wasn’t just a simple maintenance patch).  SVN update, copy back some server settings, ant clean and deploy, reindex, and we were pretty well off and running (on top of that, I was also switching SVN branches as well).  I could have probably even gotten away with using my old Lucene index, but, fresh system seemed to call for a fresh index.  Interesting side note, I was talking with Chris Falzone over at Edinboro, and he thought upgrading from 1.5 straight to 1.6.5 proved easier than going from 1.5 to 1.6.

My apologies for no real stats on this on my end, but prior to rolling out the upgrade live, I did do some load testing against our 1.6.5 test server using Apache JMeter, and the results were excellent.  I was able to run up about 100 threads within a 30 second time period without drops.  Actually, the machine I was testing with crapped out before the server in some cases.  I also saw some stats that Jason, their lead developer, shared on the mailing list after testing it with Siege that I’ll pass on:

I wanted to share some performance numbers with you.

The test was run on a server with 8 GIGs Ram Dual Quad-Core Xeon E5320 / 1.86 GHz processor running the latest Ubuntu LTS. The server had Postgres 8.3 install locally and Java 5 64bit.

dotCMS configuration
JVM 1536M of memory for the start and max of the heap. The connector was set to 380 Max thread and the Db had 390 max db connections.

The benchmark tool used was siege. We ran siege on 2 separate boxes each with 180 concurrent connections and 0 delay between the connections, so the traffic was a full 360 users attempting to hit the server throughout the entire test. For those who use siege this was the command siege -f urls.txt -r 100 -c 180 -b -i. I will attach the urls used for the test. They were a mix of urls from a starter site. The test was of course performed on our lan. There were 3 switches between the server and the machines running siege.

The total test results were
Total Transactions : 35987
Availability : 99.97%
Average Response time : 0.85 seconds
Total Concurrency : 313.12

Here are the individual numbers from each siege machine.
Transactions: 17996 hits
Availability: 99.98 %
Elapsed time: 97.50 secs
Data transferred: 145.27 MB
Response time: 0.85 secs
Transaction rate: 184.57 trans/sec
Throughput: 1.49 MB/sec
Concurrency: 157.58
Successful transactions: 17996
Failed transactions: 4
Longest transaction: 28.39
Shortest transaction: 0.00

Transactions: 17991 hits
Availability: 99.95 %
Elapsed time: 97.65 secs
Data transferred: 145.22 MB
Response time: 0.84 secs
Transaction rate: 184.24 trans/sec
Throughput: 1.49 MB/sec
Concurrency: 155.54
Successful transactions: 17991
Failed transactions: 9
Longest transaction: 29.46
Shortest transaction: 0.00

We also have some similar numbers with 1.6.5 running on VMWare ESX Server. We will be sharing them soon here. Need to gather all the compiled data for it and specs for the test.

On of the two big, important updates you’ll see a lot of is the much improved edit mode interface, seen above.  Gone are the clunky, vertically columned edit, preview, and live mode buttons.  They’ve moved to the top, and been replaced by a clickable side bar that lets you hide the whole side console completely.  Some much needed UI facelifting was applied with nicer icons and a splash of color.  On the page, container interfaces are squared up nicer, and dropdowns were (mostly) eliminated in favor of one click editing, sorting, and deleting.  That doesn’t sound like much, but if you did much with Flash, you probably fell prey to the dreaded layer monster, where the popups would pop “under” the Flash interface, making it hard or impossible to click the options there.  All in all, the changes were largely cosmetic, somewhat functional, and much needed.  It just feels a little more put together now.

Example of the Widget Modal Window

The other big change is somewhat related, and that’s the addition of widgets.  Widgets have been discussed before, usually as pieces of Web Page Content that are just designed to be dynamic.  This is all well and good, but it didn’t really allow for proper segmentation of functional elements of a site.  Now, widgets are properly defined with a structure.  You set them up like structured content, except your fields basically serve as the variables for a macro.  Think of it as a UI for macros.  The primary difference being a macro can be embedded in content, while a widget goes in place of content.  Why this is so awesome is because one, like I said, it allows you to properly segment dynamic tools for your site, and two, even though a macro is only like a line of code, we all know that for some people one line is just one line too many.  This let’s them fill out a simple, non-threatening form, and see results.  The interface itself is cool, as adding a widget brings up a nice little modal window with all the widgets defined available right there in the page.  Plus, any container can be widget enabled, so that you can make containers fairly multipurpose, in the event that you want one container on a particular page to do something different, but still adhere to a basic layout option.

Speaking of the modal window, I would love to see this applied throughout the site to get rid of popups all together.  Get rid of the content search popup forms for reusing content. Use a modal window for quick adding or editing of content.  It makes people feel connected when they don’t physically have to juggle windows, especially when in some cases (like adding a picture to content), the popup adds information to a different popup below it, and it doesn’t go away when you click it, so you don’t actually get any feedback from the system that a change has taken place.  So there’s a wishlist item from me.

What makes this so worthwhile is that there is a definite maturing of the code base taking place.  The attention to the UI of edit mode, while not changing substantially in a functional way so much, shows dotCMS becoming a much more professional system that cares as much about the user experience as the functionality.  This is an excellent sign as we get ever so closer to what should be a real dam breaker of a version 2.0 next year (hopefully), and the rate of progression is fairly steady.  Like I said, rather than being 1.6.5, this would really almost qualify as a 1.7 in my books, but hey, I don’t do the versioning.

Extending the pageContent() Macro in dotCMS

// September 19th, 2008 // No Comments » // Software, Tech, Web

Here’s my latest nugget of info to help you extend dotCMS.  This isn’t anything complex by any stretch of the mind, but it should be a little time saver for those of you looking into doing blogs, archives, or anything else where you have several pages of content to flip through.  By default, dotCMS comes with a nice little macro called pageContent() that allows you to pull in a list of content, and get a set grouping of results based on where they fall in the sort order.  The only failing was that you had to write your own way to go back and forth with results.

No longer!  With this simple, handy macro, you can automatically get forward and backward navigation after you’ve written your foreach() loop for the pageContent() results.  It makes use of a few of the variables that are created when you run pageContent().  You can supply your own text for the navigation links, as well as give it a custom class name that will automatically be applied to all the elements so that you can format it as you wish.  The code for the macro itself follows below the example.  Here’s a sample of it in use, and what it looks like:
#set($showPageNum = "true")
#set($pageLinkVar = "year=${yearVar}&")
#pageContentNavigation("$page")

Example of the navigation macro

Here’s the code to get you going.  Just paste it into your /[$DOTCMSROOT]/dotCMS/WEB-INF/velocity/dotcms_library_ext.vm file, or wherever else you’re loading your custom macros, restart the server (if needed), and get going.  There are any number of ways you can modify it to suit your needs, so go nuts!  You can copy the code below, or download the pageContentNavigation_macro.vm file.

Gettin’ Your Macro On

// July 29th, 2008 // 4 Comments » // Scripts, Software, Web

I think it’s time to rock out with another dotCMS blog, and dig into some basic stuff that can be very useful to people getting started in the content management system.  There was a topic recently on the mailing list that was asking about where to look to see some examples of macros, and where you can save your own.  This looked to be a good topic to cash in on (speaking of cash, wanna loan me some?).

Macro output on a page

In dotCMS, a macro is basically like it is in other applications: it’s a collection of code one calls to reproduce complex processes in a simple, and straightforward fashion.  You know, to help stupid people out.  To put it in other terms, it’s effectively what one might call a function in other languages.  You give it a name, set parameters it can take, and define some output.  dotCMS comes out of the box with a lot of macros you can readily use, sometimes without even realizing it.  These include things like the commonly used pullContent(), and fancier tools like videoPlayer().  Naturally, dotCMS has a list of these built in macros, and how to use them.  If you want to take a look at example code, the built in macros are stored in /[dotcmsroot]/dotCMS/WEB-INF/velocity/dotCMS_library.vm.  Don’t mess with those though, because odds are you’ll break stuff.  Just trust me on this.

There are a couple ways to approach writing your own macros.  The first is to write a second .vm file, and store it with the default one in the velocity folder.  It will be loaded when the server starts, but requires a server restart when you change or update them, because they are cached in the system when it starts.  Conventionally, the file name dotCMS_library_ext.vm is recommended.  By convention, I just mean that’s what the dotMarketing programmer folks have recommended to me.  The other way is to use a static container, which you include first thing in all the templates you want the macros to be available to.  Macros edited and saved this way do not require server restarts to use, but naturally you must make sure you’ve included the container in EVERY template where you suspect your macros might be used (and also ones where they might not. Trust me, never underestimate the stupidity of your users).  Neither method is right or wrong, and you should just choose the way that fits your development style best.  I’m still looking for a solution that would qualify as chaotic neutral, however.

Okay, so naturally you need to have something of a working knowledge of Velocity code to write a macro, so if you’ve never taken the time to learn some, I suggest you go do that.  Mr. Falzone has talked about this issue, so go get yourself some learnin’ and stop trying to run before you can walk.  Now, get off my lawn. If you already know Velocity stuff, go anyway, he’s a nice guy and worked hard, so you owe it to him to read it and pretend like you’re new to the subject.  Don’t worry, it’ll be our little secret.  But, assuming you understand the basics, this should be pretty straightforward.  So, starting up, we need to define the macro.  I’ll use as an example the macro I wrote that pulls in a list of events from the calendar by a tag:

So, we start by calling the macro() function, naming it, and giving it a list of variables to expect.  You can have more than one variable, just list them one after the other (no commas or anything).  As you can see, before I open the macro, I list a couple variables in comments that aren’t required, but can be set before you use pullEventListByTag() and they’ll be pulled in later.  Next, I’m going to set three variables that I’ll use when querying for events, then I’ll build out the query and set it as a variable, and set how I want it ordered (date1 is the event start date).

Okay, next up I’m going to check for those optional variables, and if they aren’t set, it will give some default values.  After that, I’ll call the pullContent() macro (as you can see, you can call other macros within macros), and save the results in its own list variable.

That takes care of the heavy lifting.  Next I’ll start the output.  In this case I’ll build out a <div> to hold the contents, pump out a header, and start a standard foreach() loop for the results in $eventMacroList just as we would when using the pullContent() macro like any other time.  I told you this wasn’t hard.

And believe it or not, that’s it. Once this is saved and initiated in the manner you choose, you can call your custom macro with the parameters you’ve set, and it will execute the code. In this case, you could call pullEventListByTag(“football”) to return all events tagged football. On another page you could call pullEventListByTag(“baseball”) and get a different set, but still get the same structured output.  Such is the nature of macros.  Any process you would need to do over and over again, or need to simplify for less savvy users are prime candidates for macroification.  In simple terms, this example is nothing more than a macro to automate a pullContent() call and foreach() loop.  If you look at the default macro file in dotCMS, you’ll see how much more complex you can get with them.

So, the complete macro for this example would be as follows.  Feel free to copy and modify it to fit your own needs, or to play around with to get used to the process.  I’d be happy to see how you improve or modify it.  You can also discuss it in the forums, or share your own creations.

Getting my Fixx

// July 10th, 2008 // 3 Comments » // Scripts, Software, Web

I have not, and will not claim to be the tidiest person on the planet.  What I am, however, is a very organized unorganized person.  That is to say, while my desk looks like a mess to you, I absolutely know where every little thing is on it (so don’t touch anything).  Where my failing is, is that I tend to be forgetful, especially through the course of a day where I tend to switch tracks several times going from task to task.  This is especially a problem at work, where I will tell someone I’ll get something done, get distracted by some other problem, then forget about it.  This set me on a mission to find some software to help me out.

When I’m doing stuff over in dotCMS land, I am usually working with JIRA issue tracking software.  That’s what they use for dotCMS development tracking, and we’re using it to manage tasks on our site redesign project with them.  So I used that as a baseline.  While it isn’t the most clean, modern looking interface, it is very functional.  Besides, it’s also what I’ve gotten very used to, it’s comfortable.  But, it’s also not free.  There are some other systems out there I looked at, like JTrac, Eventum, Bugzilla, and a few others, but for various reasons, I didn’t run with them.  For instance, Bugzilla was too involved to install, and wouldn’t be very portable, Eventum I gave a pretty fair shot, and it  was very nice and very flexible, but had a very dated interface.  Most of the others I came across you had to pay for, or they were just too watered down.  Naturally, I posted to Twitter about my hunt, and along came the name of a product my search hadn’t turned up: Fixx.  Immediately, their interface certainly jumped out as the cleanest of the bunch, and they made used of AJAX (Asynchronous Javascript And XML) to speed things along.  It supports multiple projects which can have parts/components, you can give client access to projects, create custom resolutions and priorities, manage version releases, and even tie it in to Twitter for notifications.  It even supports OpenID.

Like JTrac, Fixx is a Java based system, and with my usage of dotCMS, I’m not uncomfortable working with that these days.  All that means is that you better make sure you have the Java JDK installed, and have your JAVA_HOME environmental variable set.  Fixx is not free for groups, but it is free for single user usage, which is exactly what I was going for (there is an educational license discount if you want to use it at a school or college).  When you download it, it comes prepackaged with Apache Tomcat, and running the startup file will set up a webserver on your machine on port 9000.  So, to access your new tracker, you just open http://localhost:9000/ in your browser.  In my case, I tossed it on my server at home, punched a hole in my firewall, and set up a subdomain on that port so that I can access it from any machine that I’m at.  All said, this was all surprisingly painless.  Before I put it on my server, I was even running it right off a thumb drive for a little while testing it out.

So, with it up and running, I jumped right in.  I’ll give them credit, they took their goal of a simple but flexible issue tracker to heart, and seem to have made a nice run of it.  While Eventum is pretty crazy powerful, the interface is dated and cludgy.  Fixx isn’t so flexible, but it’s just plain nice to use.  It makes me think of why Twitter remains so much more popular than Plurk.  Plurk is better, with more options and features, but there are times when simplicity really sells.  And Fixx still gives you enough options and controls to make it worthwhile.  You can do unlimited projects, set project leads, make your custom resolutions and priorities, set sub areas, move tasks to new projects, and plenty more. Basically, all the “important” functions are there.

Creating a task is as simple as most others: click the link, fill in the fields, and save.  But the interface that displays the task is soft and easy to look at (compare that to Eventum some time).  Information is easy to find, and it’s laid out in a manner that makes sense.  All your controls open up right on the screen, so you can move the task, resolve it, log time, attach files, comment, etc all on the same page without jumping around.  And again, they have tried to keep the options trimmed down a little, and aim more to make it fast, easy, and straightforward.  To me, this is absolutely perfect for the worker or freelancer doing a lot of things on their own.  For a larger group, the Bugzilla or JIRA path makes a bit more sense probably, but even then, there’s not much that those do that Fixx can’t.  The one big exception being nested tasks, or making a task dependent on others.  You can’t designate a task as a parent to another one or such.  That’s one big area I’d like to see change (and I already put in a feature request on the matter).

Overall, I’m impressed.  I haven’t seen another free issue tracker that is nearly as polished as this one is.  It also is perfectly suited to my particular needs.  I think anyone who needs a hand getting some of their tasks organized could stand to benefit from an install of this.  Heck, I even created a project for development of my Grayplicity WordPress theme, so now I can keep a nice bead on what I need to do to improve it, and store recommendations I want to follow up on as they come in.  What about you?  Is there a system that you are partial to, or maybe on online project tracker that you think is very good?  Share your thoughts.

Overview of the New dotCMS 1.6 Calendar

// May 14th, 2008 // 7 Comments » // Software, Web

With the release of dotCMS 1.6 came one of the more anticipated new features in recent months, their new “social” calendar system. This calendar represented a noted improvement over the previous event portlet (though the event portlet still exists for some additional functions, like scheduling and managing conferences, etc). This new calendar makes use of a portlet to manage the system in the backend similar to the old event system, and builds out on the front end through a combination of Velocity, .vtl files, and some layout bits. It also creates event entries as content in a structure, so that it can be queried and pulled no different from any other content in the system (note that you might want to make sure that you don’t currently have structures called “Event,” “Building,” or “Facility” already, as it will want to make them if you are upgrading. New installs obviously don’t need to worry about that).

If you want to take a look at the calendar first hand, you can see the example that is up at the demo site, at http://demo.dotcms.org/calendar/. To access the back end, login to the admin console by going to http://demo.dotcms.org/c and plug in the user ID test@dotcms.org and the password test. In there, you’ll see the tab for “Calendar” that you can click on that will take you to a similar, but stripped down looking version of the calendar. I’ll be using the demo’s site front and back end when describing features, and in the video below.

Part of the major improvement with this system is that as opposed to the event portlet, it provides true calendar functionality, whereas the event system provided more of a list view, regardless of how you tackled it. This calendar is very AJAX (Asynchronous Javascript And XML) heavy, and thus can present an accessibility problem by default, but since all calendar events are simply structured content, you can very easily create a standard HTML based variant for users with special screen needs (and really, do you know a calendar that is truly 508 compliant out of the gates in the first place?). Storing events as structured content gives you an extra layer of control over the presentation of those events, control that is unmatched in other calendar systems. The interface itself is fairly smooth and intuitive, without a lot of tricky links, or confusing navigation (I’m look at you, Active Data Calendar). Though fairly simple in terms of usage, the fact that everything just works, and works like it should, carries a high value. And in keeping it simple, it is actually a very powerful little tool.

The calendar itself supports all the traditional types of functionality: Time, date, place, multi-day, all-day, list-week-month views, RSS feeds, detail view and searching. Additionally, you can add events from the front end (if enabled), tag events, filter by tag(s), filter by calendar(s), attach files and images, provide detail popups, and provide automatic integration into Google Maps. Using categories, you can effectively create completely self contained calendars of different events, and one event can exist within multiple calendars simultaneously. All in all, it’s a pretty nice feature set. It will also save you trying to integrate with a third party calendar for your business that would require and supply it’s own login, back end, and functionality challenges. With some CMS (Content Management System)’s, they might have a calendar, but it’s an additional install, plugin, or module that you have to manage. dotCMS includes the calendar out of the box (still free), and if you don’t want to use it, you just don’t display the page in the back end, so there’s no extra labor involved. Other enterprise CMS’s come with built in calendars, but you’ll pay dearly for those packages.

The primary challenge right now is that if you do not load the starter site, or are upgrading from 1.5.x, you don’t have all the front end files (not to be confused with the back end admin portlet for the calendar, which is totally there whether you upgraded or installed fresh and will be in need of no special attention) that you’ll need to display the calendar (this will be changing soon, however, as I understand they plan on bundling all that with the CMS so you’ll have it if you upgrade or don’t use the starter site). What I did to solve this problem was to go to the demo site, and lift all the files out of their /calendar directory. This had one disadvantage: all the files are hardcoded to the /calendar directory. So, what I have done is taken all these files, and modified them so that you can upload them into your system, and they will adapt to wherever you want to put it. Just download this zip file which contains everything needed to display the calendar to your visitors, extract it, and upload it either with the multi file uploader or over WebDav. The only other thing you need to do then is make a page in whatever folder that will be your calendar, let’s just call it index.dot, and place the following code into a piece of web page content that you use on the page:

This will load and fire off all the rest of the Velocity templates. There are two CSS (Cascading Style Sheets) files (grids-min.css and reset-min.css) that are needed because the calendar was designed with Yahoo Grids, and those styles make sure everything lines up properly. A third CSS file (cal-base.css) that is included is the base starter site CSS, and you could remove it as necessary to meet your styling needs, but I included it at the moment for the sake of appearance reproduction. All three CSS files are included in the zip file above already, and you’ll see them referenced in velocity/calendar.vtl. Using the files from the zip file, and the content above, you now have a calendar that you can move to a new directory should the need arise, or should you simply not want it in /calendar. If you move it, just make sure you change the /absolute/path/to in your piece of content to point at the new location of the load-calendar.vtl file. You could even make that portion dynamic if you wanted to, simply by removing the first line from load-calendar.vtl, and adding it to the content, making it:

The only reason I didn’t do that by default was so that way you never have to copy and paste that set() from here (since I can’t “include” a piece of content in the zip file), and to keep the information going into the content item that tells it to make the calendar as simple as possible. Odds are, once you place the calendar, you won’t be moving it around anyway. But this way you don’t have to change all the paths yourself, and you aren’t locked into using /calendar.

Below, I have included a brief overview of the new calendar, so that you can see some of the functionality and how it all comes together.

Download the customized dotCMS social calendar front end files (.zip, 131KB)