Toward Business in Real-Time

In 2002 I attended the PeopleSoft on Tour conference in Toronto. This was before the merger with JD Edwards and the subsequent acquisition by Oracle. The software industry was still trying to recover from its recent dot-com bubble. I had gone to the conference with the naive hope of learning more about PeopleSoft’s products and possibly landing a job working with them.

Dashed hopes notwithstanding, I was inspired by the keynote address presented by the then CEO of PeopleSoft Craig Conway. The theme of his address was “The Real-Time Enterprise” and it still reverberates today. Conway recapped the history of business reporting.

Annual reports came first, prepared for the traditional annual meeting and often synched with and necessitated by accounting’s annual tax reporting cycle. In fact, most of the content for the annual reports came straight from the accounting department. Annual reports for an enterprise were and still are also used for shareholder and board meetings. Although nice if the news from the previous year was good, annual reports are not very helpful in correcting problems before they get worse when the news is bad. It’s hard to imagine today, but at one time these reports had to be compiled manually.

As computerization made reporting faster and easier, reports for the quarterly board meeting became the norm. It soon followed logically that monthly reports would be even more useful. Monthly reports allowed managers to make mid-stream adjustments based on the quarterly board meetings to help avoid having to report something bad at the end of the year. Reporting had moved from the boardroom to the manager’s office.

Reporting cycle times have hovered around 24 hours or less for decades now. Most companies have data up to midnight of the previous day available for reporting and planning purposes. Conway’s vision of a “Real-Time Enterprise” involved reducing the time of reporting cycle times to zero, showing the actual current state of the company.

Many companies — even those not using PeopleSoft/Oracle products — now have “dashboards” that display KPI’s on one convenient screen, mimicking manufacturing process control systems, although unlike those manufacturing control systems, most dashboards are not yet truly real-time. Users now have information at their fingertips that allows them to steer companies with more control than ever before. In just over ten years’ time, the vision of the real-time enterprise has almost become reality.

US Government Surveillance – A Historical Perspective

In 1925, during Prohibition in the US, former police officer Roy Olmstead was convicted of running a huge bootlegging operation in the Seattle, Washington area. The conviction was largely based on transcripts of wire-taps on Olmstead’s telephone, a relatively new evidence-gathering technique. Olmstead appealed his conviction all the way to the Supreme Court. Olmstead claimed that the wire-taps were in violation of his rights under the Fourth Amendment to the US Constitution which protects US citizens against unreasonable search and seizure.

In 1928, the Supreme Court upheld Olmstead’s conviction, but Justice Louis D. Brandeis gave this prophetic dissenting opinion:

“The progress of science in furnishing the Government with means of espionage is not likely to stop with wire-tapping. Ways may someday be developed by which the Government, without removing papers from secret drawers, can reproduce them in court, and by which it will be enabled to expose to a jury the most intimate occurrences of the home.

…if the government becomes a lawbreaker, it breeds contempt for law; it invites every man to become a law unto himself; it invites anarchy. To declare that in the administration of the criminal law the end justifies the means—to declare that the government may commit crimes in order to secure the conviction of a private criminal—would bring terrible retribution. Against that pernicious doctrine this court should resolutely set its face.”

It was not until 39 years later that the US Supreme Court changed its view on privacy. In 1967 the Supreme Court overturned the conviction of Charles Katz, who had used a telephone booth to relay bets as part of a gambling operation. The telephone booth had been bugged by the FBI using an external microphone.

Justice Stewart explained the ruling as follows:

“The Government’s activities in electronically listening to and recording the petitioner’s words violated the privacy upon which he justifiably relied while using the telephone booth and thus constituted a ‘search and seizure’ within the meaning of the Fourth Amendment.”

A year later the US Congress passed the Omnibus Crime Control and Safe Streets Act which authorized electronic eavesdropping and wire-tapping as long as a warrant was obtained.

Another 46 years down the road and things are again complicated by further technological advances:

  • Can someone talking on a cellphone in a public place justifiably expect privacy?
  • Does gathering information for later analysis constitute search and seizure?
  • Is tracking someone whose GPS-enabled smartphone is broadcasting their location illegal?

Watch this educational video on the website of the American Civil Liberties Union for a primer on what can already be done using modern technology:

BI minus IT

As I read Business Intelligence (BI) newsletters, company websites and blogs a common theme emerges: people are sick of waiting for IT to deliver what they need because if and when they finally do it is usually way too late.

This is not new. Many years ago, while working as a product manager, I watched order desk staff struggling with catalogs the size of phone books (both thankfully now obsolete) and took some night school programming courses so I could create a computerized cross-reference of competitors’ part numbers. Thinking I would be congratulated, I was instead reprimanded. (The order desk staff thanked me.)

Flash forward a couple of decades and not much has changed. One place I worked, users knew more about SharePoint than the person who was supposed to be administering it and were quietly using features like workflows that IT considered “dangerous” (translation: features IT did not understand and did not feel they had to learn or support).

Movements like BYOD, personal clouds and self-service BI are user rebellion against the old “if it ain’t broke, don’t fix it” mentality of most IT departments. When all arguments are exhausted, IT usually falls back on the “security” catch-all objection.

Most modern business analytics tools stress the same feature: getting it done quickly without IT involvement. At this rate, most IT departments will be reduced to simply “keeping the lights on” and will become increasingly irrelevant when business strategies are being formulated and executed.

Dr. Michael Stonebraker

Just finished listening to an interview with MIT professor Dr. Michael Stonebraker explaining why traditional databases are obsolete.

The interview is available here on the Software Engineering Radio site:

Seventeen years ago, as a newbie professor at a private technical college I was thrown into teaching database fundamentals — a theory course that no other professor appeared to want (an attitude reflected in many of my students it seemed). Preparing for the course I poured over my own undergrad notes and reviewed the seminal work of Dr. E. F. Codd and his paper from the early 1970’s that eventually formed the foundation of relational databases as we know them today. Everything that I have learned about databases since then has adhered to Dr. Codd’s model — until now.

Dr. Stonebraker is a Dr. Codd for the 21st Century and in his hour long interview explains why databases have to move on from their traditional row-based model to one that more easily handles the vast amounts of data being generated and used today. Dr. Stonebraker is not out to tear down Dr. Codd’s work, but rather to build on it as someone who has spent decades working with something that was just a theory in Dr. Codd’s day. During the interview he uses solid practical examples drawn from modern business models. He comments on the main players in what is coming to be known as NewDB. He also plugs his own start-up company VoltDB.

Overall, the effect is both exciting and scary for traditional database practitioners.

Starting small with Big Data

It’s difficult to read a tech blog or IT newsletter lately that does not mention “Big Data”. Definitions of what Big Data really is can be hard to come by. One of the best I have found is by Patrick Schwerdtfeger and is available on YouTube at

A simple explanation is that Big Data is the third wave of computer data.

The first data wave was data entered manually into relational databases and then queried later. This is still going on judging by the number of “data entry” positions available.

The second data wave is also still with us and is what most people think of when they think of data and databases. This is data that accumulates as a by-product of other computerized processes such as accounting or CRM. It may be imported from text files or spreadsheets. It might be the result of adding the sender of an email to your contacts list or posting something to your Facebook account. The amount of data accumulated can be enormous and often must be stored in “data warehouses” for reporting purposes.

The third data wave is a veritable tsunami of automatically generated “machine data”: cell phone logs, online reservations, GPS locations, CCTV records. This data is all around us all the time and is constantly changing. It is “live” data that is too big to store in any one place but must be accessed and utilized on the fly in other ways.

If you have a website and want to begin taking advantage of Big Data, a good start would be setting up and using Google Analytics. It is a free service that provides amazingly in-depth information about who is visiting your site, how long they stay, whether they return, where they are from, what browser and operating systems they use, even what their screen resolution is. With some practice you can create custom reports that allow you to judge the efficacy of your individual web pages and marketing campaigns, that is if the available standard reports and dashboard views are not enough.

Your company does not have to be big to use Big Data; in fact Big Data can empower smaller firms that do not possess the resources for large data centres and data warehouses.

Database Application Fundamentals: ANSI/SPARC architecture

ANSI/SPARC architecture is a design strategy for relational database applications that separates database applications into three components:

– Physical Data Storage
– A Database Management System
– User Views

Physical Data Storage refers to the actual hard data — the bits and bytes of digital information written to a device like the hard-drive in the server. This storage is managed by the computer’s operating system and typically arranged in “files” and other types of pre-allocated space on the disk.

Data stored in digital format would be useless without a means of cataloging it for later retrieval. The database management system, or DBMS, is equivalent to the card-file system at a library, allowing users to quickly search for and find information based on various search criteria. The system also allows the librarian the all-important task of filing new or modified information in the appropriate place so that it can be easily located later.

Finally, User Views are the part of your database application that users see and interact with — the so-called “front-end” or graphical user interface (GUI). Graphical user interfaces could include windows and menus on the computer screen as well as output sent to other devices such as printers and fax modems.

The purpose of maintaining this functional separation is flexibility and scalability. Any of these three components can be modified or replaced with minimal disruption to the others. When an application is created based on the principals of ANSI/SPARC architecture, it is fairly easy to create a new interface for existing data, or even use more than one interface simultaneously. For example, at the office you could connect to your data using an application running on your network while your salespeople and even customers and suppliers could be connecting to the same data using a laptop and a modem from a hotel room or a web-page running on a browser at another office on the other side of the world.

Similarly, the actual stored bits and bytes of data can be transferred to a larger, more powerful server that may even (depending on your choice of DBMS) be running using a different operating system with minimal disruption to the application the user sees on the monitor.

Business Software: Rent, Buy or Build

My older sister worked for many years as a costume designer in the movie business (or “film” as they prefer to be called).

Despite the artsy veneer, the movie business is a business and a very bottom line driven one at that. In the costume department they had a very simple rule:

“If you can rent it, then rent it. If you can’t rent it, buy it. If you can’t rent or buy it, then build it.”

The same thinking can be applied to the software you use to run your business.

Custom-coded software is hands down the most expensive and time-consuming way to go. Even larger organizations can seldom justify re-inventing the wheel when the same idea has been covered by dozens or even hundreds of others.

Even buying software tools can seem questionable when the licensing costs are as high as the obsolescence rate.

“Renting” software on a monthly basis can make a lot of sense. Monthly billing satisfies accounting’s “time principle” and requires no capital cost allowance calculations at tax time. If you need more, rent more. If you need less or the product turns out to be crap, then simply stop renting. Many software rental plans include cloud storage by definition and allow access from anywhere and any machine with just a web connection.

Something to think about the next time you consider shelling out hundreds or even thousands of dollars for software…

Data is power. Power to the people.

It seems that not everyone likes the idea of Microsoft’s new Report Builder 3.

Sure, it’s loaded with features like shared datasets, an Office-like interface, and wizards that make it easy for power users to make the move from spreadsheets to ad hoc reports that they design themselves. But it seems this can cause some job-security angst amongst uneasy IT people used to holding the keys to the kingdom and doling out half-baked reports to the data-starved masses.

There’s really no need for concern, as the shared datasets still need the skills of a dba or developer familiar with the back-end data and all its esoteric table and column names and datatypes. In fact, freedom from the chore of mundane report creation might actually mean that dba and developer skills can be applied where they are needed in more creative database applications and automation projects.

Here’s hoping that requests from consumers of data for Report Builder 3 grow in intensity and persistence until all the political roadblocks at your workplace are flushed away.


This wasn’t even on my radar until the senior accountant at work asked about creating xbrl files to submit information to an Ontario government agency. Seems that xbrl files have been around since 1998. Where have I been?

XBRL is an acronym for eXtensible Business Reporting Language. It is an xml variant designed to standardize financial report submission. It is already required by many government agencies and financial bodies around the world, including the SEC in the US and just about every government department in Australia.

Although there is software available to aid in the creation of xbrl files or “instance documents”, most tools seem to be geared more toward the creation of custom “taxonomies” (basically dsd files).

I plan to see if I can create xbrl output files directly from SQL Server in a manner not unlike outputting data-driven html source code from MySQL using php’s “echo” function.

I’ll keep you posted…

Is traditional web development dead?

I’m just discussing my sister’s website with her and we have decided to try a “blended” approach using normal “test and ftp” content and some newer features such as WordPress.
I’m guessing that once she starts blogging she won’t stop.