I recently asked a number of logicians, historians of logic, and logic enthusiasts the following question:
“When exactly and why did logic stop being a core requirement for every single educated person in the west and become seen as a technical, niche elective that only a tiny fraction of educated people know anything about?”
I received 2 responses in the Ontolog Forum:
John Sowa said: "I blame Bertrand Russell. He wanted schools to stop teaching traditional logic and replace it with symbolic logic. He got 50% of what he asked for."
Chris Menzel, a logic professor at Texas A&M, said: "I attribute this far more to the utter havoc wrought upon higher education by conservative, specifically brainless Republican, politicians. They've managed to transmogrify our once glorious system of state universities to a collection of education 'dealerships' whose purpose is to provide a 'service' to their 'clients' that guarantees them a high paying job in business or industry. The on-going gutting of the liberal arts has been a sad consequence of this."
There were numerous responses at Academia.edu:
John Corcoran said: "I have never given this any thought, but it is an interesting question. One thing to bear in mind is that over the years logic got competition from 'critical thinking' and kindred subjects."
The discussion below is from John Corcoran's session titled CORCORAN ON LOGIC TEACHING IN THE 21ST CENTURY at Academia.edu:
OLAP cubes are among the most powerful resources available for business intelligence and analytics. Here's what the product manager for Microsoft Analysis Services said about OLAP back in 2002:
OLAP multidimensional databases combine incredible performance with unsurpassed analytical power and, in my opinion, are the foundation of the BI platform.
The multidimensional data model is vastly superior to the relational data model when it comes to the expressiveness of analytical operations. The ability to have random access to any point in space, both detailed data and aggregates, makes it a breeze to express calculations that would otherwise take pages of SQL statements using a relational database.
This remains true about OLAP today, and it is likely to remain true for a long time. While technologies change, the underlying concepts remain the same. But OLAP seems to be falling out of favor recently. Why would that be?
The top Google search result for OLAP claims "OLAP Is Becoming Obsolete". This is actually a paid advertisement that invites users to download a paper titled Selecting the Right Database Technology for Your Business Analytics Project. Interestingly, it says very little about OLAP and does not even use the word obsolete anywhere in the paper. And the only negative thing it says about OLAP is the single sentence below:
Storing the results of these pre-calculations takes exponentially more storage resources than the actual raw data does, limiting the size of raw data that can make up a cube to gigabyte scale.
But that is actually false, because the most popular storage format for OLAP cubes are multidimensional structures which require far less storage than the original source data. Production OLAP cubes have exceeded more than 20 terabytes of raw data, and their continued growth is limited only by computing power, memory and storage.
In the real world, there is no reason to believe that OLAP is becoming obsolete. But there are more signs that people think it is, notwithstanding the facts. The statement below is from the Microsoft website more recently:
For new projects, we generally recommend tabular models. (rather than OLAP cubes)
Microsoft has been a leader in promoting OLAP, so why are they now downplaying it and steering people towards tabular models? The reason they give is that "tabular models are faster to design, test, and deploy..." But that is true only when the requirements and data are very simple. When the requirements or the data become more complex, even a little bit, the complexity of developing a tabular model explodes and quickly becomes far, far more complex than a multidimensional OLAP project with exactly the same requirements and data. And the end products are less flexible and far less able to adapt to changing requirements.
So what gives? Why would Microsoft make a claim that relies on an implausible assumption of simplicity which does not exist in most enterprise environments? And why would another company claim that OLAP is becoming obsolete in a paid advertisement with only a single dubious claim to back it up?
Here's what I think is going on: OLAP is most valuable when the underlying source data is clean and well-integrated. And clean, well-integrated data is difficult to achieve for many organizations. Claiming that OLAP is obsolete is a marketing ploy to promote products that work well with poor-quality data and data that is not well integrated.
There is a legitimate need for such products, and they have a huge market potential – but vendors should make that case and sell those products without claiming that OLAP is becoming obsolete because it is not. For organizations with clean, well-integrated data OLAP is far and away the best choice for business intelligence and analytical applications.]]>
Big data can provide powerful insights into large data sets. Some scholars and practitioners have even suggested that big data tools and techniques might replace relational databases for ordinary business use, but this claim offers only false hope for organizations that struggle with relational databases. Here's why:
Big data systems work with information organized into small 2-part chunks known as key-value pairs (or some related format). For example: Last Name = Fuller; City = Redmond; Car = Honda Accord; Order status = complete.
Organizing information this way is great for things like analyzing trends and detecting patterns. But big data formats cannot be used for ordinary business reporting unless each record is tagged with additional information to tell which other records it is related to. For example: this address belongs to that person; this item goes with that order, and so forth. Applying these kinds of tags to information in a big data format requires exactly the same kind of discipline and pre-planning as it would if it were organized for a relational database. Big data offers nothing new in this regard.
Even when a big data record set includes complete information about the relationships between each pair, big data technologies do not offer anywhere near the flexibility of relational databases for reporting purposes. So any claim that big data presents a plausible alternative to relational databases for general business use is uninformed and false.
The idea that managing information involves cost-benefit tradeoffs is not new, but unfortunately those decisions are treated as engineering issues focuing on implementation issues like performance and storage, while business utility is ignored. By business utility I mean the capacity of an information resource to meet expressed and unexpressed needs and adapt to changing requirements.
The statements below were made by Don Chamberlin who was one a co-designers of SQL, the world's most widely-used database query language. He is describing decisions made in the mid-1970s to give users more flexibility with cost-benefit tradeoffs:
“When the original SQL designers decided to allow users the options of handling nulls and duplicates, they viewed these features as minor conveniences, not as major departures from orthodoxy, taken at the risk of excommunication."
"SQL trusts the database designer to decide whether the costs ... are justified. To impose these costs on all applications ... seems a little heavy-handed, and seemed even more so in 1975 given the costs of storage and processing at that time."
Even today (in 2015) Dr. Chamberlin and his former colleagues continue to bear harsh criticism for those decisions. This is completely unfair because they did not force anyone to do or ignore anything. Rather, they gave users the freedom to make their own cost-benefit decisions.
But unfortunately those decisions, and others of equal importance, do not fall to the stakeholders who pay the price of poor choices or who are in any position make good ones. Instead they are decided behind the scenes by technical professionals, while information owners remain unaware that such tradeoff opportunities even exist. The result has become that information systems are too expensive, do not adapt well to changing requirements, and do not meet the expectations of their owners.
Information owners do not need to become technical experts to make good decisions about organizing information, and it is absolutely essential that business experts, rather than architects or engineers, have final decision authority. The role of IT departments in these matters should be strictly limited to cost-benefit advisement.
Even when technical professionals carefully consider cost-benefit tradeoffs they are constrained by a technical perspective. The following statement is from a paper titled Justifying database normalization: a cost/benefit model:
"the determination of appropriate normal forms frustrates many systems analysts ... a trade-off exists among system performance, storage, and costs."
But the author omits a critical factor: the primary overall consideration in any cost-benefit equation should always be business utility. Technology professionals cannot be expected to reocgnize the full range of implications for business utility in any given tradeoff scenario. They must instead rely on a set of requirements specifically spelled-out in advance by the business customer. That is why scope creep is such a huge problem in most IT projects, and why most information systems cannot adapt well to changing requirements.
It would probably be impossible for a business owner to make a complete list of every possible use-case for a given resource. But a business owner can easily determine whether a resource can satisfy their foreseeable needs even before any requirement has been expressed. What an owner considers to be foreseeable can change over time, but it will always change more slowly than the perceived requirements when technology professionals have to guess or fill-in the blanks. New technologies can expand the range of use cases for information, but for information to be usefull it must still be organized in a way that allows the desired use – technology cannot change that.
For related discussion see:
The Fundamentals of Information Management is the title of a presentation I gave to various audiences at Microsoft Corp. in 2014 and 2015.
Not every discipline is based on fundamental principles. When someone talks about the fundamentals of marketing, or the fundamentals of business communications, for example, they are usually talking about introductory concepts or things you should know if you want to succeed in those endeavors. Other disciplines however, such as chemistry and music, are deeply rooted in fundamental principles or laws of nature. Achieving a level of professional competence in that kind of field requires not just mastery of concepts and tasks, but a sound understanding of the underlying principles.
When fundamental principles are discovered within some discipline, or when techniques based on those principles are developed, the result is often significant advancement and rapid growth of that discipline. For example: in the field of chemistry the existence of chemical compounds and reactions were known long before the structure of the atom and its fundamental properties were understood, but alchemy was a primitive and unpredictable craft. The publication of the periodic table laid the foundation for the scientific discipline of chemistry and allowed it to advance very quickly.
Music is also based on fundamental principles, and people have created music since prehistoric times. The ancient Greeks understood that music had mathematical properties, but early notation systems did not fully reflect that understanding. As a result, early written music could be used to help musicians remember a tune they had already heard but was not a reliable way for someone to learn one they had never actually heard. Modern music notation brought significant improvement because it allows the mathematical elements of a composition to be described in precise detail, and allows even the expressive elements to be described in some detail. The standardization of modern notation led to an enormous and rapid expansion of creative output during the golden age of classical music which has continued ever since.
The field of Information management is not generally recognized as having a basis in truly fundamental principles. Sources using the phrase "fundamentals of information management" usually include content ranging from tips on data governance, to best practices for business intelligence, to lists of terms and definitions for various information technology topics. Regardless of the merit of these sources or how mportant such topics may be, none of them actually reflect any connection to truly fundamental principles.
The most persuasive evidence I have seen pointing to a fundamental basis for information management comes from the profession of accounting, which is the discipline of managing financial information. Modern accounting has evolved from the practice of double-entry bookkeeping which has been used since at least 1299 AD, and was first described in a published work in 1494 by the Franciscan monk Luca Pacioli, who was the most influential mathematician of the Renaissance and a friend to Leonardo DaVinci.
The preface to Ancient Double-Entry Bookkeeping (1914), which is the earliest modern English translation of Pacioli's original treatise, indicates that the author recognized a basis in fundamental principles for double-entry bookkeeping. He wrote:
It is a significant fact that the rules and principles elucidated by Pacioli are contained in a book given over to mathematics. One cannot help but believe that the derivation of double-entry bookkeeping is an explanation of the algebraic equation used with such skill by the ancient Greek mathematicians, applied practically to the scientific recording of business transaction.
The algebraic equation he refers to is logic of Aristotle and Euclid. As a database professional, this struck me as significant because the theory underlying modern database technology, known as the relational model, is based on principles of mathematical logic discovered in the late 19th and early 20th centuries which also have roots in the logic of Aristotle and Euclid.
With this extraordinary historical information I decided to look further; and in January, 2014 I discovered something remarkable that has enormous positive implications for business and the future of commerce: The system of interconnected books (which I'll refer to as relational bookkeeping) used in double-entry bookkeeping is precisely consistent with the relational model. This means every object and process can be described in terms of the relational model and can be operated upon using a relational language:
In other words, Italian merchants in the Middle Ages invented fully-functional, manually-operated relational database management systems nearly 700 years before the mathematical foundations of such systems were to be formally described.
This astonishing fact stands as evidence that computers and technology have not fundamentally changed the basic nature of information management as has been assumed since the 1970's. It reestablishes that information management is a business discipline, rather than a technology or even a hybrid discipline; and that the business discipline of information management is, indeed, based on fundamental principles. And not just any fundamental principles but principles considered by some to be the most fundamental of all principles. Kurt Gödel, the best friend of Albert Einstein and whom their colleagues saw as his intellectual equal, described mathematical logic as:
a science prior to all others, which contains the ideas and principles underlying all sciences" (1945)
Compare this language to the following statements about double-entry bookkeeping by economist Werner Sombart in 1919:
Double-entry bookkeeping came from the same spirit which produced the systems of Galileo and Newton, and the subject matter of modern physics and chemistry.
Double-entry bookkeeping is based on ... the basic principle of quantification which has delivered up to us all the wonders of nature, and which appeared here for the first time in human history in all its clarity.
Sombart and others were sharply criticized for supposedly overstating the nature and significance of double-entry bookkeeping. But that was before anyone recognized its congruence with modern mathematical logic – notwithstanding Sombart's claims quoted above, which is understandable, since relational bookkeeping was practiced for more than half a millennium before mathematical logic was developed and Sombart did not reveal that he recognized any connection – but it is very clear that Gödel and Sombart were talking about the same thing. With this new understanding it is hardly possible to overstate the significance of relational bookkeeping. It represents the ultimate intersection of theory and practice – the ne plus ultra of practical application emerging from the pre-primordial fabric of the cosmos; with obedience to the same principles of natural logic underlying not only the behavior of matter and energy, but our ability to contemplate and manipulate them.
Sombart goes even further and suggests a spiritual connection:
...we cannot regard double-entry bookkeeping without wonder and astonishment, as being one of the most artistic representations of the fantastic spiritual richness of European man".
Along those lines the following excerpt shows that people have recognized the significance of logic in our relationship with the universe for a very long time:
John 1:1 In the beginning was the Word, and the Word was with God, and the Word was God
Orig. Greek En archē ēn ho Lógos, kai ho Lógos ēn pros ton Theón, kai Theós ēn ho Lógos
Logos is the ancient philosophical concept of divine reason, which Heraclitus described as both the source and fundamental order of the Cosmos. It is also the word Aristotle used to describe his system of formal reasoning which became the basis of modern mathematical logic. Christians believe Logos in the passage above refers to Jesus Christ, "in whose name", Luca Pacioli wrote, "our transactions must always be made". Muslim scholars embraced classical logic during the Golden Age of Islam when it was the most scientifically advanced civilization on earth. Hindu clerics also independently described principles of formal logic in religious texts known as Sutras.
The purpose of my course and lectures on the Fundamentals of Information Management (FoIM) is to explain how logic is not just a tool for technical specialists to query information in databases, but must also be recognized as a tool for information owners to express requirements for information resources. This will allow business organizations to finally gain effective management control over their information and achieve new capabilities that have been impossible without a solid understanding of the underlying principles.
Dr. Thomas Haigh tells a fascinating story in The Business History Review that explains why information management is seen as a technical discipline rather than a management one: The systems men were members of the Systems and Procedures Association during the 1950s and 60s. The purpose of this association was not to promote research or continuing education, but rather to seek increased status and management authority for its members within their employing organizations. They offered an implicit bargain to corporate executives:
"You put us in charge and we’ll deliver to you more power over your firms than you’ve ever dreamed of" (p 43)
Executives for the most part were not convinced. They understood that technical expertise does not translate into management ability. By the 1970’s the Systems and Procedures Association was defunct, and the various roles of the systems men merged into corporate IT departments. But they left a stubborn cultural legacy that persists still today, the idea that managing information is a job for architects and engineers rather than business experts:
"For better or worse, to speak of something as an information system continues to imply that it should be engineered by an information specialist and built using information technology" (p 59)
"It seems unlikely that the idea of information can ever truly be separated from these roots: it is just too historically and culturally charged" (p 59)
The full article is available here, and a free version here.]]>
Here are 3 ways to lower costs and improve outcomes in any BI or analytics project, or for that matter any information management effort. They require varying degrees of commitment ranging from easy, free, and doable right now, to a need for significant change in organization and culture:
Database normalization is a dubious concept for 2 reasons:
As an example of reason 1, the table below contains an error which you can see in the last row: an address in Grand Junction, Tennessee has the same ZIP code as the address above it in Grand Junction, Colorado. This is clearly a mistake, but without investigating further we cannot know whether the error is with the State or the Zip.
Breaking City and State off into two separate tables as shown here eliminates any potential for this kind of inconsistency, however the advantage might come at a cost because performance can suffer when applications and reports have to query 2 joined tables instead of 1.
So what is more important, better performance or lower risk of error? Who should decide? Only information owners have the appropriate perspective and incentive to make that kind of decision wisely (see Sophotaxis). Architects and engineers can provide valuable cost-benefit advisement, but final decisions should be made by owners.
The example above is pretty simple, but large business databases can have thousands of similar cost-benefit scenarios that can be far more complex. The more complex the situation, the greater the need for ownership perspective and expertise in the specific business issues at hand.
For an example of reason 2, consider a simple table containing only one column with phone numbers. Many engineers would agree that this table is in first normal form. In most situations it would work fine just as it is, or as a column in a larger table. But for a company such as a telephone service provider, where users would want to group or sort phone numbers by area code or exchange, this table would not be in first normal form. In other words, it would not satisfy even the minimum theoretical standard for use in a modern database system. Instead, in this particular case, the numbers should be broken out into each meaningful component as shown in the lower image.
This shows that even in very simple cases the normal form of a table can only be determined by someone who understands the full-range of potential uses for the information. Business databases are filled with situations like this, but unfortunately most decisions about organizing information are made by technologists rather than business experts. That is why businesses struggle with databases that cannot adapt well to changing requirements.
Information should be organized in a way that makes the most sense to its owner, not according to some predetermined normal form which cannot even be reliably achieved. The normal forms are guidelines to improve performance and protect consistency. But any decision made for performance can potentially degrade the usefulness of the information, and the range of decisions needed to protect consistency can only be determined by a business expert.
A far more useful and comprehensive tool for thinking about the way information should be organized is the principle of sophotaxis, which I explain in The Ancient Secrets of Information Management.
For related discussion see:
A widely held assumption in the academic field of business information systems is that technology and behavior are inseparable, but this is false. One widely-cited source that promotes this idea is Design Science in Information Systems Research, where the authors state:
"Technology and behavior are not dichotomous in an information system. They are inseparable (Lee 2000)"
The source they cite (Lee 2000) explains their rationale:
"The problem of 'technology vs. behavior' is a dilemma in the following way: If we take a technology approach to IS, then how would we be different from engineering and computer science? But if we take a behavioral approach to IS, then how would we be doing research that any behavioral field could not already do?"
"Just as a physician cannot design a remedy for a patient’s body and emotions separately, and just as an architect cannot design “form and function” independently, the IS field similarly does not have the option of designing the technology subsystem alone or the behavioral subsystem alone – we have to design both together."
My response to the first paragraph is this: Deciding how business information is organized has nothing to do with engineering and computer science and everything to do with management priorities and objectives. The role of technology professionals should be limited to implementation and cost-benefit advisement. This should be an easy distinction to make, but the conventional wisdom is clouded by historical and cultural biases. For example, technology workers in the 1950s persuaded business leaders to think of information management as an engineering discipline instead of a management prerogative. This is discussed further in The Legacy of the Systems Men. There are plenty of purely 'behavior' related issues that the academic IS community could focus on which have nothing to do with technology, such as the problem of poorly-organized information, which I describe further in Sophotaxis.
My response to the second paragraph is that it is false. Decisions about how information is organized can and should be made before any automated system is created, even when those decisions can change (and they always do). The design of the automated system might raise cost-benefit issues that will impact the information decisions, but those cost-benefit trade-offs cannot be well-understood until decision makers know what the trade-offs will entail, and that is possible only when the information resource is defined first (see Cost-Benefit Value is Ignored).
When business-oriented priorities are subordinated to technology-oriented factors without a deliberate cost-benefit analysis, operational and analytical capabilities suffer. Organizing information is an act of business administration, which is a role IT departments are not intended for. The role of IT should be limited to implementation and cost-benefit advisement.
The following statement is evidence of how deeply the issue of information management is misunderstood. It has been repeated in various forms more than ten thousand times by authors at universities, technology companies, and government organizations:
"With the proliferation of information technology starting in the 1970s, the job of information management had taken a new light, and also began to include the field of data maintenance. No longer was information management a simple job that could be performed by almost anyone. An understanding of the technology involved and the theory behind it became necessary. As information storage shifted to electronic means, this became more and more difficult."
This statement is false; information management has never been a simple job that could be performed by almost anyone – at least not since the birth of modern accounting in the late 13th century. The techniques used in manual accounting systems rely on a set of cross-referenced and interconnected books which use structures and rules that are fully consistent with the theory behind modern relational database systems, which I explain further in The Ancient Secrets of Information Management.
Accounting is the discipline of managing information about money. The same logic-based techniques can be used to manage other kinds of information as well, but manual accounting is so painstaking and time consuming that it is easy to understand why early merchants only made the effort with the kind of information they considered to be most important. When relational database software was introduced in the 1970s, business leaders and scholars mistakenly assumed that it had created an entirely new computer-based method to organize information. But in reality it created a new computer-based way to automate old logic-based techniques that had been used with success for 700 years. If this were understood the shift to electronic automation would have made things far easier rather than more difficult.
Early merchants certainly did not look to the craftsmen who made their tools to also define their accounts. But that is exactly what modern organizations do. It makes no difference that the old tools were made from paper, feathers and dye, and the new ones from computers, software, and networks. The old tools served exactly the same purpose as the new with respect to the organization of information. The new tools serve an additional purpose of automating processes and workflows, but that is no reason to believe that the engineers who create the tools should also be responsible to organize information. And there are important reasons to understand why they should not.]]>
Managing information is the most difficult and costly operational challenge facing most businesses. At the root of the problem is a failure to recognize the distinction between information resources and technology resources. To their detriment, businesses treat them as the same thing. My evidence for this claim is that no distinction is ever made in the requirements expressed for either, or in the way each is managed. They are delivered and maintained by the same people and no distinction is recognized at any point in the lifecycle processes of either type of resource. Information resources are mistakenly treated as components of automated systems.
As a result, some of the most important management decisions at every level of enterprise organizations are unwittingly delegated to technical specialists instead of business experts. Efforts to address the resulting problems without addressing the root cause only make the problems worse. It is a vicious circle that creates thick layers of artificial complexity in the form of initiatives, roles and processes which lead to additional costs and complexity. The only way to solve the problem is to recognize that information resources are not the same thing as the technology-based tools used to access and maintain them. Businesses must develop a capacity to determine and express requirements for information resources separately from those of automated systems.
Read more at The Ancient Secrets of Information Management. To learn more about organizing information to meet the needs of its owners see Sophotaxis,]]>
Information is the sine qua non of all commerce – a status not even money can claim. Money is, after all, a form of information.
A business resource is anything that brings value to a business. Classical economists described business resources in terms of factors of production. Land, labor and capital are the primary factors because they do not become part of any finished product and are not consumed or significantly changed by the production process. Resources such as raw materials and energy are secondary because they are derived from the primary factors. From the classical perspective even things like entrepreneurship, intellectual property and the time value of money are derived from labor and capital, so they too are considered secondary formulations of the primary factors.
So where does information fit in? Information is obviously an important business resource, but is it a primary factor or secondary? Or is it something else?
Information is consumed in the production process but not in the sense that it is depleted or reduced; in fact new information is created by every act of production and commerce. Further, information is non fungible, which means it cannot be substituted one unit for another such as a kilowatt of electricity, an ounce of gold, or a computer. Information cannot be replaced the way a building or an executive can be. No business resource can be effectively utilized without information.
For these reasons information must be acknowledged as superior to every other business resource. It is more primary than the primary factors. Information is the sine qua non of all commerce – a status not even money can claim. Money is, after all, a form of information.
As late as 1946 there were in the combined professional, technical and scientific press of the United States only seven articles on the subject of information
So why did the classical economists not have anything to say about information as a factor of production? My guess is that information is so essential to every aspect of commerce that until the mid 20th century it was not even recognized as a distinct resource class. In 1963 a professor of management noted “As late as 1946 there were in the combined professional, technical and scientific press of the United States only seven articles on the subject of information" (see here).
Information is like the air we breath – nothing can happen without it, but it is easy to ignore until you have reason to notice.
An information resource is information organized for some purpose. It can take the form of anything from a memorized telephone number to the Library of Congress or the entire internet. The following table lists various types of information resources, how they are organized, and what they are useful for:
|Information resource:||Organized by:||Useful for:|
|File cabinet||Drawers with alpha or numeric sorting||Manual document retrieval|
|Novel||Sentences, paragraphs, chapters||Entertainment, relaxation|
|Library||Subject, author||Finding publications|
|Relational database||Tables, columns, rules, relationships||Flexible storage, retrieval and analysis|
|XML file||Tags, nested hierarchies||Transporting and sharing data|
|Big data||Key-value pairs||High-volume capture and processing|
|Semantic ontology||Triples (subject, predicate, object)|| Making information discoverable
The way information is organized determines how it can be used, so decisions about the organization of information should be carefully considered by the owner and managers of the resource. Unfortunately, owners and managers usually only provide high-level guidance, and the actual decisions about the way information gets organized are instead delegated to an architect or technical specialist. This is a costly mistake with long-term consequences. The outcome is almost always an information resource that cannot be used the way its owners intend without being modified for every newly desired use.