18 comments

• 

  Chris

  10 months ago

  Hi there... in your article "The Disadvantages of Microsoft SharePoint 2007 as a Document ..." you state that "Documents housed within database" as beeing one of the big disadvantages of Sharepoint...
  
  I was wondering why you thought this... if the database houses metadata, index information etc, as well as the image/raw doc why is this a problem... does not the benefit of having a database supply the data integrity for all items better than having to worry about the joys of links/tags to an external data store...
  
  As you may see at DL we have created a Online Document Archving solution (Instant Intelligence Archiving) which we sale via a Channel using a SaaS model, and all the documents/images are stored with a database along (but in a seperate DB) with indexing data (ocred text, index information supplied by the user, etc)... we did this to help with our own DR process... Yes there is a speed issue with getting Blob data out of the DB, but with the speed of processors that exist now this speed hit is becoming less and less of a problem.
  
  I would be very interested in hearing your thoughts....
  
  Kindest Regards
  Chris
  reply
• 

  ChangeForge | Ken Stewart

  10 months ago

  Chris, thank you for e-mailing me, and I would like to thank you for stopping by ChangeForge. I greatly appreciate your question.
  
  First, let me qualify that I am not a database engineer or DBA. That aside, I work in a position whereby I have been exposed to a small number of CMS/DMS solutions to include some big names like EMC (Legato) Application xTender, and some smaller ones you probably have never heard of.
  
  So here's my take:
  
  We have 2 differing formats for CMS/DMS prentations: 1) the unstructured and "crawl the sprawl" route (e.g. Google), and 2) the highly structured route as in traditional CMS/DMS offerings. I am focused more on the latter, just to clarify.
  
  Traditionally, metadata is stored within a structured format to increase the transactional return of information - and to increase overall transaction speed and efficency. You even see this in Business Intelligence (BI) software where they are cubing data to help increase the return of large volumes of information. However, in most cases of document management we are not in need of this high a computational load as would an operations company at a billion dollar+ organization. Again, my article was focused more around SMB's - which I would think would be appropriate to your SaaS offering as well (not having looked indepth at the offering).
  
  To clarify, my statement was geared more towards what I consider maintainability of the infrastructure. As you know, text is smaller and can be compressed moreso than binary image files (traditionally TIF, PDF, BMP). As such, thought would indicate searches on raw text should be much faster than having to parse image files.
  
  Second, in maintaining the necessary archives (in an on-premise solution) keeping the image files outside of the database can make for much cleaner backups. Traditionally, backup agents handle backups of raw files (in an NTFS file format for instance) much more cleanly than in very large databases. Usually, the image repository of a CMS/DMS is the largest part of an installation - so making this as flexible as possible is to the benefit of the maintainer.
  
  Third, ability for administrators of the CMS/DMS soluiton to access and maintain images is very key. We have found it much easier to manage documents outside of the database in instances where an image file has gone corrupt (or thought to be corrupt) and we can access the file directly. This usually happens in situations where the originals are often and quickly destroyed once reliability of the system is established. You might argue security as a counterpoint to this, and this is a difficult challenge but one that can be answered generally.
  
  Last, and to harp on the backups, many solutions I've worked with support multiple DB's (e.g. SQL, MySQL, Oracle, DB2, etc.). I have worked with a MySQL version of a databae where the images were stored within the database, and major backup software vendors do not (at the time of my research) make an agent that allows for differential and/or incremental backups, thus making restoration a very dangerous thing - especially in situations where documents are destroyed very soon after initial scan.
  
  I would submit that I am not familiar with IIA architecture or design - and have no doubt CMS/DMS development may one day over come this. At this point, my experience over the last 3 years has led me to this conclusion. This is not completely scientific, but many ECM vendors and experts alike also share my opinion. SharePoint has some limitations outside of this as well, as I have learned in working with one of our Microsoft Gold Certified Partners that recently conducted an indepth study for a worldwide automotive corporation.
  
  Again, this is not to say storing the documents within a database is a bad thing in a SaaS offering. I might enjoy taking a tour of your software as time permits over the next few weeks. I firmly believe both SharePoint and SaaS have a huge role to play in the CMS/DMS space, and I have on-going research to do in these areas.
  
  Obviously, you e-mailed me so I was wondering if you would be agreeable to me posting this conversation thread in Discus comments? If not, I will abide by your wishes and look forward to continuing this conversation.
  
  Thanks for making me think about this,
  Ken
  reply
• 

  Chris

  10 months ago

  Thank you for such a detailed reply and on the weekend as well…
  
  Firstly I am more than happy for you to publish this conversation, and also give you permission to edit it as you see fit. From your reply and the articles you have published (the ones that I have read) it would appear you have no vested interest in editing our conversation to change the context of my thoughts…
  
  I would agree with many of your points, if not all of them. At Data Liberation we have worked with images, within our DMS application, but also with our Data Capture (OMR) application, and we moved very quickly to using a SQL database for storing the images after enduring the pain of lost files etc within file systems.
  
  Just to cover a couple of point you raise, as I mentioned in my first email we use two distinct databases one database for meta/index data and one for images/documents. Our system takes what ever it is supplied and stores the file as a blob image in one database, thereby ensuring the integrity of the documents (users can look at this, but can not update it, but can of course add new versions). If we recognise the file type (e.g. most image formats, Word, RTF, TXT, PDF etc) the system will either OCR the document or strip the text of the document out and store this in the other database. Additionally users can add there own metadata to the file. By having the all this text based information in one database we are able to perform queries to the documents very quickly and then retrieve the document only if the users requests it.
  
  As we use MS SQL 2005 (with sights on SQL 2008 before the end of the year) we have the benefit of being able to do incremental backups of either database. In our case we do log backups every 15 minutes on both database giving us almost continuous backup protection, full backups happen over night.
  
  The one advantage that you highlight with regards to direct access to documents, in the situation where the file has been corrupted. This I would agree is much easier with a file system and extremely (in comparison) difficult with a database. My response to this is that by using a database and the additional integrity that a SQL database provides is that it would be very unlikely that a single document would be corrupted, with a great chance that the entire image database becoming corrupted.
  
  I think we can both agree no matter what approach is taken, backups and the backup strategy is vital to any CMS/DMS system. The systems become a hugely valuable resource to a company and the loss or even partial loss of any of the data contained within them can be potentially devastating.
  
  I am very happy, when time permits you, to supply you with any information you would like on iiArc. The only area I would have reservations on is the way that we implement encryption of the uploaded documents/images, but otherwise I would thoroughly enjoy defending our approach.
  
  As you mentioned SaaS and Sharepoint will have a huge impact on the CMS/DMS market in the coming years… It could be easily argued that Sharepoint already has changed the CMS/DMS landscape massively already… and I believe that some of the current bigger players within the SME market will need to change their sales models and products sets to meet the more demanding and much better informed clients that now exist, or run the risk of losing market share and potentially disappearing all together…
  
  Kindest Regards
  
  Chris Morgan
  
  Managing Director
  
  Data Liberation Ltd
  reply
• 

  ChangeForge | Ken Stewart

  10 months ago

  Chris, this sounds very interesting. The architecture indeed sounds strong - especially given the SaaS aspect of your offering. It makes me wince to even think of backup/restore operations on DB's, but I have had to do many for various reasons. I would also submit, this is one very strong advantage for SMB's to look at SaaS offerings - to which I penned an article as well. There is still the classic, on-going debate on "owning the data" versus "renting the application" - especially in mission critical or applications.
  
  With regards to the shifting marketplace, I would whole-heartedly agree. Microsoft, if not by education alone, has shifted the landscape already. I look for the future of CMS/DMS to have many consolidations and many closures... That being said, I would venture to say you are positioning your company very smartly if trends continue.
  reply
• 

  Steve

  9 months ago

  I have a few limitations outlined at http://sharepointdocumentmanagement.wordpress.com/
  
  The big one lies in the fact that Microsoft recommends keeping Document Libraries at 2000 objects or less for performance reasons. Some serious planning needs to take place, especially if you are using it as a repository for scanned documents.
  reply
• 

  ChangeForge | Ken Stewart

  9 months ago

  Steve, thanks for the lead on that. I have heard many of the same things. MS's 2007 offerings for SharePoint technology has taken a huge leap forward, and it is getting noticed my many IT consultants who have an inside track with SMB's... They are instantly comfortable with a 'free' MS product and are strongly recommending its rollout for starting the chain reaction of document management.
  reply
• 

  MSMatt

  6 months ago

  This is not accurate. The 2000 limit is for rendering only meaning you should only have up to 2000 items at any level. With a good folder structure you can have up to 5,000,000 documents per library. You can also have 5,000,000 documents in a library without folders but you would then have to use a content query web part or something to view the library and page it at 2000 items per view. You can have upt to 50,000,000 documents in any content source for search as well. Here are the true limits of SharePoint libraries:
  
  http://technet.microsoft.com/en-us/library/cc26...
  
  I have experience deploying SharePoint to very large corporate law firms that have over 10,000,000 documents and SharePoint performs fine. The Library of Congress also uses SharePoint. SharePoint scales very well.
  reply
• 

  ChangeForge | Ken Stewart

  6 months ago

  Awesome news, MSMAtt! In reading through some of this technet article, I can see a definite increase in throughput when introducing structured document libraries.
  
  I have 2 points of contention, to which I would hope you could help me clarify:
  
  1) In my humble opinion, I discourage use of folder structure as this reintroduces the same mess a typical file server might introduce by obfuscating needed files unnecessarily. The object of an EDM is to surface information for easy findability at all levels and by running various queries to return expected search results.
  
  How is it that folder structures within SP aid in "findability"? This would seem to limit it in my humble opinion - especially when the folder structure becomes extremely deep.
  
  2) When dealing with such large numbers of images, how can returned results be faster in SP since images are stored as BLOB's in the DB, vs. traditional EDM solutions that segregate meta data from images (and even OCR tables)?
  
  PS - I hear MS is going to move away from this in 2009?
  
  MSMatt, I would love to hear your opinions of overcoming these challenges as you obviously have some experience in enterprise SP deployments vs what I have worked with in smaller deployments (primarily WSS).
  reply
• 

  T.C.

  6 months ago

  Recently, I received a support call from a user enquiring why doing a search in a MOSS site on a folder name didn't come back with any result while searching the parent folder name came back correclty. Upon testing, it seems MOSS search would return folder names up to 5 levels deep. Beyond that, only content is included in the search results but not folder names.
  
  Can anyone confirm?
  
  Thanks.
  T.C.
  reply
• 

  ChangeForge | Ken Stewart

  6 months ago

  T.C. I asked an SP developer friend of mine, and they had no exposure to this, nor do I. I typically work in a WSS environment.
  reply
• 

  Racing School

  6 months ago

  I don't think Microsoft Sharepoint is making any inroads into big corporations nor is it making a big impact in Asia and China.
  reply
• 

  ChangeForge | Ken Stewart

  6 months ago

  Racing Schools, interesting. Did you have data to support this? I would be very interested to see this as my research and those MS partners I'm dealing with are working with very large, multi-national organizations for overarching SP initiatives.
  
  What I am seeing over the past 2 years has been a ramp up of rogue, departmental installations of SP, and now the enterprise is stepping in to determine if/how they can make SP work across the enterprise.
  
  With over $1 billion dollars in MS licensing being sold in this last year, this would indicate MOSS is being actively deployed. Also, keep in mind, this number does not include the instances of WSS deployments (as it is free) and ISV professional services to customize both WSS and MOSS installations.
  
  As I said, though - if you have data to counter these trends I'm seeing, by all means I would love to post here to give a more accurate picture of what might be happening.
  
  Thank you very much for swinging by... I love hearing from readers!
  reply
• 

  SEO

  4 months ago

  I am intend to give a presentation to a group... Now i am evaluating how to apply this Microsoft SharePoint in my existing enterprise..??
  reply
• 

  ChangeForge | Ken Stewart

  3 months ago

  SEO, I must've missed your comment earlier last month. Think of SharePoint as network play-dough. It is pliable, and even more so if you have .net development expertise. Also, it is generally very reasonable to hire in "outsourced" expertise to assist with standing this solution up. We use it internally for dashboarding and workflow, with some minor document storage for always used docs.
  reply
• 

  Web Design

  3 months ago

  this post has opened me to some more critical aspects, I was already planning to shift to share point server, but I guess I need to study more details about it... now.
  reply
• 

  ChangeForge | Ken Stewart

  3 months ago

  SharePoint is a great solution for many things, Web Design. If you have programming skills, it is even better. We use it heavily in our company for many purposes, but EDM is not one. There are more options appearing to surface your existing EDM with a SharePoint front end as well, and even 1 I've seen that is built in SharePoint. That said, SharePoint is really a platform at this point and I frankly disagree with Microsoft touting it as an actual EDM - now a CMS yes, but EDM no.
  reply
• 

  Arthritis

  1 week ago

  I think Sharepoint loses it's edge in the document management system market to big boys like Xerox because it does not really offer any hardware solutions. I hope Microsoft might want to look into that or partner some of the digital copier manufacturers who are lacking in this area. This would form a formidable partnership.
  reply
• 

  ChangeForge | Ken Stewart

  1 week ago

  Art, I don't think SharePoint was at first intended to be an EDM solution, but let's face it... I don't think Microsoft really needs to enter the "copier" arena to be a formidable solution. Microsoft sold a billion dolllars in software licensing at the time of this writing, and this isn't even including all of the free software they offer and professional engagements from consultants and the like.
  
  In the conversations I've been having, everyone is scrambling to partner with Microsoft, and it is rumored that in their next release of SharePoint they are leveraging some options like keeping the documents and metadata separate. Given development items like this, Microsoft has always been happy to sell you their solution or allow you to use their solution as the portal and have an EDM vendor simply skin their application for them.
  
  Either way - they win!
  reply