IBM is not going to send out a press release
by Volker Weber
When IBM canned DoesNotWorkplace, Lotus General Manager Mike Rhodin told the press it "was only an experiment". I would add, an expensive one. I am not sure what they will call NSFDB2, but I would add again, an expensive one.
Take this as an unsubstantiated rambling, but consider what I just said. Might save you money.
Comments
Hmm. Well, well.
There still needs to be a general ability to hit NSFs with SQL queries, not unlike NotesSQL:
Vote for it
I think NSFDB2 was a good idea that got caught in the wheels of "perfection-itis". It was not perfect for every possible use of Domino but it was good for some situations and some applications. However, it seemed there was something driving the theory that it had to be perfect for all possible use cases and thus it got developed, redeveloped, further developed, post developed, over developed, and eventually it just got to be too late. Kind of like a good red-wine - not a great red wine, but a good one. It should be consumed at the right time and waiting too long leaves you with something less than it once was.
Well if you want do joins with Notes-sourced data, XPages throw up some intriguing possibilities in that area. Just sayin’ :)
Or you could use Lotus Approach to perform joins of Notes data and create reports, and normalize multi value fields :-D
Now Carl, that’s just crazy talk ;o)
NSFDB2 works just fine with Domino, if you use it right. You need to know which methods to avoid, and how to workaround them.
@Mika .... on Domino on iSeries?
@Craig, it might be possible also on iSeries, but I don't have enough experience with Domino on iSeries to confirm that. It works on normal OS's though, like Windows and Linux.
Mika, it does not really matter anymore.
The whole thing driving it was customers who said they wanted to standardize on a single storage backend. And they'd like a pony. So IBM tries to deliver. Customers see all the shortcomings when they look close, because what they wanted really doesn't make sense from a tech perspective. NSF wasn't designed by dummys, but many at IBM, particularly in the DB2 business, think it was. They thought it would be easy to re-implement on DB2.
Somebody finally realized the whole thing is in a deep hole. And if you find yourself in a hole, you have to stop digging.
Well, if NSFDB2 will not be supported anymore, it will have negative impacts on both Domino and DB2:
1) Domino can handly only 64GB, and it's actually is quite horrible already at 32GB because the compacting and indexing takes forever.
2) DB2 will not be used by many Domino companies anymore, if it isn't free, like it was with NSFDB2. I think MySQL is already challenging Domino's existence for certain projects.
3) There will be no more possibility to use SQL joins and commands in Domino which use the same database as Domino can use. Then it's either full Domino database, or full MySQL database. Of course you can use MySQL/InnoDB as database from Domino also, and only use Domino as the web server. But then there might be also more efficient solutions to do so, like PHP.
Anyway, the loss of NSFDB2 will clearly seperate Domino from SQL databases, and although Domino might still work fine with larger databases, it can not gain the additional benefit of using SQL databases as storage, like very large and complexly joined databases.
I thought Domino should be always improved, and not made worse.
It was an interesting idea. It did some interesting things. It also failed for many of the wrong reasons. Sadly. It was a real pain in the butt to set up and try. If you weren't already a db2 shop and also an expert at interpreting the directions, and an admin on the db2 environment you were targeting -- then forget it.
The "spoils" of Workplace became the foundation for current products (I won't comment on market place damage, it's the feeding and hand thing here), so there is hope for NSFDB2 too. Just a little dream:
Bob Picciano, the current Lotus GM, was the lead architect for pureXML in DB/2. (For the uninitiated: pureXML allows to store XML documents and fragments in a DB/2 database and use xPath/xQuery expressions to get data in/out). There is DXL that can express any note (design and documents) as XML. Once that is feature complete (which is a top priority demand by the Lotus community) some intriguing possibilities open, that were not available when work on DB2NSF started (pre pureXML / pre DXL complete). Ideally an interface between Domino and DB/2 would transmit DOM objects rather than XML strings (but there are people smarter than me to figure that out).
My 2c
:-) stw
Technically speaking, there's not much more IBM needs to do to bring Domino up to the next level DB-wise.
Domino 8.0 finally got rid of the last of the 16-bit memory handles. If IBM continues to push into the 64-bit realm with Domino, we should see the expansion of DB sizes past that 64GB limit.
All of the latest compression options obviously help to reduce size. DAOS helps move attachments out of an NSF, which is a huge help in many cases. But more importantly, the abstraction introduced with DAOS lays the groundwork for other abstraction.
For example, getting the $Collation item on view design docs (i.e. a view index) OUT of the database would be a huge help. It's not replicated, so why even store it there? Many complex Notes apps have over 50% of their space in view indexes.
And with complex indexes comes a serious performance hit. Domino locks an entire NSF any time ANY write is performed to it. This includes updating view indexes, a doc update, etc.
As a result, getting view indexes out of the NSF files themselves would shrink the filesize further, likely allow for faster simultaneous index updating, and possibly allow for truly complex JOIN indexes to be stored on-disk.
Throw in the capabilities NSF currently has (solid replication, readers/authors security, an ultra-flexible storage model), and it's not far from being the obvious db of choice for 90% of most modern applications, provided that somebody knows the query language (@Formula).
After that, flesh out the SQL querying capabilities and it's a slam-dunk.
But let's hope IBM wants to do this work.
There is DXL that can express any note (design and documents) as XML. Once that is feature complete…
I’m not holding my breath on that one; how about you?
Well if IBM continues to store everything outside of an nsf then the Notes Storage facility one day will become blatantly fast (bad people will say that is because there is nothing left inside). I do like Erik's idea but I have fear that the world will not wait another few years for IBM to come out with something great. Not that I was a big fan of nsfdb2 but again much time and effort has been wasted. Still I keep asking myself how a product that is as open as Domino embracing so many open standards (all not my words) is unable to store its data in a SQL database?
You may consider standing in line behind some DB2 folks.
@Henning - My response got a bit lengthy so I've put it on my blog: http://www.bleedyellow.com/blogs/erik/
I think we are seeing a general trend away from the relational database towards schemaless object data stores anyway, especially in the platform-as-a-service space. Think Google BigTable, Force.com datastore, CouchDB.
The NSFDB2 thing did a few very valuable things for the Lotus product line, IMCO -- even if it has failed as a specific feature (which I believe it has, even if that isn't an official decision at this point).
First, it helped finally make the argument around IBM that not all data is relational data. Semi-structured data like Notes has its own kind of value. I personally like to make the distinction between "DATA" -- which tends to be raw, voluminous, and hopefully structured -- and "INFORMATION" which is what you get when you apply thought, process, or interpretation to raw data. Information is well handled in Notes databases. Data is handled better than in the past, but it really does belong in an RDB if its past a certain volume.
Learning that you can't just stick Notes "Information" easily into a normalized relational database without loosing much of the value of an RDB was an important lesson. I liken it to learning that you can take the Domino "Services" and simply replace them with the same services provided by J2EE without destroying the value proposition of the platform.
Second, it gained more attention from the DB2 team looking at the NSF as a data container. The NSF continues to improve as a container in each version. Moving to DAOS will be a huge addition to that value because it will take the throughput of all that data out of the path to otherwise maintaining the file. Moving the indexes out as distinct files rather than as note item attachments to the view design notes is a really interesting idea to me.
In the end, I suspect nsfDB2 is a very expensive feature to maintain, and not used to a significant degree by IBM's customers to justify that expense. If that's true, I'm in favor of pulling it out. It just may reduce some of the bulk a little bit too.
DAOS ... somehow it reminds me of Shared Mail. Hopefully it's not the same thing with another name because Shared Mail sucked like hell. Shared Mail was and still is a nice idea but was a horrible shot in the dark.
That's probably true for all major mistakes. You can learn something from them. Imagine what could have been done with all those development resources.
DAOS (removing attachments from mails, and storing them only once instead of multiple times in all recipients' mailboxes) is an example. This comes after how many years? I always have these nagging question if you can improve a product by leaps and bounds: I thought it was already optimized? Why not years earlier?
Simple answer: You cannot do everything at once. You prioritize. And if you do the wrong things first, you lose. Unless your competition makes more mistakes.
There are so many things that could be done with NSF, for instance moving the view indexes and the search indexes from expensive, resilient, backed-up storage to inexpensive storage. There are millions of frustrated users out there who think Notes can't search. Why? Full text index too expensive and switched off by corporate IT.
Putting search indexes into a separate tree that can be moved to cheap storage would probably has saved more Notes seats than NSFDB2.
Now imagine Domino would be able to create valid CSS-friendly XHTML output. Just imagine.
Ah, thanks Vowe, for that valuable piece of be-prepared-news. I always thought the NSFDB2 was quite messy and I could never really see the business value, so this comes as no surprise. Rather to spend the funding on providing better import/export capabilities (e.g. LEI, just less... once again, messy) or a proper SQL tool (like the old NotesSQL, also a bit... messy but just needed some bug fixing) or, first prize, providing excellent web services capabilities (almost there, still... messy though with regards to consuming services).
Notes has always excelled when things were kept simple and it was require to solve problems fast: quick business value provided by combining GOOD developers and a rapid development environment. I think the RDB backend is a great idea, but then you need to approach it from the database side and tackle it as a classic three tiered application, not start off with a quick Notes application, front-end driven with little concern for tables and field relationships, that also, automatically, will create some (really ... messy) backend tables in an RDB if you require it. That just makes no sense...???
Comparing nsf to Google Big Table? The demo setup stores against 1786 machines with 2 400 GB harddisks each. I like that.
I am not sure if I fit well into the DB2 camp but I do like the idea that your data is not locked in a storage facility that currently has no use outside of the Notes and Domino world.
Now imagine Domino would be able to create valid CSS-friendly XHTML output. Just imagine. --@vowe
We'd all really like this. I know major improvements are being made, but not all the way there yet. This was high on Bob B's list as well, but just didn't get prioritzed ahead of some other things. I'm sure I don't agree with all the prioritization decisions made, but then, I never do.
DAOS ... somehow it reminds me of Shared Mail. Hopefully it's not the same thing with another name... -- @Ralf
Shared mail was terrible. DAOS is a far, far different and better animal. Based just on what's in the public beta, its going to be a very important feature and I believe will be widely adopted. Instead of a hacked mail-file specific solution, DAOS is a fundamental part of the NSF and server container system now and is well designed from what I can tell. Someone who's technical abilities and attention to detail are above reproach (Daniel Nashed) has literally beat the hell out of it in testing and seems to approve. That's a big winner to me.
The issues around DAOS that will be a challenge will be in building experience, best practices, and tools for backup and restore issues.
Putting search indexes into a separate tree that can be moved to cheap storage would probably has saved more Notes seats than NSFDB2. --@vowe
I think at one time, major customers were pushing for nsfdb2 with the idea of simplifying backup and data storage farms. I'm sure the idea sounded better to non Domino people than it did to us. Who knows what pressure there was for the feature. IBM does, IMO, spend too much time listening to a few giant customers who, if they do migrate, will hurt a great deal -- instead of paying better attention to the lower end mass market. I think the DP program is an attempt to balance that, and may be helping but it takes time.
Moving those .ft directory trees to alternate storage locations would be a HUGE help. I think it could be done using symbolic links in linux, (and it can be done in ntfs with the right utility as well) but I haven't tested this.
re "if you can improve a product by leaps and bounds: I thought it was already optimized? Why not years earlier?"
I agree with your simple answer about prioritization. But there's another answer, and Ray Ozzie alluded to it back when Damien Katz posted his essay about the Notes formula engine rewrite: it was optimized -- for the capabilities and limits of the hardware and OS environments it was designed for 20 years ago, and for the jobs it was reasonably being asked to do 10 years ago. Ray was talking about parts of the formula engine, but the same thing applies to so many things in Notes and Domino.
@Henning Nobody is comparing NSF to BigTable. I was simply pointing out that traditional relational databases like DB2 aren't necessarily where enterprise data is going to be stored in the future. Who really cares if your data is stored across 1 or 1786 machines when it is in the cloud and you don't maintain the infrastructure? Forget about the Google research study and try the real world implementation for yourself ( http://code.google.com/appengine/ ).
Well, first thanks for the news, I was thinking on getting more info on Lotusphere about NSFDB2 and probably spend some time before the event to try a demo on getting iSeries DB2 data into Domino views. So I'll better use my time in other stuff.
I went to the Lotusphere session dbs, I was pretty sure I had seen NSFDB2 mentioned there, and in a quick check I just saw one Jump Start session that mentions DB2.
We had a good use for NSFDB2 in showing DB2 data in Notes. I guess we'll continue looking for a good solution (using LEI now).
NSFDB2 is a way of storing NSF data in DB2. LEI is about making DB2 (and more) data available in Notes/Domino. Two different things.
As someone who has presented on NSFDB2 for a couple of years now, the huge elephant in the room for me has always been that this feature was implemented on Windows/Linux platform, but not on the iSeries. Since this is where a large proportion of DB2 shops are implemented, if they'd been able to run it natively on the iSeries, I think there would have been more adoption of the feature. As it is, they have to install a DB2 server on windows or linux and then attempt to federate data from the iSeries DB2 server to be able to integrate with Domino. At that point, it's a non-starter.
I understood that NSFDB2 also added the ability to show DB2 data (Query Views) in a Notes/Domino application, so it could be used to show some federated data from another DB2 database (besides the one that stored Notes data) ... maybe that part will survive :) ... or maybe it's one of the reasons of the death, too many people (like me) trying to use this feature for things it wasn't built for without understanding it first.
Where I've come from (10 year Notes/Domino development) the NSFDB2 feature was huge ... the proverbial dogs ***. Only becasue it did 2 things:
+ Gave you SQL based views allowing the join of Notes data.
+ Because the views were SQL based the view could be far more dynamic - e.g. factoring usernames or other parameters to present the data.
I did a presentation at my then work showing how this could effectively solve the aged old "reporting from Notes" problem - which I know Andrew has alluded to before.
And I know Notes does information well but the ability to arrange that information in relational ways is fantastic - even for the most simple Notes client apps.
And now as an almost 2 year .Net/SQL developer I totaly appreciate the rapid platform that Notes provides. MySQL might be cheap but connecting code to a SQL db is still far more work that the Notes/Domino model - so the additional of a SQL datastore I always thought could really make Notes a platform of choice for a lot of applications (lets face it - most applications in business don't store anywhere near 64Gb).
The killer for me, as has been mentioned, was the setup - surely they could have made it as simple as a Domino install -or created a Db2 install for Domino!
I think I'm essentially agreeing with what is written above - and its going to be a real shame if it gets pulled. If it does - please IBM (or should that be Santa), get a replacement for JOINed data and dynamic queries.
"NSFDB2 is a way of storing NSF data in DB2. LEI is about making DB2 (and more) data available in Notes/Domino. Two different things."
LEI is more than just making DB2 data available in Notes. LEI can replicate between Notes and DB2, similarly to two Notes databases replicating, meaning native Notes data can be pushed, selectively and on schedule, to an RDB environment. It can also update a custom-built RDB in real-time, as the Notes data is manipulated via documents in a Notes/Domino front-end.
The difference is that the RDB linked to Notes through LEI is typically designed by a decent RDB architect, using proper data modeling techniques and logic, filtering the Notes data appropriately and making the correct relationships; as opposed to NSFDB2 which automates the database build using Notes internal non-relational architecture and pushing ALL metadata and data. So, although NSFDB2 then provides access to the NSF data in DB2 it is typically in a crude data model, and it is open for debate on whether the information is really that much more accessible in the end.
I'm not saying I'm against the idea of an RDB backend for a Notes application: I think it is crucial and brilliant. I just think the NSFDB2 tool was crude. I would absolutely love being able to do proper three-tiered development in Notes, using it for what it is great at: Fast and intelligent GUI development. And then be able to plug in my own RDB design (and CHOOSE the platform, not be DB2 bound) and have freedom on where I want to do my processing (Notes/Eclipse, etc). Now that would really make my day.
The use of db2 for replacing the NSF wsn't bad, but the DAV & Query Views are really powerful. It's going to maintenance mode, so it's still usable as a reporting mechanism.
It would never run on iSeries until the iSeries was upgrade to use db2 9.
The first production release that was available to all is D0mino 8 and it required db2 v9.13 to run.
I use it to pull reports from the iSeries using db2 federated data. The Notes database is empty, no documents, just query views to pull reports.
Using it with decs/lei lets you open a form and have data there.
There isn't anything else to match the power of query views in Domino.
