Tuesday, December 16, 2008

New FamilySearch (Where we are now)


New FamilySearch (NFS) is a single family tree that all of us share and work on in common, as if we all shared one PAF file. FamilySearch is temporarily calling it New FamilySearch and it is temporarily located at http://new.familysearch.org. But eventually it will be called FamilySearch Family Tree and will be relocated to www.familysearch.org. The 1.0 version will be available to all, members of the Church and non-members alike.

However, the first priority for NFS was to stop the flood of duplicate temple ordinances by replacing TempleReady and the associated need to check the International Genealogical Index (IGI) to avoid duplication. Accordingly, a version 0.9 was written to this end. It lacks many niceties considered standard in a genealogy product, but those features are not central to replacing TempleReady and can wait for a 1.0 product.

After alpha and beta testing were complete, a multiple-phase rollout was commenced on 26 June 2007 when St. Louis started using NFS. From that moment on, NFS has not been in some extended beta test as some suppose, but has been in real use in real temples.

Early users of NFS found bugs, of course, as well as user interface problems. That is one reason for doing a phased release. But these problems were nothing, in retrospect. Like the hero of a tragedy, NFS 0.9 contained an unknown fatal flaw that doomed it to failure as soon as the rollout began. Ironically, the flaw arises out of the problem that NFS is designed to avoid: too much duplication.

Our hero's tragic flaw

Somewhere along the line, two conflicting mantras were established for NFS. Our hero's fatal flaw results from an unforeseen interaction between the two.

No one can change your data except you.
To keep things simple, New FamilySearch combines Ancestral File, Pedigree Resource File and the International Genealogy Index into one database.
Not knowing the number of duplicates beforehand, the first mantra was kept by keeping a full copy of every piece of data, no matter how often duplicated. After the rollout began, users began combining duplicates from these three databases. Some individuals were duplicated many, many more times than expected and when NFS users combined them, "individuals of unusual size" (IOUS) started to grow.

Steps were taken to slow IOUS growth. The addition of PRF disks was halted. The import of complete GEDCOMs was frowned upon. And NFS continued to perform its primary function, adequately replacing TempleReady.

The Arizona

Then on 5 February 2008 the rollout hit Arizona. Because of the large number of Church members there who are descended from early Church members, the growth rate of IOUSes exploded and IOUSes became large enough to swamp the computer servers running NFS. The system was sometimes too slow to use. I'm sure within a week FamilySearch knew they had big problems. The decision to freeze would have been gut wrenching and probably had to reach to the highest levels. On the 19th, word first leaked out. On the 21st, official word was sent out. The rollout was stopped, frozen solid.

Emergency steps were taken to rehabilitate system performance. Hard limits were placed on the number of duplicates that can be combined. (I believe it is currently 85?) GEDCOM import size was restricted. I imagine the length of the delay was predicated on how long it took database engineers to scan through all the millions of individuals in NFS to find and split the IOUSes into pieces small enough for the system to handle.This had to be done while the system was in operation, actively serving 26 temples.

The result was an NFS that worked near flawlessly as a TempleReady replacement for anyone who doesn't have ancestors that were famous or were members of the Church. These ancestors were the ones becoming IOUSes. When the freeze thawed, the rollout could continue only in districts where most members didn't have many ancestors fitting this characterization. By 14 October the tragedy had run its course. Utah, Idaho and Las Vegas have been on hold ever since, waiting for a true fix to the IOUS problem. Rumors have pointed to Q3 or Q4 of next year before this newer than New FamilySearch will be ready.

Family Tree

While all this was going on, work on a 1.0 user interface was progressing. Developers are using a system called Agile Development that encourages regular user interaction during iterative development. This allows us, the future users of the program, to try things out along the way, identify design flaws and influence the product before it is set in stone. If you have a New FamilySearch account, you should feel a responsibility to do this at labs.familysearch.org.

That brings you up to the unexpected announcement that Vegas was going live without the rest of us! What does that mean? Obviously, FamilySearch feels like the system is robust enough to handle the additional traffic and that letting Las Vegas start using the system is worthwhile, despite members inability to combine all duplicates together.

Originally published on "The Ancestry Insider" Blog.

No comments: