Our tenth newsletter:
WE WON! The New York City marriage index 1930-1995 will be free and open data!
Reclaim The Records has done it again. We are proud to announce that our Freedom of Information lawsuit in the Supreme Court of New York, fighting for the right to a first-ever public copy of the New York City marriage index, has been successful!
The New York City Clerk's Office announced their intention to settle with us on Tuesday, July 5, 2016, less than twenty-four hours before they were due to face us in court on Wednesday, July 6. A draft version of the stipulation and settlement papers were agreed to yesterday, Thursday, August 4, with two small items awaiting final sign-off, and we expect the final papers to be signed in the next week. The City has already started duplicating the microfilms for us, and our attorney has given us the all clear to finally tell people the good news.
This data set, which covers sixty-five years, contains the index to about three million New York City marriage records. The vast majority of the content in this records set has never been available to the public before, anywhere, in any format. Not online, not on paper, not on microfilm, nada, nowhere.
This is the second time in less than eleven months that Reclaim The Records has had a successful outcome in a lawsuit against a government agency for the right to free and open genealogical records. Our first Freedom of Information case last autumn was against a different agency, the NYC Department of Records and Information Services (DORIS), fighting for a copy of the 1908-1929 marriage index stored at the NYC Municipal Archives. This new case covers the later years of those marriage index records, 1930-1995, which are maintained at the New York City Clerk's Office. A small number of these early records are searchable (but only on microfilm) onsite in lower Manhattan, but the vast majority of years of this data have never been seen or accessible by the public in any form, ever.
And now they're going to go on the Internet, for free, forever!
Here's how we did it, step by step
- December 14, 2015: Reclaim The Records writes a friendly "heads up" e-mail to the New York City Clerk's Office, letting them know that we would soon be submitting a records request to them under the New York State Freedom of Information Law (FOIL). You can read that letter online here. We even provided for them the legal precedent under which we would be making the request. (For the legal nerds out there, it's Gannett Co., Inc. v. City Clerk's Office, City of Rochester, 596 NYS 2d 968, affirmed unanimously, 197 AD 2d 919 (1993).) We hoped this letter would ease the way for our records request and help make the whole process run smoothly. Turns out we were naive...
- December 30, 2015: We formally submit our records request under the New York State FOIL. As usual, we use the website MuckRock to post and organize all our requests and official responses in real-time, visible to the public. Under NY FOIL, the City Clerk's Office is required to acknowledge our request within five days, and give some kind of yes or no response within twenty days. They do neither.
- January 14, 2016: We follow up by mail, asking for a response.
- January 29, 2016: We follow up by mail, asking for a response.
- February 10, 2016: At this point, not having heard any kind of response is legally considered the same thing as having the request be denied. Therefore, we lawyer up and formally file our appeal. The City Clerk's Office is supposed to respond to our appeal within ten days. (At this point, it's still all done through postal mail or e-mails, not a legal case.)
- February 15, 2016: We follow up by mail, asking for a response.
- February 23, 2016: The designated FOIL officer for the City Clerk's Office, who is also their attorney, finally responds to our attorney, on the day the clock runs out on the appeal timeline. He agrees to provide only some of the records, claiming (correctly) that marriage certificates less than fifty years old are protected under New York State law. We agree, but remind him that marriage indices are supposed to be open under New York State law, without a privacy barrier. He tells us he'll look into the legality of releasing the rest of the records, and will get back to us.
- Late February and early March 2016: We wait to see if he will follow through on telling us what kinds of records they have (what formats are they in? what years do they cover?) and what kinds will be available to us. Our attorney e-mails and calls the City Clerk's Office several times. They don't respond to her.
- March 16, 2016: By now, they've blown all their deadlines and we're done waiting. We file our lawsuit. We posted our Notice of Petition legal papers on our website; go have a look if you want to see what a state Freedom of Information lawsuit looks like, all twenty-five pages of it.
- March 21, 2016: We announce to the public in Newsletter #7 that we've formally filed our "Article 78" legal petition, and that the City Clerk's Office has been served with papers.
- April 7, 2016: They're due in court. They ask for a delay of a month to get their act together and figure out the answers to our questions about formats and years. We try to be nice, so we agree to the delay.
- May 9, 2016: They're due in court. They ask for another one month delay. We say okay again.
- June 6, 2016: They're due in court. They ask for another one month delay. We say okay again, but seriously you guys, this is the last time, get your act together already.
- July 5, 2016: They're less than twenty-four hours out from their court date, and suddenly they scramble to contact us and finally settle. We win! But we have to wait for the final stipulation papers to be signed for the settlement to take effect. We spend the next month wanting to shout the news from the rooftops, but unable to blab it publicly.
- August 4, 2016: They're less than twenty-four hours out from the day their thirty-day settlement deadline expires (are we noticing a theme here?) and they finally provide us a draft invoice for the microfilm copies, and the draft stipulation and settlement papers. We ask for two small additions to the settlement, one of them involving adding the shipping logistics in writing -- we told them we want the records sent to us by FedEx or UPS, something with a tracking number, and carrying appropriate insurance -- and the other being a question of whether they'll take credit cards or check only. But everything else looks good, we're due to have it signed within the next week, and our long-suffering attorney tells us we can finally publicize the news.
Here's what we've won
According to the terms of the draft settlement, our records are legally supposed to be arriving in the mail in the coming days. We haven't actually seen them yet, but we've been told that the records are arranged like this:
- We're getting 110 reels of microfilm, covering just the 1930-1972 years of the marriage index. We have been informed that these microfilms are "identical in format" to the 1908-1929 marriage index microfilms we won from DORIS last year, which we put online for free public use back in April. Assuming that's correct, these microfilms should also be broken down by borough (county), then arranged by year, then (sometimes) by section of the year, then alphabetically by first the two letters of the surname, and then finally separated out by brides and grooms.
Under NY FOIL, people who win public records requests can get the copies for the actual costs of reproducing the records; the agencies can't add any kind of mark-up to the price. That means that we just got millions of records for only $37 per microfilm roll plus one CSV file, which is a pretty great deal compared to the typical costs of most other record acquisition projects.
- We're also getting one text-searchable database, covering just the 1950-1995 years of the marriage index. This database was created for in-house use by the City Clerk's Office. This is great news, because it means for these forty-five years of data there won't need to be a new transcription project, since this is already transcribed and searchable! Of course, we won't know how well they transcribed the handwriting on the files until we actually get to take a look at it, which we haven't yet.
We will be receiving this data as a CSV file, which is kind of like a very basic spreadsheet, on a USB thumbdrive. As far as we know, the columns of available data will be the same sort of basic data available in the microfilm images: surname, given name, year, borough, etc. However, two columns of data will be redacted from this database before they give it to us: the spouses' dates of birth. (More on that below.)
We'll post this CSV file online for people to download and search through. But because this will have so many rows of data (probably millions!), it may crash traditional spreadsheet programs like Microsoft Excel, so we realistically may need to wait for an online service to add it to their website, or for some group make a searchable front-end for the data.
Yes, of course everyone's favorite for-profit and non-profit genealogy websites are welcome to integrate this new database into their online offerings -- the data is in the public domain! But whether they will do so or not is entirely up to them.
- This also means that there is some overlap between the microfilmed content and the text-searchable database content, so just for the 1950-1972 years of the marriage index, genealogists will be able to search for records in either format, scrolling through microfilmed images online and/or searching through the text database. (We assume most people would prefer to use the database format.)
Here's what we don't know
We're not sure whether the dates listed on these records will be the date that the couple originally applied for the license -- usually a few days or weeks before the wedding took place -- or if the date will be that of the actual marriage. Maybe the microfilms will have the application date, just as they did for the earlier 1908-1929 records, while the text database will have the actual marriage date. Maybe we'll get both dates. We really don't know. Guess we'll find out!
And we don't know yet whether or not we're going to get attorney's fees and court fees paid. The draft settlement has explicitly allowed for it, but we still have to submit our hours and get a sign-off from the higher-ups in the City. The New York State FOIL allows for, but does not mandate, the awarding of attorneys fees to records requestors who "substantially prevail" and win their records from an improperly withholding government agency. But it's not guaranteed like how it is in some other states, like New Jersey, Florida, California, or a few others. Getting reimbursement is usually a crapshoot in New York, even if you do win the records.
But because the New York City Clerk's Office screwed up so badly in handling our perfectly legitimate and properly submitted records request and its subsequent lawsuit, as part of this settlement New York City may be paying for all our court and attorney fees. We won't know for sure until thirty days after the stipulation is signed.
Here's what we didn't win
To finally win these records in the settlement without enduring further legal hassles, we did have to compromise a little bit:
- Compromising (temporarily) on the date range: We had originally asked the City Clerk's Office for a copy of the marriage index for 1930-2015. But we only got it for 1930-1995. Why? Well, it turns out that starting around 1996, there isn't any separate marriage index for these records! New York City switched to a "born digital" marriage database, with the information for all marriage license applicants being entered directly into the computer systems right at the City Clerk's Office window.
Luckily, the information in that 1996-present marriage database should still be legally available to the public under New York FOIL, as long as any overly-personal information gets redacted from the database first. But in order to get a copy of it, we'll need to make a new and separate FOIL request, with more narrowly-tailored language, because we'll now be asking for a partial database dump, rather than trying to cover these recent years of data with our "index" request.
So, in order to complete this data set, Reclaim The Records will be making a new records request for the 1996-2015 (or perhaps 1996-2016) New York City marriage index later this year or early next year. Hopefully, now that the City Clerk's Office knows us, and knows that we're willing to take them to court if need be, and knows that they really are subject to the requirements of the New York State FOIL, they won't fall down on the job so badly when they receive this new request.
We are committed to getting this post-1996 marriage index data, too. For one thing, it just so happens that the founder of Reclaim The Records got married in New York in 2003, and she'd love a chance to legally liberate her own genealogical record, instead of just her ancestors' records! For another thing, a 1996-2015 database would include all the same-sex couples whose marriages started to be recorded in New York City starting in June 2011, so this would (we think) be the first-ever genealogical marriage records data set open to the public on any website to finally include same-sex marriages. That definitely seems like a worthwhile thing to contribute to the world.
So, we're definitely going to file that follow-up records request, once we're done processing all the details from this main request. We'll let you know how it goes.
- Compromising on the date of birth or year of birth: Because New York City is very large and a lot of people living there have the same names, we had originally asked the City Clerk's Office to include the date of birth for each of the spouses in the marriage index. And indeed, that 1950-1995 in-house database they have does have that information available as database columns for the brides and grooms, although the older microfilms do not. But the City Clerk's Office told us that birthdate data was too intrusive, we couldn't get that. So we asked, could we perhaps just get the year of birth instead, dropping the month and day? Or even just get the couple's age at marriage?
Now, we wouldn't have asked for this data at all, except that we know from our research discussions with the awesome and super-helpful New York Committee on Open Government (COOG) that the years of birth of the spouses in a marriage index is still a legally-grey unsettled area under New York's FOIL. It has never been outlawed by the courts, like how providing street addresses was ruled to be intrusive, but it hasn't been granted by them either, like how the names of the parties to the marriage are public. And the couple's ages or years of birth are commonly included in lots of other states' marriage indices, and they're not generally thought of as a big deal. So we figured, why not try to push it and ask for the data here?
But the New York City Clerk's Office was adamant that nope, nuh-uh, sorry, we could not get any kind of birthdate data, not even just the year or just the age. So we sighed and we decided not to fight them on this issue, which could potentially have gotten us tied up in court for months. They will be redacting the birthdate columns from the 1950-1995 database before they give it to us. If we have to compromise a little, this seems okay to us.
But we're mentioning all this background here, just for the record, in case anyone reading this newsletter wants to try obtaining this piece of data through their own Freedom of Information Law records request someday.
Here's what's next
The 110 microfilms will be mailed from the New York City Clerk's Office to us in California. The City's attorney has told our attorney that the microfilm copying has already begun, even before we formally sign the settlement.
Once the microfilms arrive, we'll turn around and send them to the generous folks at FamilySearch in Salt Lake City, where they will all be digitally scanned for free on their professional-grade equipment. (Thank you, FamilySearch! And the next time you see them at a genealogy conference, will you please make sure to say thank you to them on our behalf?) When they're done, the box of microfilms will be sent back to us, along with a portable hard drive holding all the newly-digitized images.
Then we'll upload those images to the Internet Archive (archive.org) for free public usage, just like how we handled the earlier 1908-1929 marriage records, and how we're about to release our New Jersey records. Because we'll be uploading so much data, we'll probably do at least some of this work onsite at their headquarters in San Francisco, where they have a ridiculously fast Internet connection. This project is going to be a lot of data to wrangle, with probably over 200,000 images to upload, so this may take a while, perhaps until the end of the year to get it all done.
The CSV file of the 1950-1995 searchable text database will also be uploaded to the Internet Archive. We'll also start working with the Roots-Dev team of genealogy nerds (we say that with love, since our founder's a member) to see if we can get some kind of standalone searchable version of that CSV data thrown together online.
And once everything's done, we won't really need to have the original 110 microfilms sitting around forever, so we'll eventually be donating the films to the New York State Library in Albany -- which, incidentally, is where we recently sent the box of the 1908-1929 microfilms we won from last year's lawsuit. (But those of you who follow Reclaim the Records on Facebook already knew about our donation.) This should happen by early 2017.
Meanwhile, we'll also be managing several other pending public records requests with several other agencies in at least two states, and more to be announced soon. Can't stop, won't stop.
Thank you to our awesome attorneys at Rankin and Taylor in New York for once again taking our public records case and seeing it through to a successful end. If you ever need to take on a recalcitrant or incompetent New York government agency, you should call them, they're good at this sort of thing.
Thank you to MuckRock, for being the best way to research, submit, and organize Freedom of Information records requests, at both the state and federal level. We learn new tips and tricks from this site all the time. And they recently became a real 501(c)3 non-profit, which makes them extra-awesome.
Thank you to FamilySearch, who will once again be generously donating their time and equipment to scan the microfilms we've won so that we can make the images available to everyone. (And before you ask, we don't know if they will put the digital images or the text database on their own website too, but they are certainly welcome to do so, and so can everyone else.)
Thank you to the Internet Archive for offering free web hosting and free bandwidth to our data. They too are a non-profit organization, so if you enjoy using the images or data we upload, we hope you'll consider donating to them.
Thank you to all our genealogical and open data friends and supporters around the world.
- The New Jersey indices are almost done and almost ready to be announced. Thanks for being patient while we get all those images uploaded, along with finalizing the instructions on how to order the actual certificates.
- Reclaim The Records will be presenting a talk at the annual IAJGS conference in Seattle on Monday, August 8, 2016. That's this Monday! Come say hi!
- Congratulations to Hawaii adoptees and birth parents on the passage of a new law giving you open access to your adoption records!
- Were you (or someone you know) politically active in New York City in the 1950's, 1960's, or early 1970's? If so, the same New York FOI law that helped us win access to the marriage index will let you win access to newly rediscovered police surveillance files.
- The long-awaited FOIA reform bill finally passed the House and the Senate, and was just signed by President Obama. Reclaim The Records was proud to be a signatory to a Project on Government Oversight letter urging its passage.
- Hey, did you hear about that awful story a few months ago where a different New York government agency tried to retroactively classify several decades worth of genealogical records, even though they'd all been public for many years, and people from the agency literally came in took the books off the shelves of a public library in the night? Yeah, so did we. Ugh, so gross.
Well, when we heard the news, we were appalled. If only there were an activist group out there, one with a history of taking on government agencies for denying access to genealogical records, who could do something about that situation... Oh, wait.
So we talked to our attorneys, and we talked to many genealogists, and we talked to the adoptee rights community, and we talked to a whole lot of people involved with Freedom of Information law, not just in New York but in other states as well. And we're going to be doing something about all that. We want those records back.
Stay tuned. 😉