Ability to load a parms file directly into RC2

Currently Parms files are loaded into FC2 from FC1 after FC1 has been updated. They are incorporated into FC2 through the overall FC2 monthly update.
We need to be able to a)load a parms file directly into FC2 and b) this needs to be independent of the overall FC2 update.
This is an urgent requirement for loading 1901 and 1911 Params files and to allow for checking of CVS uploads of 1901 and 1911 records.

Pushed the changes to origin

For testing, @Vino, does this need to be uploading unmodified PARMS files which were created by @richardofsussex ?

@Vino-S Does the upload process the file to create the individual pieces?
Is it possible to upload a csv version of the file?
If the latter is there a conversion of the csv parms file to the pieces.
We need the csv version to overcome the length limitation on Civil District  and Enumeration district names.
We should move the upload into the Manage County set of actions and have it as a county coordinator capability.
Can you base further code modification on cen_820

Hi @Captainkirkdawson

The upload does not process the parm file. It adds the file to directory fixed. I will work on adding the Process Action
The Upload supports csv and dat files
We need the csv version to overcome the length limitation on Civil District and Enumeration district names. will work on it
We should move the upload into the Manage County set of actions and have it as a county coordinator capability. Agreed. will do it

Thanks @Vino-S
PS please build on cen_820 it has your code merged in

@Vino-S before you go too far can I draw your attention to the freecen_parms collection. Cen2 has controller, models and views for the collection written years ago by Doug for uploading csv parms files and their processing.
Stumbled across it while looking for something else.

Ah yes Kirk.

_We need the csv version to overcome the length limitation on Civil District and Enumeration district name_s. @richardofsussex already has some

Hi @Captainkirkdawson ,
I have made the following changes in the code. I have kept Doug's code as it is and used it as reference.

I have moved the Manage Parms Action  to the  Manage County action

I have changed the Destination for uploaded parm files to MyopicVicar/parms_files

At Present Manage Parms action takes us to index page which will show list of files uploaded through FC2

I have created an action to Add new Parm File to upload a Parm file.
Upload process does the following validations:
a) validates file type to be csv/dat
b) Validates file name
c) validates file header
d) validates file content

Removed length limitations on Civil District and Enumeration district names

Also added Process and Remove Action for each file on index

There was a recent discussion about 'bundling' the lower-level place concepts. Looking at the 1901 census data from TNA, that might be the only option. http://discovery.nationalarchives.gov.uk/results/r?_q=RG+13%2F+1892 shows that places are run together without any punctuation. Comments/advice please.

Hi Richard,
Yes we have always had that issue when we do a new Parms we did the following.
We were lucky as we just click on the ’ Purple Registration’ and it takes us to the next page where we then find the list of Places.
Registration Sub-District: Wiggenhall
Civil Parish, Township or Place:
Barroway Drove
Holme next Runcton
North Delph (part)
Porter's Fen Corner
Saddle Bow
Shouldham
Shouldham Thorpe
South Runcton
Stow Bardolph (part)
Stowbridge
Tottenhill
Wallington with Thorpland
Watlington
Wiggenhall St German
Wiggenhall St Mary Magdalen
Wiggenhall St Mary the Virgin
Wiggenhall St Peter
Wimbotsham
Wormegay
From: Richard Light<EMAIL_ADDRESS>Sent: 30 April 2020 13:25
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
There was a recent discussion about 'bundling' the lower-level place concepts. Looking at the 1901 census data from TNA, that might be the only option. http://discovery.nationalarchives.gov.uk/results/r?_q=RG+13%2F+1892 shows that places are run together without any punctuation. Comments/advice please.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-621800226 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIL3VSHN3ONE55PL3JQ5MDRPFUYFANCNFSM4LUK4O7Q .

Brenda,
When I access the same record via TNA's search API I get this:
For comparison, the 1881 census has a 'heading' for each place:
Not sure how to proceed: it needs to be done programmatically.  (In 1911
the number of records explodes from around 5,000 to 35,000!)
Richard
On 30/04/2020 14:07, Brenda wrote:

Hi Richard,
Yes we have always had that issue when we do a new Parms we did the
following.
We were lucky as we just click on the ’ Purple Registration’ and it
takes us to the next page where we then find the list of Places.
Registration Sub-District: Wiggenhall
Civil Parish, Township or Place:
Barroway Drove
Holme next Runcton
North Delph (part)
Porter's Fen Corner
Saddle Bow
Shouldham
Shouldham Thorpe
South Runcton
Stow Bardolph (part)
Stowbridge
Tottenhill
Wallington with Thorpland
Watlington
Wiggenhall St German
Wiggenhall St Mary Magdalen
Wiggenhall St Mary the Virgin
Wiggenhall St Peter
Wimbotsham
Wormegay
From: Richard Light<EMAIL_ADDRESS>Sent: 30 April 2020 13:25
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file
directly into RC2 (#833)
There was a recent discussion about 'bundling' the lower-level place
concepts. Looking at the 1901 census data from TNA, that might be the
only option.
http://discovery.nationalarchives.gov.uk/results/r?_q=RG+13%2F+1892
shows that places are run together without any punctuation.
Comments/advice please.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-621800226
, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADIL3VSHN3ONE55PL3JQ5MDRPFUYFANCNFSM4LUK4O7Q
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-621823235,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAQYM3SBEMYLNXBPJZNVZZTRPFZ2TANCNFSM4LUK4O7Q.

--
Richard Light
<EMAIL_ADDRESS>/@richardofsussex/

Sorry: pasting partial screenshots into my email clearly didn't work.  What I get in the XML from the API for 1901 is: <Description>Registration Sub-District: Wiggenhall Civil Parish, Township or Place: Barroway Drove Holme next Runcton North Delph (part) Porter's Fen Corner Saddle Bow Shouldham Shouldham Thorpe South Runcton Stow Bardolph (part) Stowbridge Tottenhill Wallington with Thorpland Watlington Wiggenhall St German Wiggenhall St Mary Magdalen Wiggenhall St Mary the Virgin Wiggenhall St Peter Wimbotsham Wormegay.</Description>
whereas what I got for earlier years is e.g.:
`      Registration Sub-District 1B Wiggenhall. Civil Parish, Township or Place: Wiggenhall: St Peter. Civil Parish, Township or Place: Wiggenhall: St Mary Magdalen. Civil Parish, Township or Place: Wimbotsham (part) (Divided between RG 11/2003-2004). Civil Parish, Township or Place: Shouldham Thorpe. Civil Parish, Township or Place: Shouldham. Civil Parish, Township or Place: Stow Bardolph (part) (Divided between RG 11/2003-2004).
I was able to use the 'headings' within the text to split up the places sensibly: that is no longer possible.
Richard
`

@richardofsussex we do have a complete csv set for Somerset for 1901. Now I don't know if that was based on extract you made or where it came from. Perhaps @FreecenBren or @geoffj-FUG  can comment

Looking at the file in my hex editor - they are just spaces. Whatever separator was present in the source data has been stripped or converted to space in the generation of the XML. I'll see if TNA are willing/able to help; for example the TSV format they produced before may be more useful.
I have no recall (or record) of working with 1901 before. All my files from the 2017 work are still sitting around.

@Vino-S as noted on slack I received a crash on the initial upload test on my server

@richardofsussex a is the Registration District and b is a civil parish ie the name after the second colon. In years up to 1891 between the colons was Civil Parish, Township or Place ; in 1901 and on it has a name before the Civil Parish, Township or Place but we want what follows

To take an example mentioned in Brenda's "parms example" spreadsheet:
<Description>Registration Sub-District: Tuxford Civil Parish. Township or Place: Askham Bevercotes Bothamsall Church Laneham Darlton Dunham East Drayton East Markham Fledborough Haughton High and Low Marnham Laneham Markham Clinton Marnham Milton Normanton on Trent Ragnall Stokeham Tuxford West Drayton.</Description>
'Civil Parish' looks as though it is part of the Sub-District name. Am I correct in thinking that there ought to be a full stop (or something) after 'Tuxford', and that the full stop after 'Civil Parish' ought to be a comma?  (At least they have done this consistently!)
Of the long list of names that follows, Askham, Beverco[a]tes, Laneham, Darlton, Dunham, etc. are all parishes, but some (e.g. Bothamstall Church) are not. I can see no way of determining which to include, nor indeed of deciding which groups of words constitute a valid place name. The only idea I have is to look them up against an authoritative database of parishes (or the parishes from an earlier census, though I know they tend to change).

@richardofsussex it is important that we get that full list of names even though as you say they be things other than what is rigorously termed a civil parish. The census document for the piece is separated by those units. I need to be able to validate what the transcriber has entered against those units. Hence we must keep what  in there unadulterated.
There is a need to be able within the application to group and systematize these units into meaningful (to the researcher) groups but that is a different story and something @PatReynolds has been trying to get her head around not just for CEN but for all our apps

OK, I'll see what I can do. Presumably 1901 and 1911 are higher priority than the other years, in that we (mostly) don't have any PARMS files at all for them?

I think so; SOM 1901 is the only one that I know we have but from whence it came is unknown to me

Could you please send it to me, so I can use it as a control/reality check<EMAIL_ADDRESS>

Thanks. TNA have the first six pieces as Williton: http://discovery.nationalarchives.gov.uk/browse/r/h/C149922, The PARMS file has them - essentially - with their Sub-Districts as the 'a' record. Which is correct?

I will try to simplify it from my perspective . Not sure if this will help or not.
Background first.

At this precise  time  that could be from a small Piece of say  3 EDs up to the biggest I have so far is for Devon 1851 census  of 58 Ed’s.
That can be potential mean 58 Civil Parish Places
Each ED has a number and a Civil Parish PLACES name that is on the top left hand corner of every census image page.
You can get more than one ED for the same Civil Parish  Place in any one Piece.
These Civil Parish Places  are what Richard was mentioning and we use them on a transcription in column A. FC1
The other Places that he is not sure about are the ‘Hamlets’
They are still Places that people in the Census say they were born.
So we do get these entered in the POB  column, but not against a Civil Parish Place  name.
When the PARMS were done in the first instance in 1998 we only enter the Civil Parish name as the ‘b’  number as they not only match those in the NA  for the PIECE number but match the Civil Parish Place  name that the transcriber picks up from the ED and the Census Image. We also use the Registration name as the ‘a’ number.
Summary.

So we have the ED/Civil Parish PLACE name.
We also then have the ‘Hamlet Place Names’  that are also in the NA list that Richard mentions.
Then we have the Registration District Place Names. Those are on the NA link but often are repeated again as number 1 above.
Then we have the POB place names that the Enumerator is given from the ‘Populous’ that he is Enumerating.
On top of all those Places, we also have Places from around the world that we also enter in the POB.

Not being a Programmer I would not know where to start.
I only know what we have and what we enter in each Piece we transcribe.
I have attached an example for one PIECE number re Places
Devon 1851 HO107/1891 or as we will upload it as HO511891
1, National Archives list of Places
2. How the  ED’s and Civil Parish Place will show in the  VLD when it get uploaded.
3. The Parms from FreeCEN 2 for the Places for this PIECE number.
Plenty of places to pick up Places!!!!  Which one will we go for?
I am sorry I do not have the answer.
Pat has asked that I look at something re Places. Time not an option this week.
I will try to look at it next week.
You are welcome to ‘bin this’ if not relevant to the query.
Brenda
On Thu, 30 Apr 2020 at 18:06, Richard Light<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:
To take an example mentioned in Brenda's "parms example" spreadsheet:
Registration Sub-District: Tuxford Civil Parish. Township or Place: Askham Bevercotes Bothamsall Church Laneham Darlton Dunham East Drayton East Markham Fledborough Haughton High and Low Marnham Laneham Markham Clinton Marnham Milton Normanton on Trent Ragnall Stokeham Tuxford West Drayton.
'Civil Parish' looks as though it is part of the Sub-District name. Am I correct in thinking that there ought to be a full stop (or something) after 'Tuxford', and that the full stop after 'Civil Parish' ought to be a comma? (At least they have done this consistently!)
Of the long list of names that follows, Askham, Beverco[a]tes, Laneham, Darlton, Dunham, etc. are all parishes, but some (e.g. Bothamstall Church) are not. I can see no way of determining which to include, nor indeed of deciding which groups of words constitute a valid place name. The only idea I have is to look them up against an authoritative database of parishes (or the parishes from an earlier census, though I know they tend to change).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-621982596 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIL3VUXGZZG2SNNPIS4M3LRPGVXXANCNFSM4LUK4O7Q .  https://ci6.googleusercontent.com/proxy/aTuyLPCaYZlb08qZxTBZik2zLw4mrU8U1TsnLJGOwMQRVIz5euXUPG9i4il6RosSi6dZW64_22Dh3UAu6nsurqzmdPcCZlGdpJWWLi0j0U5KWOyrIg79_2Kb8xYR5XaCdkS7A23UFrtmHPJPs7B5lpKivI4tLxJVlqYaVkyMdHPSvCEM8xAu-WaSlcHsQj6Zz-2ypNvkOgtjFDaPSgAOLvj

To add to that in 1911 there is a one to one relationship between the piece number and the ED number.
Geoff
From: Brenda<EMAIL_ADDRESS>Sent: Friday, 1 May 2020 6:23 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
I will try to simplify it from my perspective . Not sure if this will help or not.
Background first.

At this precise time that could be from a small Piece of say 3 EDs up to the biggest I have so far is for Devon 1851 census of 58 Ed’s.
That can be potential mean 58 Civil Parish Places
Each ED has a number and a Civil Parish PLACES name that is on the top left hand corner of every census image page.
You can get more than one ED for the same Civil Parish Place in any one Piece.
These Civil Parish Places are what Richard was mentioning and we use them on a transcription in column A. FC1
The other Places that he is not sure about are the ‘Hamlets’
They are still Places that people in the Census say they were born.
So we do get these entered in the POB column, but not against a Civil Parish Place name.
When the PARMS were done in the first instance in 1998 we only enter the Civil Parish name as the ‘b’ number as they not only match those in the NA for the PIECE number but match the Civil Parish Place name that the transcriber picks up from the ED and the Census Image. We also use the Registration name as the ‘a’ number.
Summary.

So we have the ED/Civil Parish PLACE name.
We also then have the ‘Hamlet Place Names’ that are also in the NA list that Richard mentions.
Then we have the Registration District Place Names. Those are on the NA link but often are repeated again as number 1 above.
Then we have the POB place names that the Enumerator is given from the ‘Populous’ that he is Enumerating.
On top of all those Places, we also have Places from around the world that we also enter in the POB.

Not being a Programmer I would not know where to start.
I only know what we have and what we enter in each Piece we transcribe.
I have attached an example for one PIECE number re Places
Devon 1851 HO107/1891 or as we will upload it as HO511891
1, National Archives list of Places
2. How the ED’s and Civil Parish Place will show in the VLD when it get uploaded.
3. The Parms from FreeCEN 2 for the Places for this PIECE number.
Plenty of places to pick up Places!!!! Which one will we go for?
I am sorry I do not have the answer.
Pat has asked that I look at something re Places. Time not an option this week.
I will try to look at it next week.
You are welcome to ‘bin this’ if not relevant to the query.
Brenda
On Thu, 30 Apr 2020 at 18:06, Richard Light<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:
To take an example mentioned in Brenda's "parms example" spreadsheet:
Registration Sub-District: Tuxford Civil Parish. Township or Place: Askham Bevercotes Bothamsall Church Laneham Darlton Dunham East Drayton East Markham Fledborough Haughton High and Low Marnham Laneham Markham Clinton Marnham Milton Normanton on Trent Ragnall Stokeham Tuxford West Drayton.
'Civil Parish' looks as though it is part of the Sub-District name. Am I correct in thinking that there ought to be a full stop (or something) after 'Tuxford', and that the full stop after 'Civil Parish' ought to be a comma? (At least they have done this consistently!)
Of the long list of names that follows, Askham, Beverco[a]tes, Laneham, Darlton, Dunham, etc. are all parishes, but some (e.g. Bothamstall Church) are not. I can see no way of determining which to include, nor indeed of deciding which groups of words constitute a valid place name. The only idea I have is to look them up against an authoritative database of parishes (or the parishes from an earlier census, though I know they tend to change).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-621982596 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIL3VUXGZZG2SNNPIS4M3LRPGVXXANCNFSM4LUK4O7Q . https://ci6.googleusercontent.com/proxy/aTuyLPCaYZlb08qZxTBZik2zLw4mrU8U1TsnLJGOwMQRVIz5euXUPG9i4il6RosSi6dZW64_22Dh3UAu6nsurqzmdPcCZlGdpJWWLi0j0U5KWOyrIg79_2Kb8xYR5XaCdkS7A23UFrtmHPJPs7B5lpKivI4tLxJVlqYaVkyMdHPSvCEM8xAu-WaSlcHsQj6Zz-2ypNvkOgtjFDaPSgAOLvj
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622089128 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFKFM6L6D442PQE2Y3TRPHMZZANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFJADVKMWSC7EEFPAA3RPHMZZA5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUKFHKA.gif

Richard
I did the Somerset 1901 PARMS manually from the NA records. I copied and pasted from each piece.
I was taught when I started in 2004 that the sub-district went into the PARMS so that is what I have always done. So all the Somerset pieces will be the same.
That suggests that I either misunderstood Terry or all his are the same as mine.
Was I wrong?
Geoff
From: Richard Light<EMAIL_ADDRESS>Sent: Friday, 1 May 2020 5:59 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
Thanks. TNA have the first six pieces as Williton: http://discovery.nationalarchives.gov.uk/browse/r/h/C149922, The PARMS file has them - essentially - with their Sub-Districts as the 'a' record. Which is correct?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622076272 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFLSCFKYEEIZK3FPKO3RPHKBHANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFOEZLCBVH3PJZT2V6TRPHKBHA5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUKCC4A.gif

Thanks, Geoff, that's a useful insight. We'll need to agree what the 'rules of engagement' are before I can go ahead and try to churn out PARMS for everything. Sub-Districts appear to have been around since 1851. Are 'hundreds' the equivalent concept for 1841?
As a random test, I searched for Lights in Somerset in 1891. The hit I looked at (https://www.freecen.org.uk/search_records/5db72b6bf4040b9e96f3f1aa/ada-m-light-1891-somerset-bedminster-1870-?locale=en) was for Place Bedminster, Parish Dundry. Your 1901 PARMS file has Dundry within Long Ashton, not Bedminster; my PARMS file doesn't have Dundry at all.

Richard et al
As I said previously I was advised to use the sub-Districts for my PARMS when I first started in 2004. For my first couple I hunted through the images to find the first image in each ED and transcribed the PARMS form there.
I have continued to use the sub-Districts since. Nobody ever said that what I did was incorrect.
However, I have been thinking about the big picture.
It is intended, I understand, to link the searches of FreeBMD, FreeREG and FreeCEN at some time in the future so that when a person is being researched all 3 programmes contribute to the report. FreeUKGEN will need some way of linking the searches. It strikes me that the Registration District may be a key to limiting the searches and therefore the rules in each project should be the same.
The rules for FreeBMD can be found at https://www.freebmd.org.uk/DistrictInfo.html. They use the Registration District and Civil Parish.
So it makes sense to me that FreeCEN should use the same Registration District and Civil Parish in their PARMS. That gives both projects the same data filters.
If I need to amend the Somerset PARMS or FC2 I will do so.
We also have the issue of entries such as Bitton (GLS) in the PARMS (Bitton is out of County SOM) and the National Archives having terms such as (part) in their Civil Parishes. These will create errors in the new FreeCEN front end (CSVProc) test reports as they do not align with the Civil Parish in the censuses. The extraneous information will need to be edited out, either on creation of the PARMS or when the piece is uploaded to CSVProc.
Those are my thoughts.
Also the use of District or Sub-District is important to me as I am currently writing the 1911 PARMS for Somerset so that we can test 1911 pieces in CSVProc. I know it is a redundant column at the moment but I would like to be testing with the right file and not have to amend it later. It is going to be huge!
Geoff
From: Richard Light<EMAIL_ADDRESS>Sent: Friday, May 1, 2020 2:15 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
Going back to first principles here. Could someone please remind me what counts as an 'a' record and what is a 'b' record in a PARMS file? Looking at the document Brenda prepared back in 2017, I conclude that we DON'T include Registration Sub-Districts or places smaller than a Parish. Is that correct?
—
You are receiving this because you were mentioned.
Reply to this email directly,  https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-621954025 view it on GitHub, or  https://github.com/notifications/unsubscribe-auth/AKCPIFLYG2EXWSX3GOPSNRDRPGPYJANCNFSM4LUK4O7Q unsubscribe.  https://github.com/notifications/beacon/AKCPIFOCSAPJPRM6ERJRBKDRPGPYJA5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUJEH2I.gif

Richard
FreeBMD already has a comprehensive database. Is that in fact the National Archives data?
Geoff
From: Richard Light<EMAIL_ADDRESS>Sent: Friday, May 1, 2020 3:06 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
To take an example mentioned in Brenda's "parms example" spreadsheet:
Registration Sub-District: Tuxford Civil Parish. Township or Place: Askham Bevercotes Bothamsall Church Laneham Darlton Dunham East Drayton East Markham Fledborough Haughton High and Low Marnham Laneham Markham Clinton Marnham Milton Normanton on Trent Ragnall Stokeham Tuxford West Drayton.
'Civil Parish' looks as though it is part of the Sub-District name. Am I correct in thinking that there ought to be a full stop (or something) after 'Tuxford', and that the full stop after 'Civil Parish' ought to be a comma? (At least they have done this consistently!)
Of the long list of names that follows, Askham, Beverco[a]tes, Laneham, Darlton, Dunham, etc. are all parishes, but some (e.g. Bothamstall Church) are not. I can see no way of determining which to include, nor indeed of deciding which groups of words constitute a valid place name. The only idea I have is to look them up against an authoritative database of parishes (or the parishes from an earlier census, though I know they tend to change).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-621982596 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFL4YXF65U7XNV3M4JDRPGVX5ANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFO6EAEJDOX7ILP72VTRPGVX5A5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUJLHBA.gif

Richard
Somerset is the only county working on 1911 and that is currently just for testing purposes.
Somerset is transcribing 1901 but cant process the transcriptions on the old system so waiting for the new one. Nottingham, I understand, is not far away.
However I can look after Somerset. I will just need to amend my existing 1901 file and I am in the process of creating 1911.
I would put a higher priority on the missing PARMS in the 19th Century.
I believe that is what Brenda needs.
Geoff
From: Richard Light<EMAIL_ADDRESS>Sent: Friday, May 1, 2020 3:51 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
OK, I'll see what I can do. Presumably 1901 and 1911 are higher priority than the other years, in that we (mostly) don't have any PARMS files at all for them?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622006006 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFNAKAJUUFET3WXZACLRPG3AJANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFLISZ2XLMQ73CRJ2V3RPG3AJA5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUJQ55Q.gif

Geoff,
I'm actually quite keen to have a go at 1911. The fact that each record has a simpler structure means that I can probably do a better job of auto-generating PARMS files.  I'll see what I can do for Somerset, and will await your comments with interest!

Richard
This is what I have so far
Geoff
From: Richard Light<EMAIL_ADDRESS>Sent: Friday, May 1, 2020 9:22 PM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
Geoff,
I'm actually quite keen to have a go at 1911. The fact that each record has a simpler structure means that I can probably do a better job of auto-generating PARMS files. I'll see what I can do for Somerset, and will await your comments with interest!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622350368 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFKWICYDLFKF4Z3F3ILRPKWEDANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFLJ3D3C2KKPK3KSX3TRPKWEDA5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUMFAIA.gif

Geoff,
And this is what I have generated from TNA data. I'm sure there are
errors/inconsistencies, but there is nothing that jumps out at me as
being obviously wrong.  (The error I'm expecting is more than one place
name lumped together in a single 'b' cell.)  What do you think of it?
Vino: will it load?
Richard
On 01/05/2020 12:34<EMAIL_ADDRESS>wrote:

Richard

This is what I have so far

Geoff

From: Richard Light<EMAIL_ADDRESS>Sent: Friday, May 1, 2020 9:22 PM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention
<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms
file directly into RC2 (#833)

Geoff,
I'm actually quite keen to have a go at 1911. The fact that each
record has a simpler structure means that I can probably do a better
job of auto-generating PARMS files. I'll see what I can do for
Somerset, and will await your comments with interest!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622350368,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKCPIFKWICYDLFKF4Z3F3ILRPKWEDANCNFSM4LUK4O7Q.

--
Richard Light
<EMAIL_ADDRESS>/@richardofsussex/

SOM1911.zip

SOM1911.zip
Hi,
I have now written a checking procedure which uses the 1891 place names (as I have them) as an authority. It identifies each level-b place which matches a place in that file (doing a longest-first match) and creates separate entries in the spreadsheet for each place. Where it doesn't match an entry in the 1891 file, it puts question marks round it.
There are some 'funnies' in the result, but as far as I can tell these are all caused by funnies in the 1891 'authority'. For example, 'Minehead Without' doesn't appear in that file, but 'Without' on its own does, so we end up with two entries 'Minehead' and 'Without'.  By improving the authority file (or using a better one) we can get better results.
My thinking is that we can generate a CSV file in this format for 1911 for each county. All the county coordinator would have to do is to go through the CSV looking for question marks, and update the parish names to their satisfaction.
Does this look like an approach that will work?
Richard

Having looked at the commentary that has taken place while I was in the land of Nod. I have to say I am now totally and utterly confused. Our terminology wanders all over the place; as does that of the TNA. We started off talking about Registration Districts  which it now appears we do not record in a piece. There are Registration Sub Districts which we record (usually?) as the a in our parms files that we appear to call District in CEN2 and Place in Cen1. How this field could be called either District or Place is beyond me. Then we record things that could be just about anything as b in our parms files. Cen1 dodges what they are are lists them under a heading of Comprising. Cen2 calls them Subplaces. Our csv has an empty field after the piece number; I wonder what that was for?
Utter confusion. I hope @richardofsussex can make sense of this because I cannot

@richardofsussex Your extract of 1911 should not be uploaded directly; it would benefit from a manual edit as some subplaces are clearly combined because you have no ability to detect the separation. But it would not be a large task. Well done; Would love to see one for 1901

Kirk,
Please see my latest contribution; while it definitely still needs manual intervention, this is now purely to check queries. I've spent the best part of the day implementing a 'look-up' which appears to be reasonably successful in splitting apart the subplaces which TNA have cheerfully clumped together for us from 1901 onwards.
Turning to your previous point, I think the key point is to consider the purpose of these PARMS files, and to ensure that the data we submit as PARMS is fit for that purpose. As I understand it, the purpose of the 'b' places is to provide an authority against which data in individual records is validated. I'm not sure of the purpose of the 'a' places, or whether it particularly matters whether they are Districts or Sub-Districts. I would welcome @FreecenBren 's advice on these points.
In converting directly from TNA records, we are clearly constrained by their recording practice, which has changed over the period 1841-1911.

@richardofsussex unfortunately programmers as you know need precision and definition; without it chaos reigns supreme.
For both @geoffj-FUG and @FreecenBren the purpose of this new set of extracts is for CEN2 ingestion not CEN1. So that fields reflect the TNA without a 20 character constraint
The Purpose of the b field for checking is one specified by me in processing a csv data file. The b field may well have other uses; in  CEN1 you can search by it.
The a field is used as the search district
I would like to know if we can also record the Registration District in the parms in the second field as that does not appear to be used for anything????????????

Richard
I have compared the start of the 2 files (yours and mine).
The downloaded version has pieces missing. Look at 14121-14122, 14125, 14137-40.
Otherwise they look the same.
Does the fact that the program creates multiple ‘a’ rows for the same piece cause upload problems?
Geoff
From: Richard Light<EMAIL_ADDRESS>Sent: Friday, May 1, 2020 10:06 PM
To<EMAIL_ADDRESS>'FreeUKGen/FreeCENMigration'<EMAIL_ADDRESS>'FreeUKGen/FreeCENMigration'<EMAIL_ADDRESS>Cc: 'Mention'<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
Geoff,
And this is what I have generated from TNA data. I'm sure there are errors/inconsistencies, but there is nothing that jumps out at me as being obviously wrong.  (The error I'm expecting is more than one place name lumped together in a single 'b' cell.)  What do you think of it?  Vino: will it load?
Richard
On 01/05/2020 12:34<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:
Richard
This is what I have so far
Geoff
From: Richard Light<EMAIL_ADDRESS><EMAIL_ADDRESS>Sent: Friday, May 1, 2020 9:22 PM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS><EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS><EMAIL_ADDRESS>Mention<EMAIL_ADDRESS><EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
Geoff,
I'm actually quite keen to have a go at 1911. The fact that each record has a simpler structure means that I can probably do a better job of auto-generating PARMS files. I'll see what I can do for Somerset, and will await your comments with interest!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622350368 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFKWICYDLFKF4Z3F3ILRPKWEDANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFLJ3D3C2KKPK3KSX3TRPKWEDA5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUMFAIA.gif
--

Richard Light
<EMAIL_ADDRESS><EMAIL_ADDRESS>@richardofsussex

Kirk et al
I appears that we are trying to fit a square peg in a round hole.
So …
From a programming need what should we be recording to meet our needs into the future, including future reporting?
That is the position we have taken on the new fields descriptions. Why don’t we consider this issue from scratch?
I will start:
We need the Civil Parishes – they match up with the field in the transcriptions.
We need at least ‘a’ and ‘b’ codes or columns or a different separator. They define the data in the spreadsheet.
If we look at it from this perspective we will have what we need in the end.
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Saturday, May 2, 2020 2:21 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
@richardofsussex https://github.com/richardofsussex  Your extract of 1911 should not be uploaded directly; it would benefit from a manual edit as some subplaces are clearly combined because you have no ability to detect the separation. But it would not be a large task. Well done; Would love to see one for 1901
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622455352 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFKVJ4XAJVX2QFTH4Q3RPLZFVANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFOD7JG4JYYJRPQ5DRDRPLZFVA5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUM6UOA.gif

Kirks suggestion makes very good sense to me.
It will give us a natural hierarchy:
County
            Registration District

                            Sub Registration District

                                            Civil Parish

That looks like a very good hierarchy to have to me. It should meet a lot of possible needs.
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Saturday, May 2, 2020 3:27 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
@richardofsussex https://github.com/richardofsussex  unfortunately programmers as you know need precision and definition; without it chaos reigns supreme.
For both @geoffj-FUG https://github.com/geoffj-FUG  and @FreecenBren https://github.com/FreecenBren  the purpose of this new set of extracts is for CEN2 ingestion not CEN1. So that fields reflect the TNA without a 20 character constraint
The Purpose of the b field for checking is one specified by me in processing a csv data file. The b field may well have other uses; in CEN1 you can search by it.
The a field is used as the search district
I would like to know if we can also record the Registration District in the parms in the second field as that does not appear to be used for anything????????????
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622481755 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFOFHRMIQCZ6OZ3P5C3RPMA77ANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFKIMOFSCHGTVJD5ELTRPMA77A5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUNFCWY.gif

I would love to see a 1901 for Notts as we are staring it shortly.
Brenda
From: Kirk Dawson<EMAIL_ADDRESS>Sent: 01 May 2020 17:21
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: Brenda<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
@richardofsussex https://github.com/richardofsussex  Your extract of 1911 should not be uploaded directly; it would benefit from a manual edit as some subplaces are clearly combined because you have no ability to detect the separation. But it would not be a large task. Well done; Would love to see one for 1901
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-622455352 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIL3VT3VS25LWO7L2OA6L3RPLZFTANCNFSM4LUK4O7Q .

OK, this is the 1901 for Notts, as generated by my system. I notice that it has far fewer 'b' entries than the 1881 one, but haven't had time to check why this might be.  Anyway, I would welcome your reaction to it. (The names with question marks are those which didn't match a list of places which I generated from the 1891 data.)
NTT1901.zip

@richardofsussex does your comment refer to what we have on the website or what you have downloaded anew

@richardofsussex Having spent some time digging I think it is time for CEN2 break from the specification used in CEN1. Hence I would like to specify a new format for CEN2 csv parms that collects all the information that is needed for CEN2 to have pieces that it can effectively link to a variety of different locations. This will contain the fields needed by CEN1 so we can create those from CEN2.
Please give me a week to develop the spec if that is acceptable; I don't think it will have a major impact on your current coding. We could perhaps think of moving to a json formatted file.

Ah: I'm talking about the file which I generated myself when Brenda asked me to produce some PARMS files for 1881.  I'm not sure what relation it will have to what finally ended up on the web site.

I would be happy for @Captainkirkdawson to produce a new spec for the parms data. I am limited in what I can contribute by my ignorance both of the source formats we are working from, and the target information processing environment they end up in. A 'statement of purpose' for the parms, summarizing their use both for data entry/validation and for searching/retrieval, would I think be a useful start. @geoffj-FUG's sterling work on the Ireland situation should I suggest be taken into account.
Over the weekend I have been trying, without much success, to get higher-level records (counties and Registration Districts) out of the TNA Discovery API. I'm assuming that it would be useful to have a complete geographical hierarchy for each census year. If we had this, could we just indicate where each Piece fits into the hierarchy?
As regards the serialization of the new parms, I would personally prefer XML (which is the format I work in all the time), so it would be useful to have at least the definition of an XML version which I can use as a target, even if the results are then converted to json.

@richardofsussex one of my questions for you was going to be can you obtain the Registration District so you were already ahead of me by looking at trying to extract that. Perhaps you could point me towards a document on the API as that might (might not) help me.

This page gives you access to the API Sandbox:
https://discovery.nationalarchives.gov.uk/API/sandbox/index#/SearchRecords
It includes fields for you to fill in, and shows you what the result will be. It also displays your search as a URL and as a call to curl, so you can 'take it away' and run it in a different environment. I have complete files for 1841-1891 which came from their TSV files, and which might be an easier source of Reg. Districts. I'm currently investigating the 'children' API command, which gives me a list of county records (level 4). The Reg. Districts are level 5 records.

Thanks Richard A quick scan indicates that the cost of my getting to
understand that api would far far exceed the benefit. I will leave in your
very capable hands

I think I can see a way to extract Reg. Districts via a two-stage XML extraction process. If this is definitely needed I will plough on with it. I'll do 1901 and 1911 first. Does it make sense to extract the TNA identifier for each unit, as well as its title? This would give us access to the full record, both for display, as a potential link for our users, and for programmatic access to the data it contains. The approach could be used at lower levels, e.g. parishes, and would remove one of my concerns which is the fact that the parish name, of itself, is not a unique identifier (i.e. there are several with the same name, sometimes in the same county!).

@richardofsussex Is the TNA identifier the C13335 in the following example?
https://discovery.nationalarchives.gov.uk/browse/r/h/C13335
This number changes as one drills down to the actual piece so it is a reference to a specific page
The following image has a number C3139040

Now before I go further in specification do you get access to the titles of the parents?

ie the LONDON-MIDDLESEX and the Registration District 1.KENSINGTON? I know you extract the RG 10/2 Registration Sub-District 1B St Mary Paddington. and the Civil Parish, Township or Place: Paddington

Yes, the 'C' numbers are record identifiers within the TNA catalogue. Being an archival recording system, there is a hierarchical relationship between these records, and each has a specific 'level' recorded.
In my earlier work I was using the record search facility, which returns an XML response including a RecordSearchResultViewModels element for each record. This search works nicely for level 6 records, i.e. Registration Sub-Districts, but doesn't seem to be able to find higher-level records. These search results do not include links to their parent record.
More recently, I have been exploring the 'children of' option, starting from the top level, e.g. RG10, and hoping to work all the way down to Sub-Districts. This search returns InformationAssetIdentityViewModel elements, which do have a ParentId specified.  However, since we are attacking them 'from above', we know that parent identity already.
I am currently stalled by a combination of minor irritations. I want to recursively call for child records using the XSLT document() function. By default the API delivers JSON, and I have bought a licence allowing me to use the XSLT 3.0 facility to load a JSON string, but for some reason my XML editor won't accept that I have this licence. I can't load XML because the URL syntax doesn't support that.
Happy to stick with CSV.

OK, I have finally succeeded in extracting what I think is all the data we need from the TNA Discovery catalogue, for 1901. I did this (in case you're interested) by applying an XSLT 3.0 transform to the top-level XML export, then recursively asking for the child records at each level until I got to Sub-District.  I'm not suggesting that we actually load the information in this format; at this point in the proceedings I just want agreement that I have extracted everything we need. Although I had a learning experience working with JSON in XSLT, this did have the advantage that the individual parishes are all neatly surrounded by paragraph markup in this format - which is not the case with their XML download format.
rg13-overview.zip

rg14-overview.zip
Here is RG14 (1911) in the same format. Does anyone know what the codes (e.g. RD 1 RS 1 ED 1) mean?

Assuming I have captured all the information that is required, it should be a quick and easy job to convert these XML 'overview' documents into CSV PARMS files in @Captainkirkdawson's preferred format.  You might like to think about having a single CSV load for each census year, rather than uploading the data county by county.

Rats: I have just realised that these results may not be complete (default limit on search results ...). So just comment on the format and content, please.

rg14-overview.zip
rg13-overview.zip
These should be complete. 1901 looks pretty good: there are some missing labels at the end of 1911.

Richard
ED1 is the Enumeration District. In 1911 there was a one to one relationship with the piece number.
EDs start at 1 for each Registration sub-District.
There are several Registration Districts to a Registration District. In 1911 the Registration Districts each had a number.
So it is part of the hierarchy County, Registration District, Registration sub-District, ED number.
Geoff
From: Richard Light<EMAIL_ADDRESS>Sent: Thursday, May 7, 2020 2:34 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
rg14-overview.zip https://github.com/FreeUKGen/FreeCENMigration/files/4588055/rg14-overview.zip
Here is RG14 (1911) in the same format. Does anyone know what the codes (e.g. RD 1 RS 1 ED 1) mean?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-624755214 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFOZ2RCN4DA3WFC3HPDRQGGP3ANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFO3WVI2YFKVHJPUS3DRQGGP3A5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEU6QEDQ.gif

@richardofsussex At this point after a 5 minute review is to say "Hats off". That is magnificent. More comments to follow but that you but it has all of what I was going to ask for and a little more besides

@richardofsussex as Geoff indicated the RD 1 RS 1 ED 10 in 1911 refer back to the Regional District; Sub District, Enumeration District number (a number within the subdistrict)

@richardofsussex I am tending at this point to say just provide the yearly xml and we extract into the database from the XML directly rather than converting and reconverting. Which based on all of this  could be usefully reshaped. How about trying 1861?

Will do, but I'll start by tidying up 1911 a bit. There is missing data at the end, and now I know what the codes mean I think I'll make them into an attribute rather than a 'parish'.

Richard
I have just opened your XML file in Dreamweaver (the limit of my expertise I am afraid) and the structure looks good to me. It is the complete hierarchy.
If I look beyond basic FreeCEN functionality to future use as Open Data the structure you have been able to extract will enable a wide range of targeted data to be obtained from the FreeCEN database.
Geoff
From: Richard Light<EMAIL_ADDRESS>Sent: Thursday, May 7, 2020 3:18 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Ability to load a parms file directly into RC2 (#833)
rg14-overview.zip https://github.com/FreeUKGen/FreeCENMigration/files/4588321/rg14-overview.zip
rg13-overview.zip https://github.com/FreeUKGen/FreeCENMigration/files/4588322/rg13-overview.zip
These should be complete. 1901 looks pretty good: there are some missing labels at the end of 1911.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-624778335 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFL4UKWEJDKUXSSZHEDRQGLU7ANCNFSM4LUK4O7Q .  https://github.com/notifications/beacon/AKCPIFPV53NC7M27FHARNC3RQGLU7A5CNFSM4LUK4O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEU6VYXY.gif

rg14-overview.zip
rg14-overview-2.zip
Here is 1911 in more complete form, with the naval and military entries (which had a different structure from normal parishes). The first file is the default result; the second moves the ED code up to an attribute.

To my delight, the 1861 data gives a viable result first time. (Apart from the ships!) The pattern of data is different, e.g.
<parms>    <census year="1861">       <county tnaid="C62656" name="MIDDLESEX">          <district tnaid="C116862" name="Registration District 1. KENSINGTON">             <subdistrict piece="RG 9/1" tnaid="C3098036" name="">                <parish>Registration Sub-District 1 St Mary Paddington. </parish>                <parish>Parish: Paddington (9); Hamlet: Kensal Green (part) (Divided between RG 9/1, 14, 39 and 785).</parish>             </subdistrict>
I'll write a tidying-up XSLT to put the sub-district names where they ought to be. I can take the unnecessary 'headings' out of the data and split up the parish entries. Tempted to name the elements after the heading they have been given, so the second entry (Kensal Green) would become a 'hamlet' element - then you'll know which are the civil parishes. Good plan?
rg9-overview.zip

rg9-overview-2.zip
Updated RG9 overview for comments.

rg10-overview-3.zip
RG10 with tidying-up of parishes and hamlets.

@richardofsussex I need to take stock of what you have provided to date. So hold fire on tweaking just a little for feedback. Will work 14/13/9/10 if this is OK or do you want a different order? Need to have breakfast first

No problem: I'm happy with that order. Currently working on 11.

rg12-overview-3.zip
rg11-overview-3.zip
11 and 12.

@richardofsussex wrt rg14-overview-2
Excellent but there is one common issue;  the extraction of the code RD X RS Y ED Z is inconsistent.
Look at lines 66350-66514 for examples. Appears that it only works if there is 1 parish; does not if 2 or more.
The Royal Navy and Military . Each piece ends with the District number eg RD640 and 641.  Would it be possible to place that in the District name and remove as parish

@richardofsussex wrt rg13-overview
Usable as is

@richardofsussex wrt rg9-overview-2
a) Can we make the hamlet a child of the parish please
b) Can we extract the Islands from the Isles of the British

and their parish children
PS the tnaid is an absolute godsend;  just hope they never change it!!

@richardofsussex wrt rg10-overview-3
a)Really need the District name as well as its number eg

b)Like rg9 hamlet to be a child of the parish
c) For Royal Navy the ships; should be the district name

@Captainkirkdawson thank you very much for the comments. I'm packing up now for the day, but I'll finish with a not-quite-ready 1841 - the townships need separating into separate elements. Tomorrow is a Bank Holiday, so it will probably be next week before I get back to this.  Still, lots of progress!
1841-overview-2.zip

@richardofsussex wrt rg11-overview-3
a) we are missing the county name
b) as in rg10 we are missing the district name
c) we are missing the subdistrict name
d) when fixing c) please ensure that royal navy subdistrict name is Ships as noted for rg10

@richardofsussex  wrt  rg12-overview-3
a) as in rg10 and 11 we are missing the district name
b) missing district number

@richardofsussex Thank you for the work todate I have plenty to get on with here and also an issue on REG that is urgent
Like to compliment you on the work to date we are extremely close to an excellent solution that will give us a firm base for the future and some new capabilities.
Have a good weekend and avoid the virus

1841-overview-3.zip
Split the townships while the leeks were cooking!

@richardofsussex wrt 1841-overview-3
As you have correctly noted there were no district or subdistricts in 1841. They had Hundred or Wapentake depending upon the county. Soke and Liberty were also possible units but appear to have been associated with a hamlet or township.
For our use we should use the term that is in the census ie Hundreds or Wapentake as a  child within a county. It will be useful to display this to the researcher. It will have a name associated with it and have parishes as children. The parish may have hamlets or townships as children. In some counties there is also ALLOTMENTS IN FENS this should be treated as a child

It looks as though a recording policy is emerging as we work through the data. You're encouraging me to retain the original naming of place types and to express the hierarchical relationships between them where known/knowable. Thus far I haven't always done this, tending more towards harmonization of their varying practice.  Also (now I come to think about it) I have used different encoding approaches at different levels. Would it make sense to make our approach consistent, by putting all place names into a <place> element, with type and name attributes? Then when working with the data, you could choose either to notice or to ignore the place type.

Well said Richard. I think that is a really good idea.
On Fri, 8 May 2020 at 10:05, Richard Light<EMAIL_ADDRESS>wrote:

It looks as though a recording policy is emerging as we work through the
data. You're encouraging me to retain the original naming of place types
and to express the hierarchical relationships between them where
known/knowable. Thus far I haven't always done this, tending more towards
harmonization of their varying practice. Also (now I come to think about
it) I have used different encoding approaches at different levels. Would it
make sense to make our approach consistent, by putting all place names
into a  element, with type and name attributes? Then when
working with the data, you could choose either to notice or to ignore the
place type.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/833#issuecomment-625718350,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADIL3VRMSSB6JKFX2U7CKPDRQPDMLANCNFSM4LUK4O7Q
.

Thus instead of:
<parms>    <census year="1841">       <county tnaid="C132704" name="BEDFORDSHIRE">          <district tnaid="C3760734" piece="HO 107/1" name="Barford">             <parish>Great Barford</parish>             <parish>Colmworth</parish>             <parish>(3)Eaton-Socon</parish>             <township>Wyboston</township>             <parish>Goldington</parish>
we would have:
<parms>    <census year="1841">       <place type="county" tnaid="C132704" name="BEDFORDSHIRE">          <place type="district" tnaid="C3760734" piece="HO 107/1" name="Barford">             <place type="parish" name="Great Barford"/>             <place type="parish" name="Colmworth"/>             <place type="parish" name="(3)Eaton-Socon">                <place type="township" name="Wyboston"/>             </place>                         <place type="parish" name="Goldington"/>
(Note the township now correctly within the parish: this would be hard to do cleanly with our current encoding strategy.) The important thing is that the data should be in a form which is most useful for the various tasks it will have to perform, both for input/validation and for searching/browsing/retrieval.

@richardofsussex I have to say that I am now 100% confused. At the start of this week I had been working on a new specification for parms csv files that were to be populated by the API extracts. All census for England from 1851 -1911 have essentially the same structure. 1841 is uniquely different. I was evolving toward Census/Country/District/Subdistrict/Parish,/Hamlet/Notes fields which would fit into a revised database structure. I never got to the point of writing that specification with the receipt of the revised extracts based on the parent child extracts in xml. These were seen as likely making the csv specification irrelevant since xml files can be directly converted into a ruby hash and is based on TNA structure.
At no stage in this have I used the term place. That was deliberate because unfortunately place means something different to everyone; even TNA avoids its use except in the occasional text. It is a term that causes miscommunication throughout FreeUKGEN.
The new proposal appears to have a flat structure with every element a place. The actual coding is beyond my level of understanding of xml with the trailing / and the place closure eg    Programmatic processing of such a file is currently conceptually beyond me.
Perhaps a game of  golf will clear my brain.

@Captainkirkdawson no, the suggested structure is anything but flat. I apologize: I omitted the end-tags from my example and then GitHub lumped all the markup together. Hopefully the image above makes my intention clearer. This is only a suggestion, inspired in part by your suggestion that we should retain the distinction between hundreds, wapentakes, sokes, etc. for 1841. I'm happy to go the other way; towards a more uniform structure.

1851-overview-2.zip
@Captainkirkdawson here is 1851 (the last one to be addressed). I have made the XML with specific tags for each level: District, Sub-District, Parish and Hamlet. They should all have a name attribute, and possibly piece (Districts only), code and note attributes.  Is this a pattern which you would be happy for me to apply to all the later years? If we can agree on that format, then we can look at 1841 as a special case and decide how to deal with it.

rg14-overview-4.zip
@Captainkirkdawson I've made the changes you requested to rg14. Also placed the parish name in a 'name' attribute, and split off comments in brackets into a 'note' attribute.

@richardofsussex   wrt 1851-overview-2
A quick review shows a problem. There are no parishes for many sub districts. Just look at the first 2. I suspect that parish is not being extracted if there is no hamlet.

Good catch: I think I can fix this quickly.

1851-overview-2.zip
Here we go ...

@richardofsussex New version of 1851-overview-2 looks to be fine. Like the notes extraction.
Will not know more until I try to process into the database. Will not be attempting that for a while as I have too much on my plate
rg14-overview-4 has addressed the issues raised and looks OK subject to same caveat above

That's fine: no rush. So long as I know you are happy with the format I am producing, I can continue to tidy up the other years.  So long as the XML is consistently structured, it wouldn't be a big job for me to tweak it, were there to be problems when you come to load it.

rg9-overview-4.zip
Here is rg9, with the islands and the name/note analysis. I tried and failed in a search for the ships: "recorded elsewhere" ...

rg10-overview-6.zip
Here is the updated rg10, with District names restored, hamlets within parishes and the Royal Navy districts recorded as "Ships".

rg10-overview-6.zip
Ditto with the ships actually present! (They were recorded as hamlets, so were stripped out by a routine which expected there always to be a parish present.)

rg11-overview-4a.zip
rg11 updated.  Sorry: that was a really poor result first time round!

rg12-overview-7.zip
rg12 updated. I think that's the lot ...

Thanks @richardofsussex what about 1841 with the changes requested; its a 1 off

Sorry: those comments were a long way back in the thread! Looking at the routine which I used to tidy the 1841 data, I see that the areas within a county were Hundred, Ward, Borough, City and Borough, Liberty or City, and Wapentake. Given that two of them are three-word phrases, and that 'City' and 'Borough' occur in two of them, I suggest that we could keep the neutral  element within , and provide a 'type' attribute which records the type of district:
    <county tnaid="C132704" name="BEDFORDSHIRE">          <district type="Hundred" tnaid="C3760734" piece="HO 107/1" name="Barford">             <parish>Parish: Great Barford. </parish>
(You can't have spaces in XML element names, and e.g. <Liberty_or_City> seems clumsy to me.)

That should read "keep the neutral <district> element within <county".

OK as long as the parish was a child of the district in the county, put the actual value in a type field with the name field
Given that 1841 is much smaller I should be able to extract appropriately

1841-overview-4a.zip
See how this looks ...

@richardofsussex just starting to look inputting what you produced into CEN2 and in checking what I had I was left with the impression that 1841 was not reprocessed to follow the requested format. The last version I have is 1841-overview-3 which has the format

We had agreed to reformat this using the comments made above on May 7.
that generated the format in 1851

ie county/hundred(or wapentake)/parish/township (hamlet)

1841-overview-4a.zip
Here's version 4 - apologies if I failed to upload it. This has the four-level hierarchy.  I know 1841 is a one-off, so please get back to me if this isn't the format you need.
Richard

Thanks Richard

Interesting it appears that zenhub has a limit on comments. I had looked for 1841 on zenhub and did not find it; now looking at github I see it was there.
@richardofsussex  Could I request a change to 1851 overview-2 if it can be done. Currently it has the piece number in the District in 1861 and more recent it is at the sub district level. In 1851 the district and subdistrict are combined as part of the piece eg HO 107/1466 1851
Registration District: 1. KENSINGTON (Street Indexed) Registration Sub-District: 1 St Mary Paddington Parish: Paddington (3) (4) Hamlet: Kensal Green (part); You correctly extract out the Registration District but pull the piece number into it as well, it will help if the piece number is kept with the sub district

@richardofsussex a further request. rg13-overview has a very early format; after that you evolved to splitting the parish into the name and notes components (see rg12-overview-7 and rg14-overview-4) so rg13 is the only one with the old layout.
Would it be possible to reprocess rg13 to have the latest format?

1851-overview-3.zip
Updated 1851 as requested.

rg13-overview-2.zip
Updated rg13 as requested. Also noticed that some subdistricts had bracketed expressions in them, and extracted these to a note attribute.

@richardofsussex Sorry to trouble you but 1841 is incomplete; it has only about half of the counties. It ends at RUTLAND. Looking at TNA directly that is where the first page of 30 entries ends, There is a second page of entries. Starting with SHROPSHIRE , the rest of England, Wales and the Islands

@Captainkirkdawson it's a fair cop! As you say, the default page size for the TNA search is 30, and I failed to up the limit. In fact, the same applies to 1851, so I'll have to do that again too. One positive outcome of this is that I have written down what I did to generate these results, which is something that will have to be done again in 2022 when the 1921 census is released (and on into the future). Given I had trouble remembering what I did two weeks ago, I thought it was worth documenting it!
Extraction and processing of TNA census records for FreeCEN.docx

1851-full-overview-3.zip
Here's the complete 1851 result. Still working on 1841, not helped by a parsing error in the JSON generated by TNA.

Thanks @richardofsussex

I'm getting close with 1841. Just picked up some district types which were previously getting lost from the result. One question: there is the situation where Rapes contain Hundreds (e.g. https://discovery.nationalarchives.gov.uk/browse/r/h/C3761828). Should I use the  element for Hundreds in this case? It would be the only use of this tag in the 1841 data, but I don't see how else to correctly model the relationship between them.
In fact, there are trickier examples now I look. For example, in HO 107/23 (https://discovery.nationalarchives.gov.uk/browse/r/h/C3760756) there are two Hundreds: Moreton and Wallingford Borough. Which of them is the ? Or do I make each of them a separate  and give them both the same piece number? There is a hierarchical relationship between each Hundred and its Parishes which needs to be retained, whatever we decide.

@richardofsussex  the following makes interesting reading https://en.wikipedia.org/wiki/Rape_of_Bramber
I think we should simply use the Hundred of Brightford (IF it were easy to put the rape in the note do so BUT don't invest much effort to so do).
This example also has a second Hundred: Burbeach. Just like your other example. In each case we establish them a separate  and give them both the same piece number? There is a hierarchical relationship between each Hundred and its Parishes which needs to be retained,
Could you at an A/B to the piece number. TNA has done that elsewhere.
I propose to move to a string for the piece_number; Scotland also uses A/B and Geoff wants to use alphanumerics in the Ireland piece

@Captainkirkdawson thanks - I'll do that. Ignore the Rape as part of our hierarchy, and split on multiple Hundreds, using A/B/C/... suffixes for the piece number for each Hundred. That's neat, in that the one strategy deals with both the cases I have spotted.

1841-overview-full-5a.zip
Here's 1841. I applied the various transforms very mechanically, so it is possible that errors have slipped through. Let me know if you find any and I'll try to fix them.

Thx @richardofsussex

1851-tidied.zip
1841-tidied.zip
1911-tidied.zip
1901-tidied.zip
1891-tidied.zip
1881-tidied.zip
1871-tidied.zip
1861-tidied.zip
Hi, I have spent some time improving the data, by loading it into Modes and looking for places where the automated parsing of TNA catalogue entries didn't quite work. Results are attached herewith.

Thank you @richardofsussex this is most timely as I am just in the process of loading the information into CEN2 having just completed the rake task that ingests the files. In doing so I had noted the odd wart that I was going to have to come back to.
The structure has not been changed I hope.

No, not intentionally. I simply split the data associated with each TNA identifier into a separate Modes (XML) record, and left a 'the child goes here' processing instruction in its place. Then wrote an XSLT transform to pull them all back into a single XML document (for each year). The names are all indexed in Modes, so I could potentially develop a 'place name checking' web service if this would help transcribers.

@richardofsussex
TY for the clarification
I am unconvinced on the need for a separate database or whatever of this information.
The whole purpose of these extracts was to load them into CEN2 where they can be used in a large
number of different ways.

OK, that's fine.

@richardofsussex we have a problem with the 1851 extraction  in both versions
https://discovery.nationalarchives.gov.uk/browse/r/h/C132814  Islands in the British seas is not extracted effectively . Both extracts have

If we can could we extract the specific island to the district/subdistrict and then the parish perhaps replicate the island name to County/District and subdistrict names

@richardofsussex
It would appear that you have added year fields to district and subdistrict

1851 islands no problem: I have the original data and will re-process that bit of it.

Year fields: yes, I did that so the year would be available for retrieval in the separate XML records I created in Modes for each TNA id's data. If it's a problem for you, I can write an XSLT to strip them out.

@richardofsussex its no problem can likeley make use of it directly rather than a lookup from the parent.

... which is pretty much the reason why I put them in there! Good.

1851-tidied-2.zip
1851-islands-4.zip
@Captainkirkdawson here is a complete replacement for 1851, plus the islands on their own. Let me know if you find other glitches - this is entirely possible!

rg10-overview-9.zip
rg11-overview-6.zip
rg13-overview-3.zip
rg14-overview-6.zip
rg9-overview-6.zip
Here are updated versions of five of the files. Following your discovery of the missing islands, I searched all the files for empty <district> and <subdistrict> elements. The biggest issue I hit was where subdistricts for islands and navy/army were alongside rather than inside their districts. There are a handful of other issues which I edited by hand.
1891 has a chunk of data missing where they changed the label to 'Parish, Township or Parish' and I didn't pick this up. I have re-extracted the relevant records, and will process them soon. 1841 and 1851 also require some non-trivial work. There are no major losses of information, but I would like to get this transfer from the TNA data as close to 100% accurate as possible.

@richardofsussex Can I please ask that you tell me when you change names as I have just noticed that the latest version of 1851 includes a new field name prenote. Unfortunately this caused my extraction software to crash.
I would like to start specific year new parms stories as this one is getting fair too many comments

@richardofsussex The new stories are 901-908 inclusive. Please use those for any new files and I will use them to provide feedback

This is closed


New feature: Serial Terminal

Mbed CLI already has a -f option to automatically flash the program to the connected board after compiling. This PR takes the next natural next step to add serial terminal via -s/--sterm option ("s" for serial) to mbed compile and mbed detect.
Workflow
This PR extends mbed compile and mbed detect by adding -s/--sterm option. The option can be chained with -f/--flash (with mbed compile) or -r/--reset (with mbed detect), meaning that Mbed CLI will first apply the new firmware image, reset the MCU/target/board and then open serial terminal.
The serial terminal offers few commonly used shortcuts:

Quit: CTRL+C or CTRL+]
Reset: CTRL+B or CTRL+R
Echo toggle: CTRL+E
Terminal info: TAB or CTRL+I
Help: Ctrl+H
Menu: Ctrl+T (PySerial)

There are much more options behind the terminal menu (Ctrl+T).
The implementation is based on PySerial, which is one of the fundamental libraries for Mbed Greentea (mbed test). Therefore there are no additional dependencies.
Example usage with mbed compile
$ mbed compile -t GCC_ARM -m NUCLEO_F401RE -f -s
Building project mbed-os-example-blinky (NUCLEO_F401RE, GCC_ARM)
Scan: .
Scan: mbed
Scan: env
Compile [100.0%]: main.cpp
Link: mbed-os-example-blinky
Elf2Bin: mbed-os-example-blinky
+------------------+-------+-------+------+
| Module           | .text | .data | .bss |
+------------------+-------+-------+------+
| [fill]           |   102 |     7 |   11 |
| [lib]/c.a        | 27166 |  2204 |   56 |
| [lib]/gcc.a      |  3752 |     0 |    0 |
| [lib]/m.a        |    88 |     0 |    0 |
| [lib]/misc       |   236 |    16 |   28 |
| main.o           |   267 |     4 |  276 |
| mbed-os/drivers  |  1301 |     4 |  100 |
| mbed-os/hal      |  1395 |     4 |   66 |
| mbed-os/platform |  3096 |     4 |  354 |
| mbed-os/rtos     |  9114 |   168 | 5989 |
| mbed-os/targets  |  7017 |     5 |  388 |
| Subtotals        | 53534 |  2416 | 7268 |
+------------------+-------+-------+------+
Total Static RAM memory (data + bss): 9684 bytes
Total Flash memory (text + data): 55950 bytes

Image: ./BUILD/NUCLEO_F401RE/GCC_ARM/mbed-os-example-blinky.bin
[mbed] Detected "NUCLEO_F401RE" connected to "/Volumes/NODE_F401RE" and using com port "/dev/tty.usbmodem1413"
[mbed] --- Terminal on /dev/tty.usbmodem1413 - 9600,8,N,1 ---
[mbed] --- Quit: CTRL+C | Reset: CTRL+B | Echo: CTRL+E ---
[mbed] --- Info: TAB    | Help:  Ctrl+H | Menu: Ctrl+T ---
$$$07200221061A657F395EF362start
something 1
something 2
something 3
something 4
something 5
something 6
something 7
something 8
something 9

Example use with mbed detect
$ mbed detect -rs
[mbed] Detected "NUCLEO_F401RE" connected to "/Volumes/NODE_F401RE" and using com port "/dev/tty.usbmodem1413"
[mbed] --- Terminal on /dev/tty.usbmodem1413 - 9600,8,N,1 ---
[mbed] --- Quit: CTRL+C | Reset: CTRL+B | Echo: CTRL+E ---
[mbed] --- Info: TAB    | Help:  Ctrl+H | Menu: Ctrl+T ---
$$$07200221061A657F395EF362start
something 1
something 2
something 3
something 4
something 5
something 6
something 7

Inspired by @yennster's request (#639).
@yennster Do you think you could look into adding help for this feature in the README.md?

It seems odd to me that compile would open a serial terminal. I fear we are turning mbed compile into mbed do. That being said, I can't think of a verb that means build it then run it.

@screamerbg All that remains to be fixed is the two except:s

Why is this only an option to detect/compile? Can we start it by itself? like, "mbed startserial " OR "mbed sterm".
Also from a developer standpoint, people tend to keep separate terminal to run the builds. So it would be useful if I can do "mbed sterm" separately. And yes, as @sg- mentioned we can potentially expand on this to be used with Binary/ID-based tracing so that it automatically pick the right elf file to do the ID-string mapping.

@SenRamakri How is mbed sterm different from mbed detect -s? both have to detect attached targets before attaching the terminal.

If "mbed detect -s" take options in case of multiple targets, then yes, we don't need "mbed sterm".

@SenRamakri What do you mean by "take options in case of multiple targets"?  What options would it take? how do they apply to multiple targets?

Offline discussion with @theotherjimmy here confirmed that "mbed detect" does pick one of many targets if required. So, "mbed detect -s" does support what I was looking for. Thanks for confirming, @theotherjimmy

So glad this is taking off @screamerbg ! I'll take a look at the README tomorrow 😄

@screamerbg Is there a way to change the baud rate for the serial monitor? Right now it looks like it defaults to 9600.

Does this feature support terminal codes (ANSI/VT100) ?

@yennster You can change the baudrate form within the terminal using Ctrl+T + B combo. There's no commandline flag (yet)

@jupe The implementation invokes PySerial miniterm. This is from miniterm's documentation:
This is a console application that provides a small terminal application.
Miniterm itself does not implement any terminal features such as VT102 compatibility. However it may inherit these features from the terminal it is run. For example on GNU/Linux running from an xterm it will support the escape sequences of the xterm.  When ANSI.sys is loaded it supports some escapes.
http://pyserial.readthedocs.io/en/latest/tools.html#module-serial.tools.miniterm
On Mac terminal it provides a fully featured ANSI support. Had a chance to try it with an ANSI  embedded program @stevew817 wrote.

Follow up PR #664


add getpass primitive to access to implicit pass object

Codecov Report

Merging #34 (c58bc52) into master (4da10df) will decrease coverage by 0.29%.
The diff coverage is 80.00%.

@@            Coverage Diff             @@
##           master      #34      +/-   ##
==========================================
- Coverage   89.88%   89.59%   -0.29%
==========================================
  Files           1        1
  Lines         168      173       +5
==========================================
+ Hits          151      155       +4
- Misses         17       18       +1

Impacted Files
Coverage Δ

src/CassetteOverlay.jl
89.59% <80.00%> (-0.29%)
:arrow_down:


Default value of page param on reviews method

Is there some special reason for the default value of the page param on reviews method to be 1 instead of 0?
My concerning is about the compatibility with the google-play-scraper module.
I think that if we could make it compatible and work the same way, it'd be great for both APIs.

No special reason, probably just left what the store endpoint is using. You can send a PR changing it if you want.

fixed by #40


Blueprints naming convention update

Blueprints naming convention update
network-dashboard -> network-quota-dashboard
quota-monitoring -> compute-quota-monitoring
I applicable, I acknowledge that I have:

[X] Read the contributing guide
[X] Ran terraform fmt on all modified files
[X] Regenerated the relevant README.md files using tools/tfdoc.py
[X] Made sure all relevant tests pass

are there also links in the top-level readme?

Links updated, I was just missing to rename the Network Dashboard in that readme.. done that in the last commit. Everything should be aligned now


Adding Changelog URL to the README.md file

Adding Changelog URL to the README.md file

This is a great idea!
Looks like the other two lines shouldn't have been changed, though. Any chance you can change them back to what they were before?

This is a great idea!
Looks like the other two lines shouldn't have been changed, though. Any chance you can change them back to what they were before?

Thanks, @Hixie  for your reply, I added a new line for the changelog link, and it pushed the other lines one line down I didn't change anything just added 1 line

According to the diff on GitHub, line ten changed from * [Install](https://flutter.io/get-started/install/) to * [Install](https://flutter.io/setup/). Maybe this was something that changed while your branch was open or something? Anyway if you could change those to match what's on the trunk now that would let us land this PR. Thanks again for your contribution!

Hello @Hixie I pulled the latest version on dev branch and added one line for the changelog URL, please let me know if there are any other changes needed

@dahabit I've gone ahead and fixed a broken link and merged this with upstream - LGTM now!  Thanks for the contribution!

@dnfield  thanks for your effort, and sorry for the last response , yeah I think CI is messed up :D


argo_tunnel: Add initial support

Introduces support for Argo Tunnel CRUD operations (including cleaning
up connections).
API documentation: https://api.cloudflare.com/#argo-tunnel-properties
This unblocks cloudflare/terraform-provider-cloudflare#603

Thanks!


Fix deps

Please rewrite it appropriately as it may be written in an odd way.

Sorry.  I will fix it later.

Fixed.


fix: adjusting grammar in files #1

A few adjustments to the grammar.

Ready for merge?

Thanks again 👍


Make "handleStyle" optional and "top" and "left" configurable

handleStyle should be declared as optional, since it is optional on code.
"top" and "left" should be configurable, because one might want to use e.g. top: 0% and height: 100% so handle covers the complete height.

@Franziskus1988 thank you for the contribution! It definitely makes sense and looks fine for me

@titovmx  Do you plan to update the version number in the package.json?

Really cool. Thanks a lot.


Angular 5+

First of all i think this is an awesome project and for that reason: Is there an Angular 5+ version of this ?

Not yet, i have it in plans.

Is there many of you who need this?


.tik downloads not working?

When I press the green download button to get a .tik, it just redirects me to the same site, only with the title id at the end of the URL, but it can't download it.

yep, still broken (or broken again)


Drive autocomplete demo from sub-ontology

This is a stretch goal for the workshop.
TODO: make subontology with cells only.  This should be straightforward with ROBOT

Spec: subontology should only have subclasses of 'cell' (cell ontology).
Add this to OLS for us in autocomplete

Is this query sufficient to build sub ontology: https://gist.github.com/hkir-dev/17ff24bdcbd74d0d21eae4777e2dcdea
Generated cells subset is : https://app.zenhub.com/files/299404353/98b4b5a9-3f7a-4795-8ffe-fab3fb9dc9e6/download

Filtering created a much compact subontology:
robot filter --input bdscratch-full.owl --term CL:0000000 --select annotations --select descendants --axioms all --signature true --trim true --output bdscratch-filter.ofn

https://app.zenhub.com/files/299404353/4a45c4fb-74f6-41f7-b816-8da9c638e2db/download

Looks good!  Probably shouldn't be used for browsing, but much better for driving autocomplete than full file.

OLS deployment of the subontology completed: http://ec2-3-143-113-50.us-east-2.compute.amazonaws.com:8080/ontologies/bds_cell
Auto complete demo redeployed (Warning: you may need to clean browser cache via Ctrl + Shift + R the demo page): http://ec2-3-143-113-50.us-east-2.compute.amazonaws.com:8484/autocomplete/demo.html
Old version (with the whole ontology) also still exists: http://ec2-3-143-113-50.us-east-2.compute.amazonaws.com:8484/autocomplete/demo_bds.html


add test collection httpapi

I add tests in folder httpapi collection file

impressive, many thanks!

This is not the end!


Detect session and contribution moving

Currently, the directory structure is the only integral part of the "identifier" of the session and files.
If a session or contribution is moved in the agenda, in time or location, all attachments will be redownloaded and old directories will be kept.
It would be useful to have another identifier (which is unique) and be able to move files accordingly. Alternatively, downloading with unique identifiers and symlinking may be an option in case it is well supported by the usual synchronization tools (ownCloud, Synchting etc.).

Using the checksums from #11 to map things would also be an option


dbeaver (5.1.3) doesn't start with openjdk 11

With rolling release distribution like openSUSE Tumbleweed we now have openjdk-11 as default java.
OS : openSUSE Tumbleweed 20180721-0
openjdk used
S  | Name                        | Type    | Version         | Arch   | Repository
---+-----------------------------+---------+-----------------+--------+-----------
i+ | java-11-openjdk             | package | <IP_ADDRESS>~20-1.1 | x86_64 | oss
i+ | java-11-openjdk-devel       | package | <IP_ADDRESS>~20-1.1 | x86_64 | oss
i+ | java-11-openjdk-headless    | package | <IP_ADDRESS>~20-1.1 | x86_64 | oss

Error found launching dbeaver
dbeaver
2018-07-23 16:35:52.981 - Create display
2018-07-23 16:35:53.338 - DBeaver 5.1.3 is starting
2018-07-23 16:35:53.340 - OS: Linux 4.17.6-1-default (amd64)
2018-07-23 16:35:53.340 - Java version: 11-internal by Oracle Corporation (64bit)
2018-07-23 16:35:53.341 - Install path: '/usr/share/dbeaver'
2018-07-23 16:35:53.341 - Instance path: 'file:/ioda/home/bruno/.dbeaver4/'
2018-07-23 16:35:53.341 - Memory available 77Mb/1024Mb
2018-07-23 16:35:53.350 - Starting RMI server at 61870
2018-07-23 16:35:53.403 - Run workbench
2018-07-23 16:35:53.436 - Internal error in workbench lifecycle
org.eclipse.e4.core.di.InjectionException: java.lang.NoClassDefFoundError: javax/annotation/PostConstruct
        at org.eclipse.e4.core.internal.di.InjectorImpl.internalMake(InjectorImpl.java:410)
        at org.eclipse.e4.core.internal.di.InjectorImpl.make(InjectorImpl.java:318)
        at org.eclipse.e4.core.contexts.ContextInjectionFactory.make(ContextInjectionFactory.java:162)
        at org.eclipse.e4.ui.internal.workbench.swt.E4Application.createDefaultHeadlessContext(E4Application.java:491)
        at org.eclipse.e4.ui.internal.workbench.swt.E4Application.createDefaultContext(E4Application.java:505)
        at org.eclipse.e4.ui.internal.workbench.swt.E4Application.createE4Workbench(E4Application.java:204)
        at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:614)
        at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
        at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:594)
        at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:148)
        at org.jkiss.dbeaver.core.application.DBeaverApplication.start(DBeaverApplication.java:162)
        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)
        at org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)
        at org.eclipse.equinox.launcher.Main.run(Main.java:1499)
        at org.eclipse.equinox.launcher.Main.main(Main.java:1472)
Caused by: java.lang.NoClassDefFoundError: javax/annotation/PostConstruct
        at org.eclipse.e4.core.internal.di.InjectorImpl.inject(InjectorImpl.java:124)
        at org.eclipse.e4.core.internal.di.InjectorImpl.internalMake(InjectorImpl.java:399)
        ... 23 more
Caused by: java.lang.ClassNotFoundException: javax.annotation.PostConstruct cannot be found by org.eclipse.e4.core.di_1.6.100.v20170421-1418
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:433)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387)
        at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:150)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 25 more
2018-07-23 16:35:53.447 - DBeaver is stopping
2018-07-23 16:35:53.448 - null
java.lang.IllegalStateException: Workbench has not been created yet.
        at org.eclipse.ui.PlatformUI.getWorkbench(PlatformUI.java:93)
        at org.jkiss.dbeaver.core.application.DBeaverApplication.shutdown(DBeaverApplication.java:320)
        at org.jkiss.dbeaver.core.application.DBeaverApplication.start(DBeaverApplication.java:171)
        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)
        at org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)
        at org.eclipse.equinox.launcher.Main.run(Main.java:1499)
        at org.eclipse.equinox.launcher.Main.main(Main.java:1472)

If you have any ideas that I can try to help making dbeaver working with openjdk 11 just tell me the steps

dbeaver start and work (basics function) with java-11 if it is rebuild see #3861

Java 11 is now released and DBeaver does not run correctly. It's missing javax/xml/bind/DatatypeConverter

Temporary fix: ```
-vm
/usr/lib/jvm/java-10-openjdk-amd64/bin/java
add that **just right before** -vmargs in dbeaver.ini

Temporary fix:
-vm
/usr/lib/jvm/java-10-openjdk-amd64/bin/java

add that just right before -vmargs in dbeaver.ini

With that you tell dbeaver to use openjdk-10 :-) Which obviously is not openjdk-11

@tigerfoot I know, that's exactly why it works. 😉https://openjdk.java.net/jeps/320

I believe java 11 support was released like yesterday :) did you upgrade to 5.2.3?

@p91paul @tigerfoot Upgraded to 5.2.4 today. Does still not work with Java 11.
Java: openjdk version "11.0.1" 2018-10-16
OS: Ubuntu 18.04

Happens when trying to connect to a MSSQL database.

Actually 5.2.4 compiled yesterday is working for me (connected to Oracle and PostgreSQL DB)
Attached full info
dbeaver.txt

The problem seems come back with v5.3.0 released Dec 10.

Logfile (/var/log/system.log)
Dec 10 13:14:19 myhost dbeaver[1520]: DEPRECATED USE in libdispatch client: dispatch source activated with no event handler set; set a breakpoint on _dispatch_bug_deprecated to debug

Yes, unfortunately we had to downgrade to Eclipse Oxygen which doesn't support Java 11. Because Latest Eclipse release was too unstable.
Workaround (config patch): #4308
also on Windows and MacOS you could use installers which are already bundled with proper Java version.

Yes, unfortunately we had to downgrade to Eclipse Oxygen which doesn't support Java 11. Because Latest Eclipse release was too unstable.
Workarounds: #4765 and #4308
Also on Windows and MacOS you could use installers which are already bundled with proper Java version.

Thank you! The workarounds work here.

I'm waiting for a new release.
I don't have time to mess with workarounds.

I agree, a warning next time to stop people running Linux and Java 11 from upgrading would have been nice, also a visual crash error. The workarounds posted does not work on Ubuntu.


[swc_bundler] Ignore external require

May be this fix will help with https://github.com/swc-project/swc/issues/2275. But it is for shure helps me

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Reset token

What
Ask users to re-login if the iat (issued at) is older than the TOKEN_RESET_IAT
Only log events for the users that are in one of the analytics groups.
Events that don't log user groups (such as search feedback and status change) will still get logged for everyone.
Why
To ensure that new google groups are attached to user tokens after being updated.
BC2-89

Just had a thought... how does (process.env.TOKEN_RESET_IAT > checkAuth.getIat(req.cookies.hackneyToken) behave when TOKEN_RESET_IAT is empty?


Offer to open changelog when an extension is updated

With every update to VS Code, users now get a prompt to read the release notes.
It would be nice to have the same feature for extensions. Prompt to read the Changelog (if exists) on every update to the extension
Edit:
Prompt might be a bit much, too many notifications. Instead a "What's new" button or a similar option when clicking on the gear icon would be better

Slack has this "gift" icon. On hover it says "What's new". Clicking opens a side pane with the new features.
In the extensions pane, we now have the "Update to .." button when there is an update available.
We can use the same space for a "What's new" button when there is no update and changelog exists for the extension. Clicking on which should open the Changelog tab.
@stevencl Thoughts?
cc @Microsoft/vscode

I like the idea of increasing the visibility of extensions' progress. One issue might be that all extensions end up having that changelog button because (almost) all have a prior version. That would decrease its value as a 'highlight'.

-1, I have set my extensions to auto update and I would never want to see any release editor opening. I am quite happy with this kind of silent updates.

As long as it doesn't come as notification, I'm fine with that... The more I call showInformationMessage, the less chances I have to get into Heaven.

A 'What's new' button after the user has updated an extension would be a decent solution. I definitely don't think we should be notifying the user.
Once the user closes VS Code and restarts, we would remove the What's New button. We could consider then exposing a 'View changelog' action in the gear/settings menu to provide access to the changelog if it exists. That way we could avoid the problem that @chrmarti points out.

Agreed on the "no more notifications please" thought.

I would LOVE to see a feature like this. I've posted some of my comments here: https://github.com/eamodio/vscode-gitlens/issues/222#issuecomment-354311075
Personally I think it is important for an extension to be able to provide details around an update (changelog, release notes, etc). But the current notification mechanism is far from ideal. IMO, it should be something discoverable but passive.
I kind of like the idea of a badge on the extensions button (which would go away if you open the extension pane) if there have been any updates. And then individual badge each extension that has been updated -- that would bring you to a changelog or something like that. Although the downside, is lots of badges that might not get looked at and therefore not cleared, which would just become visual noise.
Another option, would still have the same badge on the extensions button, but the extensions pane would also have that badge as a button somewhere, and clicking that would open up a single screen, with a list of any updates. This could either be automatic (i.e. vscode will track the updates), manual (a new api for an extension to add an entry), or a combination of both. I think a combination would be ideal. Where vscode would track the simple, Extension Foo was upgraded from 1.0 -> 2.0 (maybe with a changelog link if there is one), and an extension could add additional details (bounded) through a new api -- maybe something like showInformationMessage (addUpdateMessage)?
Thoughts?

Is this implemented yet? Because I think I saw a notification in vscode about an extension been updated and button which opens the extension's changelog

Not yet, but I will be starting to explore based on the feedback that is provided here. So ideas are very much welcome.

There's discussion happening here - https://github.com/microsoft/vscode/issues/102139

Out of curiosity, I see some extensions found a way to accomplish this way. But is there any vscode documentation to implement it ourself?


mfu_flist_walk: handle trailing slash in dirname

The main motivation for this is that DAOS often has root paths "/", since that is the root of a container. When the paths are built, this happens:
dir = "/"
name = "something"
path = "//something"

Although DAOS/DFS handles the double "//", it creates a slight performance and caching problem.
The path building was also modified to use snprintf, which is more performant. Even with the additional "/" check, this should should be slightly faster.
Signed-off-by: Dalton Bohning<EMAIL_ADDRESS>

Thanks.  Looks good to me.
Some related functions are in mfu_path_reduce() or mfu_path_strdup_reduce_str() both from common/mfu_path.c.  Using those are probably overkill here, but this reminded me of them.  They also process out any . and .. entries.

Thanks. Looks good to me.
Some related functions are in mfu_path_reduce() or mfu_path_strdup_reduce_str() both from common/mfu_path.c. Using those are probably overkill here, but this reminded me of them. They also process out any . and .. entries.

Thanks @adammoody !
I agree that they are overkill, since the walk already ignores "." and ".." during readdir, and those mfu_ functions operate on a mfu_path (list of paths) type, which we don't otherwise need here.
But seeing those functions does make me wonder if there is room for improvement in some way (either here or there)... Another time maybe :)


User friendliness improvements

If we are reading stdin but it's not piped from a file/process, issue a warning. In a rare case this is intended (someone interactively debugging), but more often this is just a misunderstanding of the way CLI works, and an user mistakenly thinks that the CLI is hanging.
If the server reports an error and we fall back, tell users we are doing that, again to avoid a surprise.

I'm going to merge and release it.