Irish Parms

@geoffj-FUG wrote: have been investigating the structure of the Ireland census in more detail to ensure that we have everything covered.
The administration of Ireland had the following structure:
County
            Barony

                            Parish

                                            Poor Law Union

                                                            Town, Townland or Street

It therefore appears that the PARMS for Ireland should list the Barony and the Parish. The rest is picked up in the census transcription.
This makes sense as the Barony is the equivalent of the Hundred in the England and Wales 1841 census. The Parish is what we are testing transcriptions against.
But …   There are no piece numbers!
I have looked for anything that links the Form A, B1, B2 and N together and there is nothing. It had to be matched on the Townland or Street.
However there is a ray of hope:
The 1901 and 1911 censuses were not organised under the administrative structure. They were organised by District Electoral Division (DED), which can make it difficult to limit the search to a particular parish or district on the Irish Archives site.
The basic topographical divisions for the census are: County; District Electoral Division; Townland or Street. This is a simple hierarchical structure which makes it easy to access any area in the country. The returns are arranged in clusters by townland/street within district electoral division within county. For each townland/street, there are a number of original household returns, filled in and signed by heads of households, and three statistical returns, dealing with religious denominations, classification of buildings, and out-offices and farm-steadings, filled out by the Enumerator for that townland/street.
So in Ireland the DED should appear in the PARMS where the piece number appears in the other censuses.
We cannot create these PARMS automatically. However, it can be done manually. You can browse the area. Identify a return for a place-name in the area, and take note of the DED. Then click 'Browse census' on the menu bar and navigate down to that DED. You will see a list of all place-names in the DED and can explore them one by one.
What it means is that we have to be able to accept text into the piece number in the Ireland PARMS. Please keep this in mind as you redesign the PARMS file.
If we have this structure researchers can be enabled to do something on FreeCEN that they cannot do on the Irish Archives site – search by Parish.
Please don’t lose this email. It took ages to work out how things were structured. Perhaps it should kept as be a story? Topic Ireland PARMS.

I have just had a go at building an Ireland PARMS
I have copied and pasted the index for Antrim Rural from the Ireland Archives. That gave me the DED and the Parishes. No problem. It was really easy because it is a website the entries are already in a table. So it went straight into my spreadsheet.
I then looked up the Barony for a random sample.
There are many Baronies for one DED. It is not a straight forward one to one relationship. We would have to look up each one and have multiple a and b records.
So, I think the alternative is that we have to add Barony to the transcription spreadsheet. I had previously discounted it because it did not belong using the England and Wales model.
If we do that maybe an Irish PARMS should just be DED and Parish.
Sample Antrim Rural below. Ignore the links. That is where I suddenly realised I could copy and paste. Only a random sample of parishes have Barony entered. Belfast Lower belongs to balloo. I have just entered the Barony against other parishes. I know it is on the wrong line.
antrim rural
Belfast Lower
a

balloo
b
Masserene Lower
ballyarnott
b
ballycraigy
b
Masserene Lower
ballyharvey lower
b
ballyharvey upper
b
birch hill
b
blerick
b
brattonswalls
b
bush
b
caulside
b
craigyhall
b
Antrim Upper
crosskennon
b
farranshand
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Gallyhill/ Gallyhill
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Glenmullion/ Glenmullion
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Holywell/ Holywell
b
Upper Antrim
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Hungry_Hall/ Hungry Hall
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Hurtletoot/ Hurtletoot
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Irishtown/ Irishtown
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Island_bane/ Island bane
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Ladyhill/ Ladyhill
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Moylinny/ Moylinny
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Muckamore/ Muckamore
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Newpark/ Newpark
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Niblock/ Niblock
b
Masserene Lower
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Oldstone/ Oldstone
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Packhall/ Packhall
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Potterswall/ Potterswall
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Rathenraw/ Rathenraw
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Shanoguestown/ Shanoguestown
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Springfarm/ Springfarm
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Steeple/ Steeple
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Stiles/ Stiles
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Tirgracey/ Tirgracey
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Townparks/ Townparks
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Village_of_Muckamore/ Village of Muckamore
b
http://www.census.nationalarchives.ie/pages/1901/Antrim/Antrim_Rural/Whinpark/ Whinpark
b
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Tuesday, May 5, 2020 12:40 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [FreeUKGen/FreeCENMigration] Irish Parms (#884)
@geoffj-FUG https://github.com/geoffj-FUG  wrote: have been investigating the structure of the Ireland census in more detail to ensure that we have everything covered.
The administration of Ireland had the following structure:
County
        Barony

                        Parish

                                        Poor Law Union

                                                        Town, Townland or Street

It therefore appears that the PARMS for Ireland should list the Barony and the Parish. The rest is picked up in the census transcription.
This makes sense as the Barony is the equivalent of the Hundred in the England and Wales 1841 census. The Parish is what we are testing transcriptions against.
But … There are no piece numbers!
I have looked for anything that links the Form A, B1, B2 and N together and there is nothing. It had to be matched on the Townland or Street.
However there is a ray of hope:
The 1901 and 1911 censuses were not organised under the administrative structure. They were organised by District Electoral Division (DED), which can make it difficult to limit the search to a particular parish or district on the Irish Archives site.
The basic topographical divisions for the census are: County; District Electoral Division; Townland or Street. This is a simple hierarchical structure which makes it easy to access any area in the country. The returns are arranged in clusters by townland/street within district electoral division within county. For each townland/street, there are a number of original household returns, filled in and signed by heads of households, and three statistical returns, dealing with religious denominations, classification of buildings, and out-offices and farm-steadings, filled out by the Enumerator for that townland/street.
So in Ireland the DED should appear in the PARMS where the piece number appears in the other censuses.
We cannot create these PARMS automatically. However, it can be done manually. You can browse the area. Identify a return for a place-name in the area, and take note of the DED. Then click 'Browse census' on the menu bar and navigate down to that DED. You will see a list of all place-names in the DED and can explore them one by one.
What it means is that we have to be able to accept text into the piece number in the Ireland PARMS. Please keep this in mind as you redesign the PARMS file.
If we have this structure researchers can be enabled to do something on FreeCEN that they cannot do on the Irish Archives site – search by Parish.
Please don’t lose this email. It took ages to work out how things were structured. Perhaps it should kept as be a story? Topic Ireland PARMS.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/884 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFOMCSRM7M5XLCIV3MDRP3HSPANCNFSM4MYZ5YYQ .  https://github.com/notifications/beacon/AKCPIFNVLYGJPIK32N5GMFTRP3HSPA5CNFSM4MYZ5YY2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4JDZISTA.gif

A solution to the Irish problem – not that one, ours.
My confusion has been that in 1896 Ireland created District Electoral Divisions (DED) and replaced the Barony system, much as England replaced the Hundreds with Registration Districts between 1841 and 1851.
Except they didn’t! They had two parallel systems running, a registration system and an administrative system.
District Electoral Divisions originated as subdivisions of a Poor Law Union, grouping together a number of townlands to create a constituency for the election of members to a Poor Law Board of Guardians. As the Poor Law was funded by property taxes, the aim in creating the DEDs was to have areas that produced roughly similar tax takes. This meant that their boundaries ignored natural community borders, such as counties or parishes. It also means that it can be difficult to work out the geographical relationship between DEDs.
So instead of making these systems fit into a FreeCEN structure I have considered how they can be made to work.
Firstly, the England Registration Districts are the Irish DEDs so the DEDs go into the PARMS as the ‘a’ record.
Then the next level in the Census structure is the townland. So that goes into the PARMS as the ‘b’ record.
That raises 3 problems:
The townland is not the Civil Parish. They are separate entities. Therefore the civil parish cannot be tested against the PARMS for Ireland. The townland will appear in the Where census taken column of the spreadsheet. So we have to be able to test the townland against the where census taken field.
The Barony is still part of the Administrative structure. I did not have a field created for it on the spreadsheet because I thought it was part of the PARMS. So, we need to create a field for Barony and add it to the Ireland spreadsheets.
Thirdly there are no piece numbers. In my view the purpose of a piece number was an administrative access code for a library of pieces, so that they could be allocated, sorted, filed and retrieved in an orderly way. They are an administrative add on to the census itself.
Ireland chose to do this with an index based on the DED and townland, or in urban areas street. That does not suit us.
We need a code so that we can allocate, sort, file and retrieve our data in an orderly way. So, why don’t we create our own?
Each DED will have a number of townlands. I have been playing with Antrim to work this out so Antrim would start at ANT001. The coordinator would allocate a new code for each DED. If a DED was large they could break up a DED into smaller pieces, much as the towns are done by the EDs in England.
There are approximately 70,000 townlands etc in Ireland in 1901. There were 32 counties, an average of about 2200 per county. There are many townlands to a DED so a 3 digit number should suffice.
So I propose that FreeCEN should have 6 digit piece numbers with the Chapman Code as the prefix and a 3 digit number afterwards, created by whoever was doing the PARMS file. I have assumed this person to be the Coordinator.
This will then allow Ireland to be included in the FreeCEN database and the FreeCEN functions to work.
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Tuesday, May 5, 2020 12:40 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [FreeUKGen/FreeCENMigration] Irish Parms (#884)
@geoffj-FUG https://github.com/geoffj-FUG  wrote: have been investigating the structure of the Ireland census in more detail to ensure that we have everything covered.
The administration of Ireland had the following structure:
County
        Barony

                        Parish

                                        Poor Law Union

                                                        Town, Townland or Street

It therefore appears that the PARMS for Ireland should list the Barony and the Parish. The rest is picked up in the census transcription.
This makes sense as the Barony is the equivalent of the Hundred in the England and Wales 1841 census. The Parish is what we are testing transcriptions against.
But … There are no piece numbers!
I have looked for anything that links the Form A, B1, B2 and N together and there is nothing. It had to be matched on the Townland or Street.
However there is a ray of hope:
The 1901 and 1911 censuses were not organised under the administrative structure. They were organised by District Electoral Division (DED), which can make it difficult to limit the search to a particular parish or district on the Irish Archives site.
The basic topographical divisions for the census are: County; District Electoral Division; Townland or Street. This is a simple hierarchical structure which makes it easy to access any area in the country. The returns are arranged in clusters by townland/street within district electoral division within county. For each townland/street, there are a number of original household returns, filled in and signed by heads of households, and three statistical returns, dealing with religious denominations, classification of buildings, and out-offices and farm-steadings, filled out by the Enumerator for that townland/street.
So in Ireland the DED should appear in the PARMS where the piece number appears in the other censuses.
We cannot create these PARMS automatically. However, it can be done manually. You can browse the area. Identify a return for a place-name in the area, and take note of the DED. Then click 'Browse census' on the menu bar and navigate down to that DED. You will see a list of all place-names in the DED and can explore them one by one.
What it means is that we have to be able to accept text into the piece number in the Ireland PARMS. Please keep this in mind as you redesign the PARMS file.
If we have this structure researchers can be enabled to do something on FreeCEN that they cannot do on the Irish Archives site – search by Parish.
Please don’t lose this email. It took ages to work out how things were structured. Perhaps it should kept as be a story? Topic Ireland PARMS.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/884 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFOMCSRM7M5XLCIV3MDRP3HSPANCNFSM4MYZ5YYQ .  https://github.com/notifications/beacon/AKCPIFNVLYGJPIK32N5GMFTRP3HSPA5CNFSM4MYZ5YY2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4JDZISTA.gif

Geoff
I have some difficulty with where this seems to be heading. As you may have noted in the discussion on English Welsh parms we have gone away from terminology such as a) and b) instead we are talking about county, registration district, piece, registration sub district, parish with the latter sometimes having sudh components as hamlets or townships. This applies to 1851-1911; 1841 is different in that instead of districts and subdistricts we have Hundred or Wapentake. It seems to me that the Irish census can also fit into this

Specifically we treat DEDs as we do hundreds in 1941 census.  Each DED has Townlands/Streets as civil parishes The piece number 1 is assigned to the first DED and incremented by 1 for each subsequent DED. parishes.The County of Antrim would have 136 pieces or DEDs. (Your a) Each piece would have Townlands/Streets as civil parishes within it. (Your b). This way there is no need to change the piece number away from  an integer

Kirk
I have an concern with putting the townland/street in the civil parish. The civil parish is a separate entity that needs to be transcribed. It is part of the Administrative structure.
The Hundred, Wapentake and Barony are roughly the same thing. Parishes came beneath the Hundred and Wapentake. They come under the Barony also. But the Irish census does not follow that model.
The problem is caused by there being an administrative structure plus an unrelated census structure based on tax take by the poor law unions. Each tax area became the DED. The tax take ignored all normal boundaries and cut across parish boundaries as well as others. It even broke up townlands and streets. So there is no logical way that we can utilise the structure.
If we create a list of Baronies and Parishes there is no logical way we can work out where the townlands and street censuses are. Hence the difficulty in navigating the Irish Archives database if you want to research a parish.
That is why I wanted to put the townland/street in the where census taken column and test against that for Ireland. I know it would create a programming anomaly but I can see no other way of achieving this.
I have no problem in returning the piece numbers to numeric. As the PARMS would have to be created manually I am not sure how we would prevent duplication between counties. That is why I put the Chapman Code in as a prefix. We could give each County a number between 01 and 32 and that would become the prefix. The piece numbers would not be complete when looked at for the whole of Ireland as some would be missing. Does that create a problem?
I envisaged the PARMS prefix being the file name also. The files would be unique but the search for the townland against the piece would not if we remove the prefix. File names of course are flexible at this point and will depend on our solution.
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Friday, May 8, 2020 7:43 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Irish Parms (#884)
Specifically we treat DEDs as we do hundreds in 1941 census. Each DED has Townlands/Streets as civil parishes The piece number 1 is assigned to the first DED and incremented by 1 for each subsequent DED. parishes.The County of Antrim would have 136 pieces or DEDs. (Your a) Each piece would have Townlands/Streets as civil parishes within it. (Your b). This way there is no need to change the piece number away from an integer
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/884#issuecomment-625512407 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFJHARAFQ22247O4EA3RQMTONANCNFSM4MYZ5YYQ .  https://github.com/notifications/beacon/AKCPIFPN674GT64UMBQ7JY3RQMTONA5CNFSM4MYZ5YY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEVEI7VY.gif

You have  a couple of valid points there. I need time to think about the best way forward. However time to do that is limited and I feel doing this now is not the time. Irish census documents were not on the radar a month ago. Neither was the restructuring of the parms. As a result I have spent no time in completing the coding of the CSVProc nor the display of the additional fields we have been adding. That together with the spreadsheet modifications has put the online upload and processing system seriously behind and I have yet to put any real effort into the tools for coordinators and the management of their people. There are also pressing issues in REG that need addressing. Bottom line we have to focus and focus hard on what we really need at this point in time.

We require @PatReynolds input on this as it is turning into a larger story than anticipated.

Rich
In reviewing the available UK censuses for the extension into the 20th Century Ireland was identified as it was part of the UK in 1901 and 1911 - the years when the census is available.
When I looked at how it could be included in any design the current PARMS structure issue arose. If we do not take the PARMS structure for Ireland into account it will not be able to be incorporated at some time in the future. We have identified the fields needed and we need to identify the PARMS principles to be implemented.
So, my view is that if we can establish the principles for Ireland then we do not need to proceed any further at this point in time.
As I see it implementation of the CSVProc will need to be for England and Wales to start with as that is achievable.  I know that Richard Light has cracked the England and Wales PARMS problems and we are working towards a workable structure.
Then we need to look at the issues with Scotland, Transcriptions have started in FreeCEN1 but I understand that further progress is embargoed by Scotlands People. I do not know what the Scotland PARMS structure will look like. We have identified the fields for the data structure but I do not know what PARMS structure has been used in Scotland.
Transcription of Ireland is achievable as the images are currently in the public domain. We know what the PARMS will have to look like and we know the fields that will be needed.
So, my view is that we need to move forward from here one step at a time. England and Wales first. Knowing what is needed for Scotland and Ireland makes it easier to proceed further in the future.
Geoff

That's an elegant solution, Georff.
Best wishes,
Pat
On Tue, 5 May 2020 at 23:20, geoffj-FUG<EMAIL_ADDRESS>wrote:

A solution to the Irish problem – not that one, ours.
My confusion has been that in 1896 Ireland created District Electoral
Divisions (DED) and replaced the Barony system, much as England replaced
the Hundreds with Registration Districts between 1841 and 1851.
Except they didn’t! They had two parallel systems running, a registration
system and an administrative system.
District Electoral Divisions originated as subdivisions of a Poor Law
Union, grouping together a number of townlands to create a constituency for
the election of members to a Poor Law Board of Guardians. As the Poor Law
was funded by property taxes, the aim in creating the DEDs was to have
areas that produced roughly similar tax takes. This meant that their
boundaries ignored natural community borders, such as counties or parishes.
It also means that it can be difficult to work out the geographical
relationship between DEDs.
So instead of making these systems fit into a FreeCEN structure I have
considered how they can be made to work.
Firstly, the England Registration Districts are the Irish DEDs so the DEDs
go into the PARMS as the ‘a’ record.
Then the next level in the Census structure is the townland. So that goes
into the PARMS as the ‘b’ record.
That raises 3 problems:
The townland is not the Civil Parish. They are separate entities.
Therefore the civil parish cannot be tested against the PARMS for Ireland.
The townland will appear in the Where census taken column of the
spreadsheet. So we have to be able to test the townland against the where
census taken field.
The Barony is still part of the Administrative structure. I did not have a
field created for it on the spreadsheet because I thought it was part of
the PARMS. So, we need to create a field for Barony and add it to the
Ireland spreadsheets.
Thirdly there are no piece numbers. In my view the purpose of a piece
number was an administrative access code for a library of pieces, so that
they could be allocated, sorted, filed and retrieved in an orderly way.
They are an administrative add on to the census itself.
Ireland chose to do this with an index based on the DED and townland, or
in urban areas street. That does not suit us.
We need a code so that we can allocate, sort, file and retrieve our data
in an orderly way. So, why don’t we create our own?
Each DED will have a number of townlands. I have been playing with Antrim
to work this out so Antrim would start at ANT001. The coordinator would
allocate a new code for each DED. If a DED was large they could break up a
DED into smaller pieces, much as the towns are done by the EDs in England.
There are approximately 70,000 townlands etc in Ireland in 1901. There
were 32 counties, an average of about 2200 per county. There are many
townlands to a DED so a 3 digit number should suffice.
So I propose that FreeCEN should have 6 digit piece numbers with the
Chapman Code as the prefix and a 3 digit number afterwards, created by
whoever was doing the PARMS file. I have assumed this person to be the
Coordinator.
This will then allow Ireland to be included in the FreeCEN database and
the FreeCEN functions to work.
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Tuesday, May 5, 2020 12:40 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention <
<EMAIL_ADDRESS>Subject: [FreeUKGen/FreeCENMigration] Irish Parms (#884)
@geoffj-FUG https://github.com/geoffj-FUG wrote: have been
investigating the structure of the Ireland census in more detail to ensure
that we have everything covered.
The administration of Ireland had the following structure:
County
Barony
Parish
Poor Law Union
Town, Townland or Street
It therefore appears that the PARMS for Ireland should list the Barony and
the Parish. The rest is picked up in the census transcription.
This makes sense as the Barony is the equivalent of the Hundred in the
England and Wales 1841 census. The Parish is what we are testing
transcriptions against.
But … There are no piece numbers!
I have looked for anything that links the Form A, B1, B2 and N together
and there is nothing. It had to be matched on the Townland or Street.
However there is a ray of hope:
The 1901 and 1911 censuses were not organised under the administrative
structure. They were organised by District Electoral Division (DED), which
can make it difficult to limit the search to a particular parish or
district on the Irish Archives site.
The basic topographical divisions for the census are: County; District
Electoral Division; Townland or Street. This is a simple hierarchical
structure which makes it easy to access any area in the country. The
returns are arranged in clusters by townland/street within district
electoral division within county. For each townland/street, there are a
number of original household returns, filled in and signed by heads of
households, and three statistical returns, dealing with religious
denominations, classification of buildings, and out-offices and
farm-steadings, filled out by the Enumerator for that townland/street.
So in Ireland the DED should appear in the PARMS where the piece number
appears in the other censuses.
We cannot create these PARMS automatically. However, it can be done
manually. You can browse the area. Identify a return for a place-name in
the area, and take note of the DED. Then click 'Browse census' on the menu
bar and navigate down to that DED. You will see a list of all place-names
in the DED and can explore them one by one.
What it means is that we have to be able to accept text into the piece
number in the Ireland PARMS. Please keep this in mind as you redesign the
PARMS file.
If we have this structure researchers can be enabled to do something on
FreeCEN that they cannot do on the Irish Archives site – search by Parish.
Please don’t lose this email. It took ages to work out how things were
structured. Perhaps it should kept as be a story? Topic Ireland PARMS.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <
https://github.com/FreeUKGen/FreeCENMigration/issues/884> , or
unsubscribe <
https://github.com/notifications/unsubscribe-auth/AKCPIFOMCSRM7M5XLCIV3MDRP3HSPANCNFSM4MYZ5YYQ>
. <
https://github.com/notifications/beacon/AKCPIFNVLYGJPIK32N5GMFTRP3HSPA5CNFSM4MYZ5YY2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4JDZISTA.gif>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/884#issuecomment-624337151,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADK5GHMHGJ73PIBZNYZTZ6LRQCGLVANCNFSM4MYZ5YYQ
.

--
- -
Dr Pat Reynolds
Executive Director
A Charitable Incorporated Organisation registered in England and Wales,
number 1167484
VAT registration: 233 0105 70
Join us on social:
[image: : https://www.facebook.com/FreeUKGEN]
https://www.facebook.com/FreeUKGEN https://twitter.com/FreeUKGen
https://www.instagram.com/freeukgenealogy/
https://www.pinterest.co.uk/FreeUKGenealogy/
Please read our Privacy Notice
https://www.freeukgenealogy.org.uk/files/Documents/Privacy-Notice.pdf for
information on how we will protect and use your data.
https://www.easyfundraising.org.uk/causes/freeukgenealogy/?utm_campaign=raise-more
+44 1723 362616   +44 7943 145387
Westwood House,Westwood, Scarborough YO11 2JD, UK

New parms and file naming structure will support Irish census documents

This is one to file away in a suitable drawer for future reference. If I don’t document it then I will forget it.
I have downloaded a file from 'Irish Townlands' which contains the information that we couldn’t find when we tried to define the Irish PARMS. It allows us to follow our current structure.
Note that I have moved data columns and sorted data. I can easily re-download the original data.
Our structure is
County
Hundred or Registration District
Sub District (Piece)
Civil Parish
Hamlet
The Irish structure I now have is
County
Barony (which was an Irish Hundred)
DED (Electoral District) – which equates to a piece
Civil Parish
Townland
The DED is not fully populated. However the Irish Archives census index is structured as
County
DED
Townland.
So, If we can obtain a copy of the census index from them we should be able to do a join using the County / Townland  from each file.
I also think that we may find a many to many relationship between Civil Parishes and Townlands. I am not sure what effect that will have on the code. The result of my sort is very tidy so I feel that it is on the right track.
Of course Murphy’s Law says that there will be a mismatch in the number of Townlands in the files! But it’s the closest thing we have found so far.
That could give us a useable PARMS file and it would fit into our current code.
Geoff

A few weeks ago Pat asked me to investigate the Irish PARMS.
There are two structures in Ireland – the census structure based on District Electoral Divisions and the administrative structure based on the Hundred or barony system.
This exercise has identified a structure similar to the England and Wales PARMS structure. Such a structure reduces the amount of new coding required by the system.
Irish Census structure	Irish Barony Structure 	English PARMS Structure
County	County	County
District Electoral Division	Barony	Piece
Townland / Street	Civil Parish	Civil Parish
Townland / Street	Hamlet etc
The common link between the two Irish structures is the Townland. The Cities however were recorded within DEDs under streets, rather than Townlands.
If we work by Townland the fractured Civil Parish records can be collected during transcription.
The data is available in the following data sets. The Townland data set contains every PARMS field. This examination has therefore used the Townland dataset as its foundation data.
County	Barony	Civil Parish	Townland Street	DED
Civil Parishes		Civil Parishes
Barony	Barony
Townland	Townland	Townland	Townland	Townland
The attached spreadsheet has a download of all the Townlands from Irish Townlands Database. It covers the 1901 census. Whilst every Townland has been downloaded throughout Ireland, Antrim has been examined in detail as it contains a range of Townlands plus the City of Belfast.
For the purposes of creating a PARMS identical to the England PARMS we need the following information:
County, Barony, Civil Parish, Townland. This conforms to the structure of the 1841 England PARMS of County, Hundred, Civil Parish, Hamlet.
The census structure for Ireland was based on District Electoral Divisions. These were designed to raise approximately the same amount of Poor Law tax. DEDs would slice through civil parishes and therefore one civil parish could be in many DEDs.
Additional information was included in the Download so that the Latitude and Longitude plus a URL so that the information needed by the Gazetteer could be identified.
There are no District Enumeration Division numbers in the Irish census so an artificial number has been generated. I have simply used a Chapman Code and a number. Any structure can be adopted for a final PARMS file.
Structure of the Spreadsheet:
•	Column  A – Enumeration District
•	Column B – County
•	Column C - Townland Name from Irish Townlands database
•	Column D – Townlands from the Irish Townlands Database. (Note only Townlands were downloaded. Streets are not available in that database).
•	Column E – Townlands and streets from the 1901 Irish census
•	Column F – Barony from the Irish Townlands Database
•	Column G – Civil Parish from the Irish Townlands Database
•	Column H to K – alternative names needed for the Gazetteer
•	Column L – URL for the Gazetteer
•	Column M to N Latitude and Longitude needed for the Gazetteer
Methodology –
The Townland Data was downloaded from Irish Townlands and then sorted by County, District Electoral Division and Townland in that order.
The census data was copied from each DED and pasted into the spreadsheet. It was then edited to match names for the same Townland on each row.
Streets could not be matched so were simply added to each appropriate DED.
Each DED was given an identifying piece number.
Conclusion
•	The information to create an Irish PARMS exists.
•	An additional field of DED needs to be created as a field in the PARMS file. This is the key to tracking pieces in the Irish Census.
•	The PARMS needs to be built from the opposite end to the way this spreadsheet was built (i.e. Match the Townland data to the census structure rather than the other way around). The PARMS structure of County, DED, Townland or Street exists in the Irish census menu. We could approach the Irish National Archives to see if they will give us a download of the structure for each of 1901 and 1911. The Townland download does not contain all the information that the census structure needs to follow it in a practical way.
•	The Townlands data for each DED needs to be matched to the census download. This will be a significant task as spellings of DEDs and Townlands vary between each database and will need to be manually reconciled. The research did not consider the Gaelic and alternative name spellings so some of the matching may already exist in the Townland data.
•	The census PARMS will break up Civil Parishes. We need to consider whether this will affect searching (I don’t believe this will matter if we are using Lat / Long searches instead of text searches).
•	The pieces consisting of Streets with no Townlands can be allocated a name such as ‘Belfast’ if needed.
•	The Irish County Coordinators will work from a piece built as a County, DED, Townland or Street. They cannot work from a structure built on Civil parishes. The structure that the Coordinator will use is in columns A, B, C and E.
Footnote: It should be noted that there may be errors in matching in this trial spreadsheet as it was done manually on a ‘looks like’ basis. The purpose was to prove that a PARMS similar to the England PARMS could be created.
Geoff
Note: I tried to attach a copy of the spreadsheet but it has not been accepted. Geoff

@AnneV-Learn
I do not have a copy of the FreeCEN2 table structures and fields included within a table. I understand that we have a networked type of structure rather than a traditional relational structure but do need to know where individual fields are. Are you able to send me the diagrams as they apply to the England and Wales PARMS please? I can then apply the principles above to a more detailed plan for designing and populating an Irish PARMS.
Thanks
Geoff

@geoffj-FUG I am not aware that such a diagram exists. Perhaps @Vino-S could comment and maybe ask Kirk (still available for advice of course 🙂). If no such documentation exists,  it would most likely be possible to construct something though.

Provided the following to @geoffj-FUG which was constructed using DbSchema
FreeCen2 Parms 24 Sep 2022.pdf

I have analysed the tables in what I believe to be the current England PARMS. Only small adjustments seem to be required to create a structure for the Ireland PARMS. My analysis can be accessed at
https://docs.google.com/document/d/1A8B7HSWfHkcYHPoKjhnfBJDVGYufjCzJhfOEDez0HkE/edit?usp=sharing
Geoff

@PatReynoldsFUG @AnneV-Learn  I have been looking at the issue of civil parishes and using the streets in Belfast as the test data.
The civil parish in Ireland was the Church of England Parish. There was also a Catholic Parish and Diocese. For Antrim see:
https://www.familysearch.org/en/wiki/County_Antrim_Civil_Parishes
We need a policy decision as to how we are going to treat this information. It could become political in the longer term.
Geoff

@PatReynoldsFUG @AnneV-Learn I have been looking at the Baronies in the Irish PARMS.
There appears to be a one to one relationship between the townland an the barony (One townland has one barony, one Barony has many townlands). It is therefore possible to record it within the Townland collection.
Having said that there is bound to be an exception somewhere (there always is!). So I would define the Barony as an array and use this to test the transcription.
Geoff

@PatReynoldsFUG @AnneV-Learn
I am thinking aloud with this story. I have updated my document on Google Docs. My next step will be to create some sample data sets based on my thoughts so far.
As I think out each step please feel free to give me feedback, especially if you see any problems in my thoughts.
Geoff


Hibernate Temporary table

Hello,
Thanks for the project.
Unfortunataly, I'm having a problem with hibernate temp table.
I use @DatabaseSetUp to load a unique "SITE" table with unique line.
My unit test update this ligne.
Then I'm checking the data with @ExpectedDatabase and a file containing 1 table and 1 row.
The problem is when dbunit check how many tables are present against how many should be, all hibernate temporary table are present but not in my expected data file.
Is there a way to ignore those temporary tables?
Thans a lot.
Maxime

You should use a non-strict assertion mode - it will only check the tables in your dataset, like so:
@ExpectedDatabase(value = "/path/to/dataset.xml", assertionMode = DatabaseAssertionMode.NON_STRICT)


CAMISIM: Estimation of endophytes ratios

Find some articles describing the found endophyte ratios.
Otherwise, go with 70-80% fungi, 20-30% bacteria. Then dilute these for the abundance of P. zijinensis

Add references about endophyte ratios

"After quality control and chloroplast or mitochondrial sequences removal, a total of 530,608 bacterial sequences and 2,442,189 fungal sequences were obtained from 24 samples. These sequences were assigned to 636 bacterial OTUs and 718 fungal OTUs, respectively." - Lin et al., 2022, Distribution pattern of endophytic bacteria and fungi in tea plants.

"The remaining phyla viz., Cyanobacteria (blue green algae), Ascomycota and Basidiomycota (fungi) were less prominent in the metagenomes of GPU and UM" - Prasannakumar et al., 2020, Metagenome sequencing of fingermillet-associated microbial consortia provides insights into structural and functional diversity of endophytes.

"a total of 90.9 million reads from 16S and 86.8 million reads from ITS were clustered at ≥97% sequence identity using UPARSE. After processing, a total of 23,811 bacterial and 11,727 fungal operational taxonomy units (OTUs) were recovered." - Soares Correa de Souza et al., 2016, Unlocking the bacterial and fungal communities assemblages of sugarcane microbiome

Going with 70-80% Fungi, 20-30% Bacteria/Archaea and 0.5-2% Plasmids/Circular DNA/Virus


Support Hive OpenCSVSerde

@sansanichfb based on team's suggestion, I've removed the explicit serde check from HiveLineBreakAccessor as it supports multiple custom serdes.
We can eventually do away with the strict serde class check upon further investigation with the other Accessors.


Wrong start date on Android

Hey @christopherdro ,
I have tried to set a start date(2016-09-10T03:00:00.000Z) and it work very well on iOS, however on Android it changes the date that is visible on calendar showing 2 months ago.
<Calendar
          scrollEnabled={true}
          showControls={true}
          titleFormat={'MMMM'}
          dayHeadings={dayLabels}
          monthNames={monthLabels}
          prevButtonText={<Icon name="ios-arrow-back-outline" color="white" size={30} />}
          nextButtonText={<Icon name="ios-arrow-forward-outline" color="white" size={30} />}
          onDateSelect={(date) => handleDateSelection(date)}
          onTouchPrev={this.onTouchPrev}
          onTouchNext={this.onTouchNext}
          onSwipePrev={this.onSwipePrev}
          onSwipeNext={this.onSwipeNext}
          eventDates={eventDates}
          startDate={startDate}
          customStyle={calendarStyle}
          weekStart={0}
        />

Just merged this that might fix the issue.
https://github.com/christopherdro/react-native-calendar/pull/68
Can you give it a try?

Still does not work

I am also facing the same issue... Any solution?

I have notice that if I click on any day it get the right date

Hi @christopherdro  it's working!
    "react-native-calendar": "christopherdro/react-native-calendar#master",

We just have to use the master branch because it isn't released yet

Nice. I'll do a new release this weekend.

https://github.com/christopherdro/react-native-calendar/releases/tag/v0.9.0


Publish feature

Tumblr
Wordpress

I'd appreciate this very much. Any way we can integrate this with the WordPress API to create a new draft post with the title as the file name and body as the body? I'll look into this later on. Or if WordPress exposes an Intent Extra to create a post, that's an added benefit!


Browser().visit(url),the url have to include  ":"

path="www.baidu.com"
Browser().visit(path)
Traceback (most recent call last):
File "", line 1, in
File "c:\users\administrator\appdata\local\enthought\can
ages\splinter-0.7.0-py2.7.egg\splinter\driver\webdriver__
in visit
self.connect(url)
File "c:\users\administrator\appdata\local\enthought\can
ages\splinter-0.7.0-py2.7.egg\splinter\request_handler\req
23, in connect
self._create_connection()
File "c:\users\administrator\appdata\local\enthought\can
ages\splinter-0.7.0-py2.7.egg\splinter\request_handler\req
48, in _create_connection
self.conn = http_client.HTTPConnection(self.host, self
File "D:\Enthought\Canopy\App\appdata\canopy-<IP_ADDRESS>30.
.py", line 704, in init
self._set_hostport(host, port)
File "D:\Enthought\Canopy\App\appdata\canopy-<IP_ADDRESS>30.
.py", line 723, in _set_hostport
i = host.rfind(':')
AttributeError: 'NoneType' object has no attribute 'rfind'

you should include the protocol "http://" in your path
path = "http://www.baidu.com"


Add Acceptance testing framework

base testing framework for BDD (cucumber) is done, however it has to be run manually and there are only a handful of tests. I will create a new story for the tests, and the integration of the test framework in to a CI/CD process (most likely headless on Jenkins)


ArmnnConverter & ExecuteNetwork Usage

Hello
I am trying to execute ExecuteNetwork and check the output with tf model and converted tf model to armnn format model
1)I used ArmnnConverter tool to convert a tf model to armnn format.
Command used in adb shell:
#./ArmnnConverter -f tensorflow-binary -m mobilenet_v1_1.0_224_frozen.pb -i input -o MobilenetV1/Predictions/Reshape_1 -p output -s 1,224,224,3
Input model :           17MB    mobilenet_v1_1.0_224_frozen.pb
Output generated : 16MB     output
3)Run Execute Network:
command used in adb shell:
#./ExecuteNetwork -f tensorflow-binary -m output  -i input -o MobilenetV1/Predictions/Reshape_1 -d your_file.txt  -s 1,224,224,3
Runs fine, prints tensors
#./ExecuteNetwork -f armnn-binary -m output  -i input -o MobilenetV1/Predictions/Reshape_1 -d your_file.txt  -s 1,224,224,3
Error occurred:
flatbuffers/include/flatbuffers/flatbuffers.h:244: flatbuffers::Vector::return_type flatbuffers::Vector<flatbuffers::OffsetarmnnSerializer::AnyLayer >::Get(flatbuffers::uoffset_t) const [T = flatbuffers::OffsetarmnnSerializer::AnyLayer]: assertion "i < size()" failed
Aborted
The input "your_file.txt" is a flat array as mentioned in the help. It gives output when run with tf model and fails when executed with converted armnn format model.
On debugging it's found that layer_index received from flatbuffer comes to be 65 which is equal to size(), thus it fails.
for (unsigned int i=0; i<numInputs; ++i)    {        auto inputId = CHECKED_NON_NEGATIVE(static_cast<int32_t>                                           (layer->inputSlots()->Get(i)->connection()->sourceLayerIndex()));        std::cout<<"inputId = "<<inputId<<"\n";        result[i] = GetBaseLayer(graphPtr, inputId)->outputSlots()->Get(0)->tensorInfo();    }
return_type Get(uoffset_t i) const {      printf("i = %lu size = %lu\n",i,size());     FLATBUFFERS_ASSERT(i < size());     return IndirectHelper<T>::Read(Data(), i);}
Commands used to build:
`CXX=aarch64-linux-android-clang++  CC=aarch64-linux-android-clang  CXX_FLAGS="-fPIE -fPIC"  cmake ..   -DCMAKE_SYSTEM_NAME=Android   -DCMAKE_ANDROID_ARCH_ABI=arm64-v8a   -DCMAKE_ANDROID_STANDALONE_TOOLCHAIN=$ARMNNDEV/armnn-devenv/toolchains/aarch64-android-r17b/   -DCMAKE_EXE_LINKER_FLAGS="-pie -llog"   -DARMCOMPUTE_ROOT=$ARMNNDEV/armnn-devenv/ComputeLibrary/   -DARMCOMPUTE_BUILD_DIR=$ARMNNDEV/armnn-devenv/ComputeLibrary/build   -DBOOST_ROOT=$ARMNNDEV/armnn-devenv/boost   -DBOOST_LIBRARYDIR=$ARMNNDEV/armnn-devenv/boost/install/lib   -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1   -DTF_GENERATED_SOURCES=$ARMNNDEV/armnn-devenv/google/tf_pb/   -DPROTOBUF_ROOT=$ARMNNDEV/armnn-devenv/google/arm64_pb_install/   -DBUILD_TF_PARSER=1 -DBUILD_ARMNN_SERIALIZER=1   -DFLATBUFFERS_INCLUDE_PATH=$ARMNNDEV/armnn-devenv/flatbuffers/include   -DFLATBUFFERS_LIBRARY=$ARMNNDEV/armnn-devenv/flatbuffers/libflatbuffers.a   -DFLATC_DIR=$ARMNNDEV/armnn-devenv/flatbuffers  -DBUILD_TESTS=1
make -j16`
PS:your_file.txt looks something like this
0.681499082012 0.499423481863 0.699091174772 0.347890541798 0.893059256075 0.317762455033 0.238925760309 0.746247963743 0.35862938172 0.185409617886 0.0963199513358 0.583821514589 0.715978418217 0.991755379755 0.105288722455 0.318981731734 0.335481982383 0.706489821327 0.255523476279 0.136378250727 0.502332637656 0.580274818251 0.677454173864 0.548721022376 0.634912211285 0.335890986317 0.617197312224 0.145951839611 0.355783924735 0.659216082256 0.869394206999 0.368164179929 0.0682292530453 0.503974507163 0.954484705829 0.168305336361 0.740921782432 0.819585411381 0.21105322593 0.412600188305 0.632743876906 0.450042617085 0.332793748337 0.642172075798 0.564949086776 0.608918564187 0.726845569615 0.260
Please let me know what is going wrong? Didn't convert model properly, not executing Execute Network properly...

Sorry to hear you're having trouble. That does look like there might be a bug. We'll try to reproduce internally and figure out what is going wrong.

Thanks @MatthewARM .
Do we have a sample  model and a converted model in the repo for testing?

Hi @AnPay there isn't a model in the repo. We're testing with a variety of models downloaded from the internet. Because those models are big, and the copyright doesn't belong to us we don't put them in the armnn repo.
It's a good suggestion to have some simple test cases checked in though, I'll raise that internally.
For testing with tensorflow we are using inception v3 from Google's hosted models repository:
ArmnnConverter -f tensorflow-binary -m /devenv/models/inception_v3_2016_08_28_frozen.pb -i input -s 1,299,299,3 -o InceptionV3/Predictions/Reshape_1 -p jenkins/Resources/convertedFilesDir/inception_v3_2016_08_28_frozen.armnn
ExecuteNetwork -f tensorflow-binary -m /devenv/models/inception_v3_2016_08_28_frozen.pb -i input -o InceptionV3/Predictions/Reshape_1 -d jenkins/Resources/Dog_299x299.snpy -s 1,299,299,3 -c CpuRef
ExecuteNetwork -f armnn-binary -m jenkins/Resources/convertedFilesDir/inception_v3_2016_08_28_frozen.armnn -i input -o InceptionV3/Predictions/Reshape_1 -d jenkins/Resources/Dog_299x299.snpy -s 1,299,299,3 -c CpuRef
Then we compare the output of ExecuteNetwork to make sure that the output tensor is identical.
We're continuing to investigate why you might be hitting the FLATBUFFERS_ASSERT thing. There's quite a few Github users who are seeing the same problem now
Hope that helps,
Matthew

Thanks a lot @MatthewARM for such detailed response
But I am afraid I am still facing the issue.

inputId = 1527
GetBaseLayer  ARMNN/armnn-devenv/armnn/src/armnnDeserializer/Deserializer.cpp
layerIndex = 1527
Get
i = 1527 size = 1528
layerType = 9 layer_index= 1527
layer = output layerIndex = 1527
Get
i = 1527 size = 1528
Get
i = 0 size = 0
ARMNN/armnn-devenv/flatbuffers/include/flatbuffers/flatbuffers.h:244: flatbuffers::Vector::return_type flatbuffers::Vector<flatbuffers::OffsetarmnnSerializer::OutputSlot >::Get(flatbuffers::uoffset_t) const [T = flatbuffers::OffsetarmnnSerializer::OutputSlot]: assertion "i < size()" failed
Aborted

I did exactly what u did.
Just to be on the same page, i am running these commands on the device.
Could there be an issue with libflatbuffers.a?

Hi @MatthewARM
Could run it successfully now. But not on the released code.  I pulled the latest code from mlplatform(where caffe parser is also supported).
Commands:

CXX_FLAGS="-fPIE -fPIC"  cmake ..   -DCMAKE_SYSTEM_NAME=Android   -DCMAKE_ANDROID_ARCH_ABI=arm64-v8a   -DCMAKE_ANDROID_STANDALONE_TOOLCHAIN=$ARMNNDEV/armnn-devenv/toolchains/aarch64-android-r17b/   -DCMAKE_EXE_LINKER_FLAGS="-pie -llog"   -DARMCOMPUTE_ROOT=$ARMNNDEV/armnn-devenv/ComputeLibrary/   -DARMCOMPUTE_BUILD_DIR=$ARMNNDEV/armnn-devenv/ComputeLibrary/build   -DBOOST_ROOT=$ARMNNDEV/armnn-devenv/boost   -DBOOST_LIBRARYDIR=$ARMNNDEV/armnn-devenv/boost/install/lib   -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1   -DTF_GENERATED_SOURCES=$ARMNNDEV/armnn-devenv/google/tf_pb/   -DPROTOBUF_ROOT=$ARMNNDEV/armnn-devenv/google/arm64_pb_install/   -DBUILD_TF_PARSER=1 -DBUILD_ARMNN_SERIALIZER=1   -DFLATBUFFERS_INCLUDE_PATH=$ARMNNDEV/armnn-devenv/flatbuffers/include   -DFLATBUFFERS_LIBRARY=$ARMNNDEV/armnn-devenv/flatbuffers/libflatbuffers.a   -DFLATC_DIR=$ARMNNDEV/armnn-devenv/flatbuffers  -DBUILD_CAFFE_PARSER=1

make -j32

Could execute inception and mobilenet models(tf->armnn converted models).
Couldn't figure out the reason as to why is it not working on released code base.
PS: Still struggling with caffe parser ;)

terminating with uncaught exception of type armnn::ParseException: Data blob at index 0 in layer conv1/mod has an unexpected size. Expected 864 elements but got 0 elements.  at function GetDataFromBlob [/abfdata/payal.anand/ARMNN/armnn-devenv/armnn/src/armnnCaffeParser/CaffeParser.cpp:116]
Aborted

Thanks @AnPay I guess there must be a bug in the release which has "gone away" in the master branch. I'll see if we can find out what it was, and maybe we'll need to patch the release.

Facing the same error while trying to execute mobilenetv1.caffemodel..
Armnn Error: Data blob at index 0 in layer conv1/mod has an unexpected size. Expected 864 elements but got 0 elements.  at function GetDataFromBlob [ARMNN/armnn-devenv/armnn/src/armnnCaffeParser/CaffeParser.cpp:120]
Any leads? @MatthewARM

Thanks @Azutoz can you provide a download link for the model?

Closing this as the issue reported by @AnPay has been resolved (somehow...)
@Azutoz please create a new issue if you are still seeing a problem with the Caffe parser. Many thanks.
Matthew


Export to HTML fails

Environment data

VS Code version: 1.64.2
Jupyter Extension version: v2022.1.1201831736
Python Extension version: v2022.0.1814523869
OS and version: Windows 10, 10.0.19043, Build 19043
Python: 3.10.2
nbconvert: 6.4.2
Jupyter server running: Local

Expected behaviour
Export to HTML of .ipynb notebook to create a .html file
Actual behaviour
Export failure
Steps to reproduce:
[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Choose any .ipynb file
Click Export
Choose HTML

Logs
Info 13:39:08: Process Execution: > ~\AppData\Local\Programs\Python\Python310\python.exe -m jupyter nbconvert c:\Users\username\AppData\Local\Temp\88e203ff-653a-47c7-b68b-fc9c26da5fa9\Ses3.ipynb --to html --output tmp-21320bRBXXdezrOKt.html --output-dir ~\AppData\Local\Temp --debug
> ~\AppData\Local\Programs\Python\Python310\python.exe -m jupyter nbconvert c:\Users\username\AppData\Local\Temp\88e203ff-653a-47c7-b68b-fc9c26da5fa9\Ses3.ipynb --to html --output tmp-21320bRBXXdezrOKt.html --output-dir ~\AppData\Local\Temp --debug
Error 13:39:08: Export failed [Error: usage: jupyter.py [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]
                  [--paths] [--json] [--debug]
                  [subcommand]

Jupyter: Interactive Computing

positional arguments:
  subcommand     the subcommand to launch

options:
  -h, --help     show this help message and exit
  --version      show the versions of core jupyter packages and exit
  --config-dir   show Jupyter config dir
  --data-dir     show Jupyter data dir
  --runtime-dir  show Jupyter runtime dir
  --paths        show all Jupyter paths. Add --json for machine-readable
                 format.
  --json         output paths as machine-readable json
  --debug        output debug information about paths

Available subcommands:

Jupyter command `jupyter-nbconvert` not found.

	at a.executeCommand (c:\Users\username\.vscode\extensions\ms-toolsai.jupyter-2022.1.1201831736\out\client\extension.js:24:110265)
	at processTicksAndRejections (internal/process/task_queues.js:93:5)
	at async a.export (c:\Users\username\.vscode\extensions\ms-toolsai.jupyter-2022.1.1201831736\out\client\extension.js:66:1162036)
	at async _.exportToFormat (c:\Users\username\.vscode\extensions\ms-toolsai.jupyter-2022.1.1201831736\out\client\extension.js:66:1160286)
	at async _.performNbConvertExport (c:\Users\username\.vscode\extensions\ms-toolsai.jupyter-2022.1.1201831736\out\client\extension.js:66:1159502)
	at async _.performExport (c:\Users\username\.vscode\extensions\ms-toolsai.jupyter-2022.1.1201831736\out\client\extension.js:66:1159127)
	at async _.exportImpl (c:\Users\username\.vscode\extensions\ms-toolsai.jupyter-2022.1.1201831736\out\client\extension.js:66:1158576)
	at async _.export (c:\Users\username\.vscode\extensions\ms-toolsai.jupyter-2022.1.1201831736\out\client\extension.js:66:1158309)
	at async b.export (c:\Users\username\.vscode\extensions\ms-toolsai.jupyter-2022.1.1201831736\out\client\extension.js:51:588177)
	at async u._executeContributedCommand (c:\Users\username\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:93:30224)]

I'm getting similar behavior with Export to PDF, with MiKTeX installed and up-to-date, including the Jupyter command jupyter-nbconvert not found. followed by a missing TeX error:
Error 14:15:28: If you have not installed xelatex (TeX) you will need to do so before you can export to PDF, for further instructions go to https://nbconvert.readthedocs.io/en/latest/install.html#installing-tex.
To avoid installing xelatex (TeX) you might want to try exporting to HTML and using your browsers "Print to PDF" feature.

Confirmed, and can replicate; am also experiencing this issue in the latest Insiders' build.

I can't get this to repro, but it looks like we we're not able to ensure that nbconvert was installed. Can you try and manually install nbconvert in your python environment and see if this still occurs?
Also, just to confirm, what do you have for the Jupyter: Python Export Method settings?

possibly related (but should be fixed in latest insiders): https://github.com/microsoft/vscode-jupyter/issues/6838

I can't get this to repro, but it looks like we we're not able to ensure that nbconvert was installed. Can you try and manually install nbconvert in your python environment and see if this still occurs?

After performing pip uninstall nbconvert and then pip install nbconvert the export works as expected.

Also, just to confirm, what do you have for the Jupyter: Python Export Method settings?

I had Jupyter: Python Export Method set to the default direct. I tried switching to nbconvert and encountered the same error. After I manually reinstalled nbconvert then the export worked as expected with both settings.

@dynamicwebpaige - can you check if nbconvert is currently installed in your python environment as well? And also verify that reinstalling resolves the issue?

closing for now as the workaround seems to resolve the issue and we haven't heard of any more issues with it


feat(Normed/Field): completeSpace_iff_isComplete_closedBall

On the way to locally complete K iff complete O(K)
Also add some helper lemmas about uniform continuity of multiplication by a constant with a variant for rings (via smul) instead of UniformGroup (which fields are not).

maintainer merge

bors merge


[Feature request] Allow hide urls from the plot

Thanks for the great framework, it helps a lot. But I need to hide the URLs from the plot so I can focus on the images. Cab we add that feature?

Hi @BrikerMan
It makes perfect sense to me. I'll put this in the backlog - thanks!

Hi @karolzak, is there any progress on this topic or is there a quick workaround?
It is a nice package for convenience, but the url are making the output quite messy in my case.

Hey @Maarten-vr
Was busy with other stuff lately so this feature got buried and forgotten but since it's a minor enhancement I will try to add it before the winter holidays

I've dug into the code myself as well and made it work.
I've created a PR so you can use the code (either directly, or improve upon it)

@Maarten-vr that's a rockstar move!
Thank you so much for this contribution! I will review it and merge ASAP!

Thanks @BrikerMan for this suggestion and thank you @Maarten-vr for implementing it!
This feature is now available in version 1.1.1


mathPlugin adds .html attribute to node, but seems to be ignored in mystToHast

Description
Hi team!
I'm working on integrating myst into an existing web application using myst-parser, myst-transforms, and myst-to-html
I eventually want to use a pipeline like this
    const pipeline = unified()
      .use(mystParser)
      .use(mathPlugin)
      .use(transform, new State())
      .use(mystToHast)
      .use(formatHtml)
      .use(rehypeStringify);

However, when I do that any inline math or display math that I use ends up not being KaTeXified in the resulting html.
Here are some screenshots that demonstrate this with the sandbox here: https://mystmd.org/sandbox
First the source and DEMO tab of the sandbox. Notice the $\LaTeX in the source and that it is rendered properly on the right:

Then let's look at the AST before plugins (AST -> pre tab). Here we have an inlineMath node with value: \LaTeX

Then if we look at the AST after plugins (AST -> post tab) we have a new html attribute on that node. This html attribute contains the katex output and is what I want in my output:

Finally, if we look at the HTML tab we see that we don't get that nice katex output, but rather a bland ... element

I've done some digging and can see that the .html attribute is added in teh mathPlugin from myst-transforms  here: https://github.com/executablebooks/mystmd/blob/2be01f12907dacc37c326949c2f2e7b8e95a1701/packages/myst-transforms/src/math.ts#L191
I also looked into myst-to-html and found that it is (1) not checking for the .html attribute on any of the nodes and (2) responsible for adding the ... element here: https://github.com/executablebooks/mystmd/blob/2be01f12907dacc37c326949c2f2e7b8e95a1701/packages/myst-to-html/src/renderer.ts#L14-L29
In my digging I ran the following in a node console session:
> txt = "This is an equation: \$V(a) = \\max_{a'} u(a' - r a) + \\beta E[V(a')]\$"
"This is an equation: $V(a) = \\max_{a'} u(a' - r a) + \\beta E[V(a')]$"
> mystast = unified().use(mystParser).parse(txt)
{
  type: 'root',
  children: [ { type: 'paragraph', position: [Object], children: [Array] } ]
}
> mystast.children[0].children[1]
{
  type: 'inlineMath',
  value: "V(a) = \\max_{a'} u(a' - r a) + \\beta E[V(a')]",
  position: { start: { line: 1, column: 1 }, end: { line: 1, column: 1 } }
}
> mystast2 = unified().use(mathPlugin).runSync(mystast)
{
  type: 'root',
  children: [ { type: 'paragraph', position: [Object], children: [Array] } ]
}
> mystast2.children[0].children[1]
{
  type: 'inlineMath',
  value: "V(a) = \\max_{a'} u(a' - r a) + \\beta E[V(a')]",
  position: { start: { line: 1, column: 1 }, end: { line: 1, column: 1 } },
  html: '<math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mo stretchy="false">(</mo><mi>a</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mrow><mi>max</mi><mo></mo></mrow><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mi>u</mi><mo stretchy="false">(</mo><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>−</mo><mi>r</mi><mi>a</mi><mo stretchy="false">)</mo><mo>+</mo><mi>β</mi><mi>E</mi><mo stretchy="false">[</mo><mi>V</mi><mo stretchy="false">(</mo><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">V(a) = \\max_{a'} u(a' - r a) + \\beta E[V(a')]</annotation></semantics></math>V(a)=maxa′u(a′−ra)+βE[V(a′)]'
}
> hast = unified().use(mystToHast).runSync(mystast2)
{
  type: 'root',
  children: [
    {
      type: 'element',
      tagName: 'p',
      properties: {},
      children: [Array],
      position: [Object]
    }
  ]
}
> hast.children[0].children[1]
{
  type: 'element',
  tagName: 'span',
  properties: { class: 'math inline' },
  children: [
    {
      type: 'text',
      value: "V(a) = \\max_{a'} u(a' - r a) + \\beta E[V(a')]"
    }
  ],
  position: {
    start: { line: 1, column: 1, offset: null },
    end: { line: 1, column: 1, offset: null }
  }
}

Note that when we first get mystast we have an inlineMath node. Then after running mathPlugin we get the .html method, but then after running mystToHast we lost that info and are left with the single span

I think that the issue might be fixed by modifying the math and inlineMath handlers here: https://github.com/executablebooks/mystmd/blob/2be01f12907dacc37c326949c2f2e7b8e95a1701/packages/myst-to-html/src/schema.ts#L58-L71
I'm thinking we could modify them to check for the presence of an html attribute on node. I'm not sure how to cram raw html into the ast though...

The rest of the MyST team have a better idea of what should happen here w.r.t myst-to-html.
But, you can also choose to render the math client-side, e.g. https://github.com/executablebooks/mystmd/issues/824

Wow fast response here!
Thanks for the example.
I think the issue illustrated above highlights problems with either how the mathPlugin from myst-transforms works (by adding an .html attribute to the node) or how myst-to-html works (by ignoring the presence of the .html attribute)

@sglyon my point is rather that the mathPlugin itself just performs server-side rendering of the math. Obviously, in your case, "server" and "client" are the same thing, so the distinction is less obvious.
I'm deliberately not speaking to whether the HTML generated by the math transform should end up in the HTML rendered by myst-to-html; that's something that @rowanc1 can perhaps answer. But, you could also choose to create your own simple plugin that just engages KaTeX on the frontend rather than using SSR, e.g. https://github.com/executablebooks/mystmd/issues/824#issuecomment-1883058035

I see what you are saying. I'm not using react or next or anything like that. I'm in the old school (but becoming new again?? ;))  world of server-based web apps with bits of js in the client to provide interactive features. So, my use of anything in the myst ecosystem is running entirely in the browser. Because of this I've already invoked katex in the browser (by running the mathPlugin), but that work is being thrown away.
Perhaps @rowanc1 could help validate if the intent use of the mathPlugin should enable handle rendering math in any of the supported outputs (e.g. html, typts, jats, etc.) or if users need to support math rendering on their own as in your example.
I'm guessing/hoping it is the former and that there is just a disconnect in how this is currently happening between mathPlugin and myst-to-html.

Hi @sglyon - thanks for the question, sorry for the mismatch between these two plugins.
The intention of this when I was working through the basic HTML rendering was to leave the latex directly in a math node, rather than convert it to HTML there directly, this is similar to how MathJax and KaTeX generally interact with the DOM, and then they would get triggered after the render.
parse --> stringify -> trigger mathjax/katex as "normal"
The last step would be something similar to:
function onRender() {
  const elements = document.querySelectorAll('math');
  for (element in elements) {
    // check if it is display mode or not on the `inline` class on the element, and also pass that into the next render function
    // using katex or mathjax
    katex.render(element.textContent, element);
  }
}

This seemed to be easier to integrate with plain html when I was thinking through it, but that might not be the case given the questions we have had around it recently. This is also how JupyterLab works.
On the server side implementation, we are pre-rendering the math because the client is very light, and doesn't have a katex/mathjax javascript dependency, so we just serve up the HTML directly and integrate the stylesheet.
Let us know if the potential solution above makes sense and/or works for you, if that is the case, then you wouldn't have to run the mathPlugin transform at all. Otherwise we can talk through how to change it -- maybe at the least renaming or adding documentation to that mathPlugin to say this..!

OK thanks for clarifying @rowanc1
Would it be possible to adjust the css class names that get applied to math nodes inside myst-to-html to match the ones used in remark-math?
Right now myst-to-html is using math block for display math and math inline for inline math.
remark-math expects math-display for display math and math-inline for inline math.  (ref https://github.com/remarkjs/remark-math/blob/e99b9d088709d743adf6a43551fd416d7e0014ed/packages/rehype-katex/lib/index.js)
If we make this change then we can use .use(mystToHast).use(rehypeKatex) to get math rendered entirely within the unified pipeline.

That sounds like a really good change to make, thanks for looking into that. Happy to take a pr on this if you are game!


Introduce strong parameters

We need to upgrade Devise to 3.2 and Devise Invitable to 1.4, which assumes that we're using Rails 4, or at least strong params. This PR is a preliminary step to doing the full Devise upgrade. It also helps to get us nearer to Rails 4, which we should upgrade to soon in any case.
Lots of these changes relate to specific behaviour of Devise and its extensions - look at individual commit messages for more explanation of those.
We've relied on existing test coverage extensively for these changes - the coverage looks pretty good so we're fairly confident that we haven't broken anything, but should keep this on Preview for a few days before deploying to Production to be sure.
We'd appreciate a thorough review of these changes: both @mikejustdoit and I are new to Devise and (most of) Signon so it's taken us lots of archaeology to get this far.

:+1: Good work

this is good stuff! :cake:
only suggestion i have is to move the UserParameterSanitiser usages out from the controller, into Policy classes, and use the permitted_attributes Pundit helper instead. this will be a cosmetic change, and i would prefer to do that as a separate PR sometime later when most of the FIXMEs and TODOs are taken care of.


[Snyk] Fix for 1 vulnerabilities

This PR was automatically created by Snyk using the credentials of a real user.![snyk-top-banner](https://github.com/andygongea/OWASP-Benchmark/assets/818805/c518c423-16fe-447e-b67f-ad5a49b5d123)
Snyk has created this PR to fix 1 vulnerabilities in the npm dependencies of this project.
Snyk changed the following file(s):

packages/pwa-kit-dev/package.json

Vulnerabilities that will be fixed with an upgrade:

Issue
Score

Denial of Service (DoS) SNYK-JS-WS-7266574
  768

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Denial of Service (DoS)

Duplicate #1841


TypeError exception with desired properties message

Hi,
I just added a desired (writable) property to my device template in IOT Central and then updated the property value on the device twin. Now my node-red crashes on start-up with the following log entries. I hope you can help.
Regards,
Greg
23 Aug 11:02:32 - [info] [azureiotdevice:ae934fbf.de102] 2bbh5tif9v2 -> Desired properties received: {"sizerHost":"host.docker.internal","$version":2}
23 Aug 11:02:32 - [red] Uncaught Exception:
23 Aug 11:02:32 - TypeError: Cannot read property 'toString' of undefined
at MqttTwinClient._sendTwinRequest (/data/node_modules/azure-iot-device-mqtt/dist/mqtt_twin_client.js:201:40)
at /data/node_modules/azure-iot-device-mqtt/dist/mqtt_twin_client.js:157:23
at BehavioralFsm.subscribe (/data/node_modules/azure-iot-device-mqtt/dist/mqtt_twin_client.js:92:25)
at BehavioralFsm.handle (/data/node_modules/machina/lib/machina.js:613:25)
at BehavioralFsm.subscribe (/data/node_modules/azure-iot-device-mqtt/dist/mqtt_twin_client.js:115:22)
at MqttTwinClient.updateTwinReportedProperties (/data/node_modules/azure-iot-device-mqtt/dist/mqtt_twin_client.js:149:24)
at constructor.updateTwinReportedProperties (/data/node_modules/azure-iot-device-mqtt/dist/mqtt.js:383:105)
at constructor.handle (/data/node_modules/machina/lib/machina.js:613:25)
at constructor.Fsm.(anonymous function) [as handle] (/data/node_modules/machina/lib/machina.js:466:63)
at Mqtt.updateTwinReportedProperties (/data/node_modules/azure-iot-device-mqtt/dist/mqtt.js:671:19)
at /data/node_modules/azure-iot-device/dist/twin.js:115:30
at retryOperation (/data/node_modules/azure-iot-common/dist/retry_operation.js:36:13)
at RetryOperation.retry (/data/node_modules/azure-iot-common/dist/retry_operation.js:67:9)
at Twin._updateReportedProperties (/data/node_modules/azure-iot-device/dist/twin.js:113:17)
at Object.update (/data/node_modules/azure-iot-device/dist/twin.js:145:26)
at sendDeviceProperties (/data/node_modules/node-red-contrib-azure-iot-device/azureiotdevice/azureiotdevice.js:504:38)

@packnology do you have more details on the scenario, so I can replay? What is it that you setup in IoTC? Thanks!

Will be fixed in 0.2.4 version.


Generalize material info templates

based on #71
addresses #67 by extracting the templates to a JSON compatible description. The TypeScript Interface is:
interface ITemplate {
  label: string;
  groups: ITemplateGroup[];
}

interface ITemplateGroup {
  label: string | React.Element;
  fields: ITemplateField[];
}
interface ITemplateField {
  label: string | React.Element;
  property: string;
  format?: (v: any) => string | string; // the string has to be a valid d3-format qualifier
  suffix?: string | React.Element
}

e.g.
{
        label: 'Minimal',
        groups: [
            {
                label: 'Material Properties',
                fields: [
                    {
                        label: 'Derived materials',
                        property: 'degree'
                    },
                    {
                        label: 'Formation energy',
                        property: 'formationEnergy',
                        format: '.3f',
                        suffix: 'eV /atom'
                    },
                    {
                        label: 'Synthesis probability',
                        property: 'synthesisProbability',
                        format: '.1%'
                    }
                ]
            }
        ]
    }

This could be a nice change, but I don't want to lock ourselves into only rendering info templates that can be expressed with this JSON DSL. Is there a way to add an "escape hatch" so we can also use, say, an arbitrary component here instead?

sure. I don't know yet how flexible the dataset specification should be in the end. Whether just JSON or in JavaScript that is bundled with the application. re simpler templates: https://github.com/arclamp/materialnet/pull/72/commits/d078282c61e35f942ecc4111e823548cad02ba02 was a first step which just simplified the template definition

outdated will close this one


Improve the Footer

Describe the Feature you want to end
I would like to improve how the Footer looks and add some things to it.

Copyright Statement
made with love statement
all rights reserved statement

Screenshots (optional)

Additional context (optional )
after doing this, I am willing to contribute more to this project.
Please assign this issue to me @pranjay-poddar with the Hacktoberfest tag.

hello can i help you to improve your footer page

Sure, but the maintainer needs to assign us first.

Please add the Hacktoberfest label to the issue @pranjay-poddar

Hello @pranjay-poddar  I would like to improve your footer with
Copyright, made with love, all rights reserved, along with About us and follow us elements please assign me this issue.


⚠ Turnos 24x7 has degraded performance

In 79c0782, Turnos 24x7 (https://reservas-dev.alternativasinteligentes.com/) experienced degraded performance:

HTTP code: 200
Response time: 995 ms

Resolved: Turnos 24x7 performance has improved in b4c1874.


chatItems as a BTree or MutableOrderedDictionary

In the ChatDataSourceProtocol
public protocol ChatDataSourceProtocol: class {
    var hasMoreNext: Bool { get }
    var hasMorePrevious: Bool { get }
    var chatItems: [ChatItemProtocol] { get }
    weak var delegate: ChatDataSourceDelegateProtocol? { get set }

    func loadNext() // Should trigger chatDataSourceDidUpdate with UpdateType.Pagination
    func loadPrevious() // Should trigger chatDataSourceDidUpdate with UpdateType.Pagination
    func adjustNumberOfMessages(preferredMaxCount: Int?, focusPosition: Double, completion:((didAdjust: Bool)) -> Void) // If you want, implement message count contention for performance, otherwise just call completion(false)
}

We have chatItems in which each element is a UniqueIdentifiable. Maybe we should have this as a BTree.Map<T> or some sort of MutableOrderedDictionary<T>?
What do you guys think? This is a great library that might help: https://github.com/lorentey/BTree

What would be the benefit? Chatto already uses dictionaries to quickly calculate changes and access items - there's even public api of this representation: BaseChatViewController.chatItemCompanionCollection; but I don't see the benefit or making people's data sources more complicated.

Usually we don't mutate the BaseChatViewController though. We are to update from the dataSource.
For example if I wanted to iterate over the chatItems to find by it's uId, I'd have to iterate over the array. Sometimes I might want to remove messages by it's id.

Usually we don't mutate the BaseChatViewController though. We are to update from the dataSource.

BaseChatViewController is immutable by design for obvious reasons.

For example if I wanted to iterate over the chatItems to find by it's uId, I'd have to iterate over the array. Sometimes I might want to remove messages by it's id.

You're still free to use whatever data structure you want in your data source, but I don't think we should require anything more complicated than an array, just because Chatto doesn't need it. If anything, we should relax the requirement from Array to Sequence (so if you are actually using a BTree you don't have to pay the conversion to Array) - but that would require an associated type in ChatDataSourceProtocol and a generic BaseChatViewController

That sounds like a good fix, generics are great for customization. The Sequence type should certainly help.

We're talking about a micro-optimization here. Associated types and generics also make the code more difficult to use and understand - I don't think they're the right thing to use in this case.


how to build from source code?

thanks for your wonderful job firstly. I wanna know if the project can be built and how to do that?  I look forward to your reply as soon as possible.Thks

Check the README, there are instructions at the end

I checked the readme, however I am not familiar with the Qt creator build process, I was able to create a new project with the specified version. Can you provide detailed instructions on how to import the project files and actually build it.


Proposal: Add Customizable Log Handling for drf-api-logger

Hello,
I appreciate the work put into the drf-api-logger library, which has been incredibly helpful for logging API requests and responses in Django applications. I would like to propose an enhancement to make log handling more flexible and customizable, allowing users to specify their own logic for processing logs.
Current Behavior:
Currently, the InsertLogIntoDatabase class inserts logs directly into the database using the _insert_into_data_base method. This approach works well for many use cases, but it limits users who wish to send logs to external services, such as Seq or other logging solutions.
Proposed Enhancement:
Introduce a mechanism that allows users to provide a custom handler for processing logs. This could be achieved by defining a new setting in settings.py that specifies the path to a custom function or callable responsible for handling the logs. This would allow the default behavior of inserting logs into the database to remain unchanged while providing flexibility for users who wish to customize their log handling.
Implementation Details:

Custom Handler Setting:
Add a new setting in settings.py:
DRF_API_LOGGER_CUSTOM_HANDLER = None  # Default to None, or provide a path to a custom function

Modify InsertLogIntoDatabase:
Update the InsertLogIntoDatabase class to check for the presence of the custom handler setting and use it if defined. Here is a brief outline of the changes:
from importlib import import_module

class InsertLogIntoDatabase(Thread):
    # Existing initialization code...

    def __init__(self):
        # Existing setup...
        self.custom_handler = getattr(settings, 'DRF_API_LOGGER_CUSTOM_HANDLER', None)
        if self.custom_handler:
            self.custom_handler = self._import_custom_handler(self.custom_handler)

    def _import_custom_handler(self, handler_path):
        """
        Import the custom handler function from a given string path.
        """
        module_path, func_name = handler_path.rsplit('.', 1)
        module = import_module(module_path)
        return getattr(module, func_name)

    def _insert_into_data_base(self, bulk_item):
        """
        Insert the bulk item into the database or use a custom handler if defined.
        """
        if self.custom_handler:
            try:
                self.custom_handler(bulk_item)
            except Exception as e:
                print('DRF API LOGGER CUSTOM HANDLER EXCEPTION:', e)
        else:
            try:
                APILogsModel.objects.using(self.DRF_API_LOGGER_DEFAULT_DATABASE).bulk_create(bulk_item)
            except OperationalError:
                raise Exception("""
                DRF API LOGGER EXCEPTION
                Model does not exist.
                Did you forget to migrate?
                """)
            except Exception as e:
                print('DRF API LOGGER EXCEPTION:', e)

Example Custom Handler:
Users can create their own handler functions, for example, to send logs to Seq:
# my_logging_handlers.py

import logging

def send_logs_to_seq(bulk_item):
    logger = logging.getLogger(__name__)
    for log in bulk_item:
        logger.info('API Log',
            extra={
                'requested_at': log.requested_at,
                'response_time': log.response_time,
                'method': log.method,
                'path': log.path,
                'remote_addr': log.remote_addr,
                'status_code': log.status_code,
                'request_body': log.request_body,
                'response_body': log.response_body,
            }
        )

And then set it in settings.py:
DRF_API_LOGGER_CUSTOM_HANDLER = 'my_project.my_logging_handlers.send_logs_to_seq'

Benefits:

Flexibility: Allows users to redirect logs to Seq or any other external logging service.
Backward Compatibility: Maintains existing behavior by default, ensuring no disruption for current users.
Configurability: Enhances the library's adaptability to various logging requirements without modifying core functionality.

I believe this enhancement would make the drf-api-logger library even more valuable by providing greater flexibility in how logs are handled. Please let me know if you have any questions or need further clarification on this proposal.
Thank you for considering this enhancement!
Best regards,
[Mohammad]

That was excellent!👍🏻👍🏻
Great job.

How hard I worked to customize the DRF logging system.
Great Job


Refresh used submissions

@ mention of reviewers
@Didayolo
A brief description of the purpose of the changes contained in this PR.
Now when you upload a submission, your submissions count will update instantly
Issues this PR resolves

#1097 -> Refresh used submissions count

Checklist

[x] Code review by me
[x] Hand tested by me
[x] I'm proud of my work
[x] Code review by reviewer
[x] Hand tested by reviewer
[x] CircleCi tests are passing
[x] Ready to merge

I'm testing this on https://codabench-test.lri.fr and it does not seem to be working. I am using Firefox.


Specify the S3 bucket region

Hi,
While trying pinrepo (nice work btw), I had to modify a bit the requests sent to S3 as my bucket is not in the default region. Without those changes, each http request to pinrepo was getting a http 307 response from AWS.
For example I'm using eu-west-1 as the region.
Note that I couldn't test how it does react when the bucket is indeed in the default region.

@sbaogang @jparise I've changed the host used in proxy_pass and added a few words about the region variable, let me know your thoughts.

Looks good to me! Again, thanks a bunch for fixing this!


Unable to achieve the expected mIoU

Hello, congratulations on your excellent work!
I have a little problem when I use this code for semantic segmentation training on CITYSCAPES dataset. I only use a NVIDIA Tesla P40 graphics card, "IMS_PER_BATCH" is set to 3, and all other configurations and codes have not been changed.  I train 100000 iters( about 100 epoches), the loss has already reached 0.22 and no longer decreases. But why did the mIoU in the test set is only about 50.
May I ask, what could have caused this?

The net trained is "Seg.Layer16.SmallGate.Dynamic_A" without ImageNet pretrain.

import os.path as osp
from dl_lib.configs.segm_config import SemanticSegmentationConfig
_config_dict = dict(
MODEL=dict(
WEIGHTS="",
CAL_FLOPS=True,
BACKBONE=dict(
CELL_TYPE=['sep_conv_3x3', 'skip_connect'],
LAYER_NUM=16,
CELL_NUM_LIST=[2, 3, 4] + [4 for _ in range(13)],
INIT_CHANNEL=64,
MAX_STRIDE=32,
SEPT_STEM=True,
NORM="BN",
DROP_PROB=0.0,
),
GATE=dict(
GATE_ON=True,
GATE_INIT_BIAS=1.5,
SMALL_GATE=True,
),
SEM_SEG_HEAD=dict(
IN_FEATURES=['layer_0', 'layer_1', 'layer_2', 'layer_3'],
NUM_CLASSES=19,
IGNORE_VALUE=255,
# NORM="nnSyncBN",
NORM="BN",
LOSS_WEIGHT=1.0,
),
BUDGET=dict(
CONSTRAIN=True,
LOSS_WEIGHT=0.8,
LOSS_MU=0.1,
FLOPS_ALL=26300.0,
UNUPDATE_RATE=0.4,
WARM_UP=True,
),
),
DATASETS=dict(
TRAIN=("cityscapes_fine_sem_seg_train", ),
TEST=("cityscapes_fine_sem_seg_val", ),
),
SOLVER=dict(
LR_SCHEDULER=dict(
NAME="PolyLR",
POLY_POWER=0.9,
MAX_ITER=190000,
),
OPTIMIZER=dict(BASE_LR=0.05, ),
IMS_PER_BATCH=3,
CHECKPOINT_PERIOD=5000,
GRAD_CLIP=5.0,
),
INPUT=dict(
MIN_SIZE_TRAIN=(512, 768, 1024, 1280, 1536, 2048, ),
MIN_SIZE_TRAIN_SAMPLING="choice",
MAX_SIZE_TRAIN=4096,
MIN_SIZE_TEST=1024,
MAX_SIZE_TEST=2048,
FIX_SIZE_FOR_FLOPS=[768, 768],
# FIX_SIZE_FOR_FLOPS=[1024, 2048],
CROP_PAD=dict(SIZE=[768, 768], ),
),
OUTPUT_DIR=osp.join(
'./data/Outputs/model_logs/dl_lib_playground',
osp.split(osp.realpath(file))[0].split("playground/")[-1]),
)
class DynamicSemanticSegmentationConfig(SemanticSegmentationConfig):
def init(self):
super(DynamicSemanticSegmentationConfig, self).init()
self._register_configuration(_config_dict)
config = DynamicSemanticSegmentationConfig()

Thanks for your attention. We strongly recommend setting IMS_PER_BATCH to 8. Because much fewer images in each batch would harm the BatchNorm in nodes.

OKey, thank you very muck for your reply


Add ServerJre Java Option

Server JRE is suited for the environment which demand small footprint like docker, vagrant.

Sorry for not seeing this request before. Please raise this on our Gitter channel for discussion. If there is enough interest, we could look at introducing this.


fix readme and start code

The KeyError exception will occur at https://github.com/Alpha-VLLM/Lumina-T2X/blob/main/lumina_next_t2i/utils/cli.py#L150 because the train_args dictionary has the model named 'DiT_Llama_2B_patch2', while in model.py, the only model named 'NextDiT_2B_patch2'. To resolve this, I hardcoded line 150 to use the 'NextDiT_2B_patch2' model directly:
     model_dit = models.__dict__[train_args.model](
-------------------------------------------------------------
     model_dit = models.__dict__['NextDiT_2B_patch2'](

Thank you for your contribution, we will fix this problem.


Read files

Adds support for input files. The argument will be checked if it is a file and in that case, it will be read and used as input.
In addition, a default flag is set for the processors so that they can react differently to the input.
I noticed that the hash functions give different outputs when you pipe the input.
> echo "Hello World" | sttr md5
e59ff97941044f85df5297e1c302d260

> sttr "Hello World"
b10a8db164e0754105b7a99be72e3fe5

In the first case, a linefeed is passed with the echo command. I have implemented this so that if the input does not come from a file, the trailing line breaks are removed from the input.  I think for simple, single-line inputs, you usually don't want the newline passed into the hash function.

Oh, nice catch !!
so codacy is failing for exported function need to have a comment.
I'm going to merge this :)

@rverst even though I have merged it, I realized that there is an issue.
Please download the attached file
md.txt
Run
cat md.txt | sttr md5
// Output b10a8db164e0754105b7a99be72e3fe5
// Expected e59ff97941044f85df5297e1c302d260

Also I feel like this is expected behavior?
If you see md5 ( osx ) / md5sum ( osx )
echo Hello World | md5
// e59ff97941044f85df5297e1c302d260

echo Hello World | md5sum
// e59ff97941044f85df5297e1c302d260

@rverst even though I have merged it, I realized that there is an issue.

@abhimanyu003, I'll have a look at it. Might take a couple of days because I’m out of town for two days.