Analysis of expanding HCW by cadre

In this branch, we analyse the marginal and combined impact (in terms of DALYs, Deaths, Services by appointment types/treatment types) of expanding current HCW capabilities for five cadres - Clinical (C), DCSA (D), Nursing and Midwifery (N), Pharmacy (P), Other (O) - given an annual extra budget that is 4.2% of total cost of these cadres of the previous year. These main cadres are among the priorities cadres (HSSP III HRH Annex) and tlo model has fairly simulated their time usage (not sure about DCSA). The 4.2% growth rate is referred to the annual GDP growth rate in Malawi (Margherita 2024), which is 4.2%, and the assumption that proportion of GDP allocated for HRH costing is fixed.
Done:
(1) First-round 33 scenarios of extra budget allocation to the cadres are designed
(see https://github.com/UCL/TLOmodel/pull/1415/files#diff-af8994d6c7a9c0a52b2f5fb5aa35aeb92fe4f3966e50bbfcc927a21e9e75045d)
(2) First-round run settings (expansion years, mode change year, consumables availability, etc.) are set
(see src/scripts/healthsystem/impact_of_hcw_capabilities_expansion/scenario_of_expanding_current_hcw_by_officer_type_with_extra_budget.py)
(3) First-round run is finished and draf plots of results are made (see results at the bottom and more is in PyCharm project folder)
Todo:
(1) Analyse first-round results and see if any results need to be emphasised and further examined (such as representative scenarios/appt types/treatment types/cadres/causes, marginal and combined impacts of expanding cadres, relationship between DALYs averted/services or treatments provided/time capabilities of cadres)
(2) Start from the best scenario in first round, considering second-round scenarios and run design for finding better extra budget allocation scenarios if possible and sensitivity analysis

Confirm the scenarios before summit to Azure (C + P or C+NM+P, scale up ratios, time of change)

to do for scenario design: Use the GDP expenditure on HR and HR cost (salary) data to figure out given a % increase of HR expenditure, how to increase the staff of cadres Clinical, Nursing and Midwifery, Pharmacy and DCSA, both independently or in a combined way. The ways to increase staff will be the scenarios to be examined.

Hi @marghe-molaro and @tbhallett, I have created a hr expansion function in the healthsystem module, considering the fractions of extra budget to be used by the four cadres (clinical, dcsa, nursing_and_midwifery, pharmacy), updated the scenarios and scenario script, and added some relevant files as needed. I would be very grateful if you could help check if the function and logics  are looking fine. Any comment or advice would be greatly appreciated. Thanks.

Note: I have got two warnings when run the script locally. Not sure if it is related to changes in this branch. Will check.
C:\Users\jdbb1\PycharmProjects\TLOmodel\src\tlo\scenario.py:438: InconsistentLoggedColumnsWarning: Inconsistent columns in logged values for tlo.scenario logger with key override_parameter compared to header generated from initial log entry:
Columns in header not in logged values are
{'new_value': 'bool', 'old_value': 'bool'}
Columns in logged values not in header are
{'new_value': 'str', 'old_value': 'str'}
logger.info(
C:\Users\jdbb1\PycharmProjects\TLOmodel\src\tlo\scenario.py:438: InconsistentLoggedColumnsWarning: Inconsistent columns in logged values for tlo.scenario logger with key override_parameter compared to header generated from initial log entry:
o logger with key override_parameter compared to header generated from initial log entry:
Columns in header not in logged values are
{'new_value': 'bool', 'old_value': 'bool'}
Columns in logged values not in header are
{'new_value': 'int', 'old_value': 'int'}
logger.info(

Note: I have got warnings when run the script locally. Not sure if it is related to changes in this branch. Will check.
C:\Users\jdbb1\PycharmProjects\TLOmodel\src\tlo\scenario.py:438: InconsistentLoggedColumnsWarning: Inconsistent columns in logged values for tlo.scenario logger with key override_parameter compared to header generated from initial log entry: Columns in header not in logged values are {'new_value': 'bool', 'old_value': 'bool'} Columns in logged values not in header are {'new_value': 'list', 'old_value': 'list'} logger.info( C:\Users\jdbb1\PycharmProjects\TLOmodel\src\tlo\scenario.py:438: InconsistentLoggedColumnsWarning: Inconsistent columns in logged values for tlo.scenario logger with key override_parameter compared to header generated from initial log entry: Columns in header not in logged values are {'new_value': 'bool', 'old_value': 'bool'} Columns in logged values not in header are {'new_value': 'str', 'old_value': 'str'} logger.info( C:\Users\jdbb1\PycharmProjects\TLOmodel\src\tlo\scenario.py:438: InconsistentLoggedColumnsWarning: Inconsistent columns in logged values for tlo.scenario logger with key override_parameter compared to header generated from initial log entry: Columns in header not in logged values are {'new_value': 'bool', 'old_value': 'bool'} Columns in logged values not in header are {'new_value': 'int', 'old_value': 'int'} logger.info(

As crossed-checked by locally running another scenario in repo, I have got similar warnings. I understand these warnings are due to that in the log, first parameter's data type (such as bool type for 'spurious_symptoms ') and subsequent parameters' data types (such as list type of 'HR_expansion_by_officer_type') are different. Therefore, I think this is not an issue. (A copy of one tlo.scenario log is attached below.)
{"uuid": "eeda90f84c", "type": "header", "module": "tlo.scenario", "key": "override_parameter", "level": "INFO", "columns": {"module": "str", "name": "str", "new_value": "bool", "old_value": "bool"}, "description": null}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["SymptomManager", "spurious_symptoms", true, false]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "Service_Availability", [""], [""]]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "use_funded_or_actual_staffing", "actual", "funded_plus"]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "mode_appt_constraints", 1, 1]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "cons_availability", "default", "default"]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "beds_availability", "default", "default"]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "equip_availability", "all", "default"]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "use_funded_or_actual_staffing_postSwitch", "funded_plus", "funded_plus"]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "year_use_funded_or_actual_staffing_switch", 2020, 2100]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "mode_appt_constraints_postSwitch", 2, 1]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "year_mode_switch", 2020, 2100]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "cons_availability_postSwitch", "all", "default"]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "year_cons_availability_switch", 2020, 2100]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "yearly_HR_scaling_mode", "no_scaling", "no_scaling"]}
{"uuid": "eeda90f84c", "date": "2010-01-01T00:00:00", "values": ["HealthSystem", "HR_expansion_by_officer_type", [0, 0, 0, 0], [0, 0, 0, 0]]}

Hi @marghe-molaro and @tbhallett, I have created a hr expansion function in the healthsystem module, considering the fractions of extra budget to be used by the four cadres (clinical, dcsa, nursing_and_midwifery, pharmacy), updated the scenarios and scenario script, and added some relevant files as needed. I would be very grateful if you could help check if the function and logics are looking fine. Any comment or advice would be greatly appreciated. Thanks.

Hi all, I have done some checks myself and it seems the coding will work. Would still be grateful if you could help check and confirm the scenarios and settings to be run. Many thanks.
I just wonder how long it would take to run the scenario script for only first two scenarios (there are 17 scenarios in total to explore). If it is fine, I would like to submit such a job to Azure first to see the results and design analysis steps.

The two-scenario run (2 draws * 5 runs) takes 12 hours. Job ID: scenario_run_for_hcw_expansion_analysis-2024-08-16T160132Z

Thanks Tim very much for your efforts in PR #1414, from which I have learned a lot to extract and plot results of DALYs and relevant!
The results of the small run above are presented below. It seems the branch is overall working, although many details and confirmation of scenarios still need discussions.
(0) Extra staff and extra budget needed (to add)
(1) Absolute numbers of services, DALYs and deaths.

(2) Relative numbers of results, compared to no expansion scenario

(3) Return on Investment and Cost Effectiveness

Looking forward to discuss all these results and remaining issues (as listed in "todo" in the files) with you all!

@BinglingICL a minor point, but also remember that I expand capabilities from 2019 and not from 2020. I think ideally we should make this consistent.

@BinglingICL a minor point, but also remember that I expand capabilities from 2019 and not from 2020. I think ideally we should make this consistent.
Added: The Berman +2022 paper from which the data on capabilities was taken states "Staff data from districts, central hospitals, and headquarters, and enrollment data from all health training institutions were collected between October 2017 and February 2018", so my understanding was that this data would be "true" for 2018, hence why we'd expand from 2019 and not 2020.

@marghe-molaro Thanks for reminding me of this. I think that choice of 2018 is very sensible. But from my latest communication with CHAI, they said the data source should be HSSP III not Berman 2022 and I have asked them the exact year of the "current staff" of the version we have received, for which they said it is 2019, just before pandemic. And that is the reason for my choice of 2019. For next run, I will set the current year to be 2018 and change year to be 2019 to see if our results match. For the final run and presentation, I might prefer to set change year to be 2020, considering the information I have received; but I am not really sure, because your work will soon published.

@BinglingICL a minor point, but also remember that I expand capabilities from 2019 and not from 2020. I think ideally we should make this consistent.
Added: The Berman +2022 paper from which the data on capabilities was taken states "Staff data from districts, central hospitals, and headquarters, and enrollment data from all health training institutions were collected between October 2017 and February 2018", so my understanding was that this data would be "true" for 2018, hence why we'd expand from 2019 and not 2020.

@marghe-molaro Thanks for reminding me of this. I think that choice of 2018 is very sensible. But from my latest communication with CHAI, they said the data source should be HSSP III not Berman 2022 and I have asked them the exact year of the "current staff" of the version we have received, for which they said it is 2019, just before pandemic. And that is the reason for my choice of 2019. For next run, I will set the current year to be 2018 and change year to be 2019 to see if our results match. For the final run and presentation, I might prefer to set change year to be 2020, considering the information I have received; but I am not really sure, because your work will soon published.

Hi @marghe-molaro, could you please also have a quick look in that scenario file and suggest the settings (such as popsize, number of runs) to be consistent with your runs? Thanks.

Hi @BinglingICL sorry for the delayed response.
Wrt the year of increase, I don't think this should make a huge difference esp. given the uncertainty around other assumptions, but it would be good to have consistent approaches. So was the Berman +22 data never used? If available, would it be possible to compare it to estimates from CHAI to see what the difference was over the two years? @tbhallett, what's your view on this?
Wrt the scenario files, the following parameters are missing from your scenarios to make it consistent with mine. "scale_to_effective_capabilities": True, "policy_name": "Naive", "tclose_overwrite": 1, "tclose_days_offset_overwrite": 7,

Thanks @marghe-molaro. Will update scenario run settings accordingly.
Just to explain that the data source of HSSP III (and the one that generated our HR capabilities) is provided by CHAI, just an updated version of that used by Berman paper (which is also provided by CHAI and has the same officer categories, appointment types, districts, etcs, but updated staff counts).

Regarding the difference of CHAI's two versions, I am not sure it actually reflects the growth of HR or differences between year 2018 and 2019, as I am not sure how they update their data...

Let me clarify here for the record that, when including the "rescale to effective capabilities" option, the data on HRH is actually completely superfluous, because we take the HRH capabilities to be whatever was necessary to deliver all requested appts in the year before the switch to mode 2, and hence never rely on HRH data.
I think a more accurate way of describing what we are doing, rather than saying we use HSSP-III data, would be to say "We assume that all requested appointments are always delivered until 2018 (or 2019). In that final year, given that we know the time requirement from each officer type for each appt delivered, we calculate the average effective capabilities that would have been required to deliver all those appts in that year. In subsequent years, we enforce constraints on the delivery of appts based on those estimated capabilities, which will be expanded at different rates.

Let me clarify here for the record that, when including the "rescale to effective capabilities" option, the data on HRH is actually completely superfluous, because we take the HRH capabilities to be whatever was necessary to deliver all requested appts in the year before the switch to mode 2, and hence never rely on HRH data.
I think a more accurate way of describing what we are doing, rather than saying we use HSSP-III data, would be to say "We assume that all requested appointments are always delivered until 2018 (or 2019). In that final year, given that we know the time requirement from each officer type for each appt delivered, we calculate the average effective capabilities that would have been required to deliver all those appts in that year. In subsequent years, we enforce constraints on the delivery of appts based on those estimated capabilities, which will be expanded at different rates.
Added This means that the rescaling year is completely arbitrary, and not based on when the data on HRH capabilities was collected

Thanks @marghe-molaro for this explanation. This is super helpful. So, may I understand that we are actually scaling up the such estimated capabilities of the previous year (let us say 2018), not the capabilities stored in our resource file?

Dear all, the job of all 33 scenarios has been submitted. Hope it will run well and we could see the results next week.
By the way, Margherita @marghe-molaro, happy to let you know that the DALYs of year 2029 from my last run is around 9 million, looks consistent with yours!

Dear all, I would like to share some results for all scenarios that have been designed. Looking forward to meeting you next week and would very much appreciate your advice and comments.

Dear all, I would like to share some results for all scenarios that have been designed. Looking forward to meeting you next week and would very much appreciate your advice and comments.

Just for my own understanding, why is that some of the middle scenarios (e.g. CD_equal, CNO_equal) in the "Services increased by treatment type" graph seem to deliver fewer services than the "no expansion" scenario?

Just for my own understanding, why is that some of the middle scenarios (e.g. CD_equal, CNO_equal) in the "Services increased by treatment type" graph seem to deliver fewer services than the "no expansion" scenario?

Thanks for this question Rachel. Those scenarios do not have significantly less services than no expansion scenario. I have attached the percentages of service increased below. In scenarios where there is not much services increased but with more DALYs averted (as compared to no expansion scenario), there should be some trade-off between some treatments/appointments, leading to trade-off of dalys of some causes/injuries.
The contrary results (of services and dalys) of middle groups of scenarios and the right group, might also have such reasons behind, which need to be examined. The fundamental HRH allocation difference is that middle group have Clinical cadre expanded but no Pharmacy, whereas the right group have Pharmacy cadre expanded but not Clinical.

Thanks for this question Rachel. Those scenarios do not have significantly less services than no expansion scenario. I have attached the percentages of service increased below. In scenarios where there is not much services increased but with more DALYs averted (as compared to no expansion scenario), there should be some trade-off between some treatments/appointments, leading to trade-off of dalys of some causes/injuries.

Percentage service increased by treatment type against no expansion scenario
scenario | lower | mean | upper CNO_equal | 0.50% | 0.55% | 0.64% CD_equal | -0.03% | 0.13% | 0.26% Clinical | -0.54% | -0.26% | -0.03% CO_equal | -0.39% | -0.16% | -0.04%
The contrary results (of services and dalys) of middle groups of scenarios and the right group, might also have such reasons behind, which need to be examined. The fundamental HRH allocation difference is that middle group have Clinical cadre expanded but no Pharmacy, whereas the right group have Pharmacy cadre expanded but not Clinical.

Ah okay, so it's very small - do you think it's just stochasticty? I understand what you're saying in terms of DALYs etc., but I still would have thought that expansion of healthcare cadres would at least maintain the number of services provided, even if it doesn't actually increase it. (Irrespective of the effect on DALYs.)

Thanks for this question Rachel. Those scenarios do not have significantly less services than no expansion scenario. I have attached the percentages of service increased below. In scenarios where there is not much services increased but with more DALYs averted (as compared to no expansion scenario), there should be some trade-off between some treatments/appointments, leading to trade-off of dalys of some causes/injuries.

Percentage service increased by treatment type against no expansion scenario
scenario | lower | mean | upper CNO_equal | 0.50% | 0.55% | 0.64% CD_equal | -0.03% | 0.13% | 0.26% Clinical | -0.54% | -0.26% | -0.03% CO_equal | -0.39% | -0.16% | -0.04%
The contrary results (of services and dalys) of middle groups of scenarios and the right group, might also have such reasons behind, which need to be examined. The fundamental HRH allocation difference is that middle group have Clinical cadre expanded but no Pharmacy, whereas the right group have Pharmacy cadre expanded but not Clinical.

Ah okay, so it's very small - do you think it's just stochasticty? I understand what you're saying in terms of DALYs etc., but I still would have thought that expansion of healthcare cadres would at least maintain the number of services provided, even if it doesn't actually increase it. (Irrespective of the effect on DALYs.)

It might be for some scenarios, among the middle groups and right groups, which have <= 6% DALYs averted, and the last three scenarios, which have 0% DALYs averted. But I am not very sure, because looking at the results of the middle and right groups, the patterns are very consistent. If it is due to stochastic variation, some in middle group should have more services increased and/or some in right group should have less services increased.

Ah okay, so it's very small - do you think it's just stochasticty? I understand what you're saying in terms of DALYs etc., but I still would have thought that expansion of healthcare cadres would at least maintain the number of services provided, even if it doesn't actually increase it. (Irrespective of the effect on DALYs.)

It might be for some scenarios, among the middle groups and right groups, which have <= 6% DALYs averted, and the last three scenarios, which have 0% DALYs averted. But I am not very sure, because looking at the results of the middle and right groups, the patterns are very consistent. If it is due to stochastic variation, some in middle group should have more services increased and/or some in right group should have less services increased.

True, there is a pattern, so is there another reason why more healthcare workers would result in fewer services?

Ah okay, so it's very small - do you think it's just stochasticty? I understand what you're saying in terms of DALYs etc., but I still would have thought that expansion of healthcare cadres would at least maintain the number of services provided, even if it doesn't actually increase it. (Irrespective of the effect on DALYs.)

It might be for some scenarios, among the middle groups and right groups, which have <= 6% DALYs averted, and the last three scenarios, which have 0% DALYs averted. But I am not very sure, because looking at the results of the middle and right groups, the patterns are very consistent. If it is due to stochastic variation, some in middle group should have more services increased and/or some in right group should have less services increased.

True, there is a pattern, so is there another reason why more healthcare workers would result in fewer services?

I think the question is why more Clinical staff and less Pharmacy staff (as in Middle group scenarios) leading to less services and treatments (in total at least) but more DALYs averted, compared to less Clinical staff and more Pharmacy staff (as in Right group scenarios).
In an absolutely terms, there should no scenario that have more HRH but fewer services, which would indicate something not right in the simulation, except stochastic variations. Did you spot any such scenario, Rachel? I should take a closer look into that.

I think the question is why more Clinical staff and less Pharmacy staff (as in Middle group scenarios) leading to less services and treatments (in total at least) but more DALYs averted, compared to less Clinical staff and more Pharmacy staff (as in Right group scenarios).

Oh, dear all, I think the reason might be relevant to the "Pharm Dispensing" appointment (and the "Contraception" treatment), which is a dummy one; with more clinical staff, some services could be delivered rather than diverted to Pharm Dispensing. But I am not sure. Need to dig into this.

It might be for some scenarios, among the middle groups and right groups, which have <= 6% DALYs averted, and the last three scenarios, which have 0% DALYs averted. But I am not very sure, because looking at the results of the middle and right groups, the patterns are very consistent. If it is due to stochastic variation, some in middle group should have more services increased and/or some in right group should have less services increased.

True, there is a pattern, so is there another reason why more healthcare workers would result in fewer services?

I think the question is why more Clinical staff and less Pharmacy staff (as in Middle group scenarios) leading to less services and treatments (in total at least) but more DALYs averted, compared to less Clinical staff and more Pharmacy staff (as in Right group scenarios).
In an absolutely terms, there should no scenario that have more HRH but fewer services, which would indicate something not right in the simulation, except stochastic variations. Did you spot any such scenario, Rachel? I should take a closer look into that.

Do these ones I've squared in black here here not have more HRH (because they're expansion scenarios) but fewer services (negative on the axis)?

Thanks Rachel. Those scenarios have some more services like Contraception but less EPI and Firstattence, and the total services increased are close to 0 or positive. But you are right, we need to examine why some services are reduced in these expansion scenarios compared to no expansion scenario.


dask gateway usage

Describe the bug
This may be user error and not a bug. I am trying to use dask gateway as described here. My understanding would be that dask_gateway should therefore be included in the default jupyterlab conda environment here to avoid the import failure when I run:
from dask_gateway import Gateway
gateway = Gateway()

Is this correct? Shall I open a pull request?

Hmmm... i retried that and it worked in the "default" kernel. I think it just my default kernel is not "default". Will reopen if the problem is real


Fixing logging of netty client related ioexceptions to be at trace level

@sumanth-pasupuleti is this still relevant? Does it have a ticket? Otherwise please close it.


How to pass array to query.Run

Hey,
I have an array in the form of []map[string]interface{}
I would like to run a query such .[] | .id but it fails with:
expected an object but got: array ([{"id":"fb4ae77f-144d- ...)

what should I pass to query.Run in order to execute gojq over an array ?
Thanks

Please pass []interface{} for array and map[string]interface{} for map.

@itchyny thanks! keep the hard work, the project is great.


[BUG] Get-PnPTeamsTeam invalid giphyContentRating

Reporting an Issue or Missing Feature
When a team's fun stuff settings are set to giphy: allow all content, you are unable to get this team via Get-PnPTeamsTeam.

Expected behavior
Being able to retrieve the team via Get-PnPTeamsTeam
Actual behavior
null value is returned.
Steps to reproduce behavior

Pick a random team
Via the GUI settings, set fun stuff to allow all content (there is a possibility you don't see this option because it's restricted by the Teams admin center).
Use Get-PnPTeamsTeam -Identity <guid>

What is the version of the Cmdlet module you are running?
1.10.0
Which operating system/environment are you running PnP PowerShell on?

[x] Windows
[ ] Linux
[ ] MacOS
[ ] Azure Cloud Shell
[ ] Azure Functions
[ ] Other : please specify

Extra
I found out that this is the actual value returned by Graph:

However only strict and moderate is defined within the code.
https://github.com/pnp/powershell/blob/b0bcff86a7686ff81e7750d1a425ae7883d9fb17/src/Commands/Model/Teams/TeamFunSettings.cs#L41-L45
Because of this, the JSON parse to C# object fails and null is returned. I wanted to add unknownFutureValue as extra value, but this value cannot be used by other commands (e.g. Set-PnPTeamsTeam, New-PnPTeamsTeam). When trying to update a team with this value I do get following error:

What do you think about this? How can we best resolve this?

Well, if its not supported by Graph APIs themselves not sure what we could here.
Maybe once they update the endpoints themselves to return actual proper values , we would be able to change it to a proper enum value.

Yes, I don't think it's a huge issue, I think 99% of all teams use strict or moderate. It's sad that the GUI supports this, but Graph doesn't.
We could create a separate enum for the Get cmdlet and another for the Set & New. However maybe this is a bit overkill.

Yeah, that would be an overkill. Let's not do it .Once they release a proper enum value we can add that here. Adding a value a different from the one Graph team releases will introduce some breaking changes , will require some backward compatibility code and all that. Lets wait for Graph team to release it.


[fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)

Fixes #8407
Fixes #13865
Motivation
Current broker prometheus metrics are not grouped by metric type which causes issues in systems that read these metrics (e.g. DataDog).
Prometheus docs states "All lines for a given metric must be provided as one single group" - https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#grouping-and-sorting
Modifications
Updated the namespace and topic prometheus metric generators to group the metrics under the appropriate type header.
Updated function worker stats to include TYPE headers
Verifying this change

[x] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:
Added unit test to verify all metrics are grouped under correct type header
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Need to update docs?

[X] doc-not-needed
Changes to match prometheus spec

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@marksilcox Sorry, I have to revert this PR since it introduced a memory leak that was detected by a long-running continuous verification test.
The reproduction steps:

Start a standalone
Create a partitioned topic (100 partitions)
Start producer bin/pulsar-perf produce test -r 100 -bm 1 -mk random
Start consumer bin/pulsar-perf consume -st Key_Shared -r 100 -n 50 test
Access the metrics endpoint

for i in {1..1000}
do
curl -L localhost:8080/metrics/
done

After reverting this PR, the memory leak issue has been fixed.

I'm not sure where is the root cause yet. Since many people are built based on the Pulsar release branches, so we'd better revert first and then fix the memory issue and create a new PR again. @marksilcox If you need help with the memory leak issue, I believe @tjiuming or @asafm can provide some insight here.

I can't reproduce the issue on the master branch with PR (https://github.com/apache/pulsar/pull/15558); it looks like the memory leak issue only happened on branch-2.9

@codelipenghui The implementation between 2.9 and master are indeed different, hence the reason why the memory leak was introduced only in 2.9 - I checked it.


[Enhancement] Add support for FFmpeg

Summary
I think adding support for FFmpeg (or another framework with similar features) should be an issue to consider for Xamarin Community Toolkit, we are in 2021 and file decoding or encoding is something that many clients demand for their projects, but it is difficult to offer them a solution based on Xamarin.Forms mainly because generating binding libraries can be an odyssey when it comes to large libraries and the few open source libraries available are not available for all platforms that Xamarin.Forms supports given the complexity involved in the support of such libraries on all platforms. So my suggestion would be for the Xamarin Community Toolkit to add support for FFmpeg (or another framework with similar features) on Android, iOS, UWP, and macOS officially. I think FFmpeg is a large enough cross-platform library to consider this suggestion.
https://www.ffmpeg.org/
Also, this could indirectly benefit MediaElement by implementing Ffmpeg functionalities. For example the ffmediaelement project that exists for WPF implements some additional features to the native MediaElement:

Easily apply FFmpeg video and audio filtergraphs.
Specify input and codec parameters.
Opt-in hardware decoding acceleration via devices or via codecs.
Capture stream packets, audio, video and subtitle frames.
Change raw video, audio and subtitle data upon rendering.
Perform custom stream reading and stream recording.

Links of interest:
Ruslan Balanukhin has a FFmpeg interop link generator tool for C # /. NET and Mono that might be useful for this suggestion: FFmpeg.AutoGen
Who Will Do The Work?

[x] Just putting this out there for someone to pick up

@nacompllo I can see a lot of value in your proposal, but I believe that's out of our roadmap for now... That will be not an easy feature to maintain among other features that we have on this project. As a fast solution, I would suggest you use the VLC for Xamarin

Thanks! However, we are no longer adding new features to Xamarin Community Toolkit, focusing on the .NET MAUI Community Toolkit.
Please open a New Feature Discussion to implement this feature in the .NET MAUI Community Toolkit.
I've posted more information about the Future Of Xamarin Community Toolkit here: https://devblogs.microsoft.com/xamarin/the-future-of-xamarin-community-toolkit/?WT.mc_id=mobile-0000-bramin


How to make works onclick listener of a button behind the overlay Panel section

I have two buttons behind te overlay section of a panel and in this way their onClick listener's are not reachable cause of slidingPanel onClick. These buttons are on the left and right side of overlay sliding section. Is there a way to make works onClick of buttons if clicked them, and sliding panel onclick if clicked in the remaining area?

up


Access information not stored in RAM

I was wondering if there was a way to access information that isn't stored in the RAM. In some games, information is stored outside of the RAM, but I cannot figure out how to use this emulator to access that information. Anytime I use self.ram[] to access anything greater than 0x0800, I get an error that the RAM is a numpy array of 2048 entries, related to the 2KB memory of the RAM for a NES. I want to pull information stored elsewhere (0x6000 for instance). Is there a way to do this that I am not familiar with? Thank you!

My understanding is that addresses above 0x6000 are reserved for access to the cartridge RAM. This could be implemented, but is not immediately available at this time. What is your use cases for reading from these addresses?


Added skip to main content button

References

Fixes #2523

Description
Created the skip to main content button to automatically skip the header, navigation, the user setting buttons, ...
Checklist

[x] My PR is small in size (e.g. less than 1,000 lines of code, not including comments & specs/tests), or I have provided reasons as to why that's not possible.
[x] My PR passes ESLint validation using yarn lint
[x] My PR doesn't introduce circular dependencies (verified via yarn check-circ-deps)
[x] My PR includes TypeDoc comments for all new (or modified) public methods and classes. It also includes TypeDoc for large or complex private methods.
[x] My PR passes all specs/tests and includes new/updated specs or tests based on the Code Testing Guide.
[x] If my PR includes new libraries/dependencies (in package.json), I've made sure their licenses align with the DSpace BSD License based on the Licensing of Contributions documentation.
[x] If my PR includes new features or configurations, I've provided basic technical documentation in the PR itself.
[x] If my PR fixes an issue ticket, I've linked them together.

@alexandrevryghem : If you could cleanup the conflicts tomorrow (Friday) or Monday, I'd love to test this for possible inclusion in 7.6.1.  I believe the conflicts were accidentally caused by #2594

Manually ported to dspace-7_x in #2626


pdf-parser: init at 0.7.4

Motivation for this change
Add missing package for #81418
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

please rebase

Rebase done. I don't know why but I think ofborg is stuck. :thinking:


Add support for IoT attach_policy

This adds support for  IoT attach_policy.
Because attach_principal_policy is  deprecated
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iot.html#IoT.Client.attach_principal_policy

Coverage decreased (-1.0%) to 92.472% when pulling 80f860727fbacece3717d258d9079faae8fc5334 on cm-iwata:add_iot_attach_policy into e57798cb96966b83e42fb9ab3a9cc74f32879084 on spulec:staging.

Sorry,I mistook target branch


Allow OCM_URL to default to production

This allows OCM_URL to default to production, but allow the prompt to be changed by exporting a new var within the cluster.  Still allows the env var to be overridden in the container as well.

Lgtm, merging

Lgtm, merging


Output File listing

This adds a record of all output files (and file size) to job record. This expands on the outputs TaskParameter as that is a request, that can include base directories, and this is the manifest of what was actually done including all files contained in those directories.
Many workflow engines need to 'glob' the output directories to see what files were created (as some of the file names are not predetermined). Without this listing, it is incumbent on the workflow engine to independently contact the object store and get the listing manually. Having this listing removes the need for that.

Dropping in favor of #26 , which is a cherry-picked patch of the relevant changes.


segfault if CCL_PARAM_FILE points to a non-existent file

If CCL_PARAM_FILE environment variable points to a non-existent file, pyccl segfaults when creating a cosmology object:
anze@kosovel: ~ [14798] ▶ python -c "import pyccl; pyccl.Cosmology(Omega_b=0.05,Omega_c=0.3,h=0.7,n_s=0.96, sigma8=0.8)"
anze@kosovel: ~ [14798] ▶ export CCL_PARAM_FILE=nonexistent
anze@kosovel: ~ [14798] ▶ python -c "import pyccl; pyccl.Cosmology(Omega_b=0.05,Omega_c=0.3,h=0.7,n_s=0.96, sigma8=0.8)"
Segmentation fault (core dumped)
anze@kosovel: ~ [14798] ▶

Addressed with PR #387.


add webhook for tsp

Add webhook framework to crane project, currently tsp mutating and validating is added, it will be easier to support more objects.
webhooks server requires tls support, so the existing craned deployment is updated with a secret volumeMount change.
the initial key/cert for the webhook is checked in, if new certs is needed, user can run hack/scripts/gencerts.sh

rebased and resolved conflict


should we add pandas-style metadata to matrices?

Currently, we view the matrices more as numpy arrays than pandas dataframe. Ideally, we want to be in the middle and view this as columnar data. Most importantly, storing the column names would be very helpful for many different workflows.
A non-exhaustive list of implementation ideas:

add this to the current implementation of Dense|Sparse|Split|Categorical-Matrix.
create a new class that will hold the matrix as an attribute and expose some pandas-style helper functions.

#286  and #278 are steps in this direction


Q: Question on OpenMP dependency

Hi, we're thinking about removing dependency on OpenMP in our project (not linking with OMP and instead using TBB or native thread pool, as an option), and want to try MKL-DNN compiled with TBB support.
I noticed that there're some places where pragma omp is used (?), e.g.
define PRAGMA_OMP_SIMD(...) PRAGMA_MACRO(CHAIN2(omp, simd __VA_ARGS__))

Also found this comment in OpenMP.cmake:
# Compilation happens with OpenMP to enable `#pragma omp simd`
# but during linkage OpenMP dependency should be avoided

And in Threading.cmake:
# While MKL-DNN defaults to OpenMP (if _OPENMP is defined) without CMake, here
# we default to sequential threading and let OpenMP.cmake and TBB.cmake to
# figure things out. This is especially important because OpenMP is used both
# for threading and vectorization via #pragma omp simd

Not sure what implications are there for the TBB-based MKL-DNN build? Does it still need OpenMP and what would happen if we don't compile with OMP support and don't link with OpenMP library?
Also, related question - is there a way or is it possible to add a CMake option to build a sequential version of MKL-DNN?
Thank you!

(Re. second question - I found an option MKLDNN_THREADING=SEQ (undocumented?))

Hi @ilia-cher,
Yeah, MKLDNN_THREADING=SEQ is undocumented but proper way to build Intel MKL-DNN sequentially :)
Regarding OMP dependency with TBB. In order to make #pragma omp simd work you need to compile the code with OpenMP support (i.e. pass -fopenmp compiler flag). This by itself doesn't add a dependency on OpenMP RT, as opposite to the case when you use #pragma omp parallel. So if during the linkage a user doesn't pass -fopenmp or -lgomp/-liomp5, the library would not depend on OpenMP. See an example [1] below.
Since Intel MKL-DNN actively uses #pragma omp simd for vectorization, we try to use -fopenmp during compilation even if the library is built with TBB threading.
However, there are a few pitfalls about using TBB in Intel MKL-DNN:

Currently only inference is optimized for TBB. Backward pass would work as well, but there would be limitted parallelization, which would lead to not as good performance as OpenMP on the system with high core count. We have some prototype to improve TBB performance for training, but haven't had a chance to promote it yet;
If you want to build the library with TBB, you either should build with full Intel MKL binary dependency or without Intel MKL at all. You cannot use Intel MKL ML small binaries, because we only ship versions with GNU or Intel OpenMPs.

So it is fine to try Intel MKL-DNN with TBB, but please expect some performance impact, especially on backward pass.
BTW, may I know the reason you consider switching away from OpenMP?
[1]. Building Intel MKL-DNN w/ TBB (and compiling with -fopenmp to use vectorization)
$ cmake -DMKLDNN_THREADING=TBB -DMKLDNN_USE_MKL=NONE && make VERBOSE=1 -j install
# lots of output like:
# g++  ... -DMKLDNN_THR=MKLDNN_THR_TBB ...  -fopenmp ... -c jit_avx512_core_x8s8s32x_convolution.cpp
# lots of output, and build completed

$ ldd libmkldnn.so.<IP_ADDRESS>
        linux-vdso.so.1 =>  (0x00007ffd1f3a5000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f54352c2000)
        libtbb.so.2 => /lib64/libtbb.so.2 (0x00007f543508c000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5434e88000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f5434b7f000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f543487c000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5434666000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f54342a5000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f54365d9000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f543409c000)
# note, there is no libgomp.so (GNU OpenMP RT) in the dependencies

Thanks for the detailed answer!
One issue with OpenMP is that it seems to not work very well when application does some other kind of threading, e.g. it doesn't guarantee a fixed number of threads in the process (there's OMP_THREAD_LIMIT but it will crash the program if it goes over it). In particular, libgomp seems to create a new thread pool for each newly created std::thread, so we were thinking about using TBB (or native thread pool, but in this case we can't parallelize within MKL/MKL-DNN).

Yeah, composability with other threadings is definitely a weak point of OpenMP.
However, there are still some workarounds that might alleviate the problem. For instance if you use Intel OpenMP runtime in most of the cases setting KMP_BLOCKTIME=0 might solve the issue of oversubscribed system (in the assumption that you still don't run simultaneously OpenMP-enabled code with something else). But I agree that TBB or even application's native thread-pool (if Intel MKL-DNN supports it) might be a better solution.
Please let us know your findings once you try Intel MKL-DNN with TBB :)

thanks!


Added possibility to call parameters from the template. Removed bug with params.

...n parametrised string was incorrectly called.

thanks, great catch


DC due dates

Are DCs typically due on Thursdays or Fridays? On the syllabus they are listed next to Thurs, but in the DC description it says they are due Friday at 11. Just curious! See you friends tomorrow!

The DC description is the actual due date. The syllabus doesn't list a Friday and so they are placed in the week where they are due.


MongoDB

[ ] - Finalize bootstrapping mongo server, see we know how to mess around with it
[ ] - Start modeling data structure based on Open Knesset API (with @pupTentacle )

Open Knesset API
https://oknesset.org/api/v2/doc/


Experimental Kintex7 xc7k325 support

I followed the guide to add new parts, more or less, some experimental features had to be commented out. No validation on real hardware has been done, this is just to establish if the pull request changes can run

@kgugala Who should review this?

The build failed, but a necessary step: 'make db-prepare-artix7'
does not seem to be included in the regression flow. As far as
I can see this needs to run before 'make -C fuzzers part_only'

I believe that make db-prepare-kintex7 should be run and committed?

Actually maybe I'm wrong...

https://github.com/SymbiFlow/prjxray/blob/60168e9b7e89956ce8a197f3cfdf6d4bc80926d3/.github/kokoro/db-full.sh#L32-L39

========================================
Preparing database
----------------------------------------

Preparing kintex7 files
============================
Part xc7k160tfbg484-1 has an unsupported device xc7k160t
Part xc7k160tfbg484-2 has an unsupported device xc7k160t
Part xc7k160tfbg484-2L has an unsupported device xc7k160t
Part xc7k160tfbg484-3 has an unsupported device xc7k160t
Part xc7k160tfbg676-1 has an unsupported device xc7k160t
Part xc7k160tfbg676-2 has an unsupported device xc7k160t
Part xc7k160tfbg676-2L has an unsupported device xc7k160t
Part xc7k160tfbg676-3 has an unsupported device xc7k160t
Part xc7k160tfbv484-1 has an unsupported device xc7k160t
Part xc7k160tfbv484-2 has an unsupported device xc7k160t
Part xc7k160tfbv484-2L has an unsupported device xc7k160t
Part xc7k160tfbv484-3 has an unsupported device xc7k160t
Part xc7k160tfbv676-1 has an unsupported device xc7k160t
Part xc7k160tfbv676-2 has an unsupported device xc7k160t
Part xc7k160tfbv676-2L has an unsupported device xc7k160t
Part xc7k160tfbv676-3 has an unsupported device xc7k160t
Part xc7k160tffg676-1 has an unsupported device xc7k160t
Part xc7k160tffg676-2 has an unsupported device xc7k160t
Part xc7k160tffg676-2L has an unsupported device xc7k160t
Part xc7k160tffg676-3 has an unsupported device xc7k160t
Part xc7k160tffv676-1 has an unsupported device xc7k160t
Part xc7k160tffv676-2 has an unsupported device xc7k160t
Part xc7k160tffv676-2L has an unsupported device xc7k160t
Part xc7k160tffv676-3 has an unsupported device xc7k160t
Part xc7k160tifbv676-2L has an unsupported device xc7k160ti
Part xc7k160tifbg484-2L has an unsupported device xc7k160ti
Part xc7k160tifbg676-2L has an unsupported device xc7k160ti
Part xc7k160tifbv484-2L has an unsupported device xc7k160ti
Part xc7k160tiffv676-2L has an unsupported device xc7k160ti
Part xc7k160tiffg676-2L has an unsupported device xc7k160ti
Find pins for xc7k70tfbg484-1
Find pins for xc7k70tfbg676-1
Find pins for xc7k70tfbv484-1
Find pins for xc7k70tfbv676-1
----------------------------------------

make[1]: Entering directory '/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers'
.//run_fuzzer.py 000-init-db
2021-12-07T14:56:38 - xc7k70tfbv676-3/000-init-db               -    0s: Running fuzzer attempt: 0
mkdir -p fuzzer_ok
touch fuzzer_ok/fuzzer_000-init-db_xc7k70tfbv676-3.ok
.//run_fuzzer.py 001-part-yaml
2021-12-07T14:56:38 - xc7k70tfbv676-3/001-part-yaml             -    0s: Running fuzzer attempt: 0
.//run_fuzzer.py 075-pins
2021-12-07T14:56:38 - xc7k70tfbv676-3/075-pins                  -    0s: Running fuzzer attempt: 0
make[1]: Leaving directory '/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers'
source settings/kintex7_325t.sh && make -C fuzzers part_only
Traceback (most recent call last):
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/utils/create_environment.py", line 66, in <module>
    main()
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/utils/create_environment.py", line 59, in main
    environment = get_environment_variables()
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/utils/create_environment.py", line 37, in get_environment_variables
    part_info = get_part_information(db_root, part)
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/prjxray/util.py", line 48, in get_part_information
    assert part, "Part {} not found in {}".format(part, part_mapping)
AssertionError: Part None not found in {'xc7k70tfbg484-1': {'device': 'xc7k70t', 'package': 'fbg484', 'speedgrade': '1'}, 'xc7k70tfbg484-2': {'device': 'xc7k70t', 'package': 'fbg484', 'speedgrade': '2'}, 'xc7k70tfbg484-2L': {'device': 'xc7k70t', 'package': 'fbg484', 'speedgrade': '2L'}, 'xc7k70tfbg484-3': {'device': 'xc7k70t', 'package': 'fbg484', 'speedgrade': '3'}, 'xc7k70tfbg676-1': {'device': 'xc7k70t', 'package': 'fbg676', 'speedgrade': '1'}, 'xc7k70tfbg676-2': {'device': 'xc7k70t', 'package': 'fbg676', 'speedgrade': '2'}, 'xc7k70tfbg676-2L': {'device': 'xc7k70t', 'package': 'fbg676', 'speedgrade': '2L'}, 'xc7k70tfbg676-3': {'device': 'xc7k70t', 'package': 'fbg676', 'speedgrade': '3'}, 'xc7k70tfbv484-1': {'device': 'xc7k70t', 'package': 'fbv484', 'speedgrade': '1'}, 'xc7k70tfbv484-2': {'device': 'xc7k70t', 'package': 'fbv484', 'speedgrade': '2'}, 'xc7k70tfbv484-2L': {'device': 'xc7k70t', 'package': 'fbv484', 'speedgrade': '2L'}, 'xc7k70tfbv484-3': {'device': 'xc7k70t', 'package': 'fbv484', 'speedgrade': '3'}, 'xc7k70tfbv676-1': {'device': 'xc7k70t', 'package': 'fbv676', 'speedgrade': '1'}, 'xc7k70tfbv676-2': {'device': 'xc7k70t', 'package': 'fbv676', 'speedgrade': '2'}, 'xc7k70tfbv676-2L': {'device': 'xc7k70t', 'package': 'fbv676', 'speedgrade': '2L'}, 'xc7k70tfbv676-3': {'device': 'xc7k70t', 'package': 'fbv676', 'speedgrade': '3'}}
Makefile:204: recipe for target 'db-part-only-kintex7_325t' failed
make: *** [db-part-only-kintex7_325t] Error 1

[ID: 1446474] Build finished after 197 secs, exit value: 1

Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[06:56:39] Collecting build artifacts from build VM
Build script failed with exit code: 1

This is what I get running locally:
`jrrk@jonathan-NUC10i7FNK:~/prjxray$ make db-prepare-kintex7
Preparing kintex7 files
Part xc7k160tfbg484-1 has an unsupported device xc7k160t
Part xc7k160tfbg484-2 has an unsupported device xc7k160t
Part xc7k160tfbg484-2L has an unsupported device xc7k160t
Part xc7k160tfbg484-3 has an unsupported device xc7k160t
Part xc7k160tfbg676-1 has an unsupported device xc7k160t
Part xc7k160tfbg676-2 has an unsupported device xc7k160t
Part xc7k160tfbg676-2L has an unsupported device xc7k160t
Part xc7k160tfbg676-3 has an unsupported device xc7k160t
Part xc7k160tfbv484-1 has an unsupported device xc7k160t
Part xc7k160tfbv484-2 has an unsupported device xc7k160t
Part xc7k160tfbv484-2L has an unsupported device xc7k160t
Part xc7k160tfbv484-3 has an unsupported device xc7k160t
Part xc7k160tfbv676-1 has an unsupported device xc7k160t
Part xc7k160tfbv676-2 has an unsupported device xc7k160t
Part xc7k160tfbv676-2L has an unsupported device xc7k160t
Part xc7k160tfbv676-3 has an unsupported device xc7k160t
Part xc7k160tffg676-1 has an unsupported device xc7k160t
Part xc7k160tffg676-2 has an unsupported device xc7k160t
Part xc7k160tffg676-2L has an unsupported device xc7k160t
Part xc7k160tffg676-3 has an unsupported device xc7k160t
Part xc7k160tffv676-1 has an unsupported device xc7k160t
Part xc7k160tffv676-2 has an unsupported device xc7k160t
Part xc7k160tffv676-2L has an unsupported device xc7k160t
Part xc7k160tffv676-3 has an unsupported device xc7k160t
Part xc7k160tifbv676-2L has an unsupported device xc7k160ti
Part xc7k160tifbg484-2L has an unsupported device xc7k160ti
Part xc7k160tifbg676-2L has an unsupported device xc7k160ti
Part xc7k160tifbv484-2L has an unsupported device xc7k160ti
Part xc7k160tiffv676-2L has an unsupported device xc7k160ti
Part xc7k160tiffg676-2L has an unsupported device xc7k160ti
Part xc7k325tiffv676-2L has an unsupported device xc7k325ti
Part xc7k325tiffv900-2L has an unsupported device xc7k325ti
Part xc7k325tifbg676-2L has an unsupported device xc7k325ti
Part xc7k325tifbg900-2L has an unsupported device xc7k325ti
Part xc7k325tifbv676-2L has an unsupported device xc7k325ti
Part xc7k325tifbv900-2L has an unsupported device xc7k325ti
Part xc7k325tiffg900-2L has an unsupported device xc7k325ti
Part xc7k325tiffg676-2L has an unsupported device xc7k325ti
Part xc7k355tiffg901-2L has an unsupported device xc7k355ti
Part xc7k355tiffv901-2L has an unsupported device xc7k355ti
Part xc7k355tffg901-1 has an unsupported device xc7k355t
Part xc7k355tffg901-2 has an unsupported device xc7k355t
Part xc7k355tffg901-2L has an unsupported device xc7k355t
Part xc7k355tffg901-3 has an unsupported device xc7k355t
Part xc7k355tffv901-1 has an unsupported device xc7k355t
Part xc7k355tffv901-2 has an unsupported device xc7k355t
Part xc7k355tffv901-2L has an unsupported device xc7k355t
Part xc7k355tffv901-3 has an unsupported device xc7k355t
Part xc7k410tfbg676-1 has an unsupported device xc7k410t
Part xc7k410tfbg676-2 has an unsupported device xc7k410t
Part xc7k410tfbg676-2L has an unsupported device xc7k410t
Part xc7k410tfbg676-3 has an unsupported device xc7k410t
Part xc7k410tfbg900-1 has an unsupported device xc7k410t
Part xc7k410tfbg900-2 has an unsupported device xc7k410t
Part xc7k410tfbg900-2L has an unsupported device xc7k410t
Part xc7k410tfbg900-3 has an unsupported device xc7k410t
Part xc7k410tfbv676-1 has an unsupported device xc7k410t
Part xc7k410tfbv676-2 has an unsupported device xc7k410t
Part xc7k410tfbv676-2L has an unsupported device xc7k410t
Part xc7k410tfbv676-3 has an unsupported device xc7k410t
Part xc7k410tfbv900-1 has an unsupported device xc7k410t
Part xc7k410tfbv900-2 has an unsupported device xc7k410t
Part xc7k410tfbv900-2L has an unsupported device xc7k410t
Part xc7k410tfbv900-3 has an unsupported device xc7k410t
Part xc7k410tffg676-1 has an unsupported device xc7k410t
Part xc7k410tffg676-2 has an unsupported device xc7k410t
Part xc7k410tffg676-2L has an unsupported device xc7k410t
Part xc7k410tffg676-3 has an unsupported device xc7k410t
Part xc7k410tffg900-1 has an unsupported device xc7k410t
Part xc7k410tffg900-2 has an unsupported device xc7k410t
Part xc7k410tffg900-2L has an unsupported device xc7k410t
Part xc7k410tffg900-3 has an unsupported device xc7k410t
Part xc7k410tffv676-1 has an unsupported device xc7k410t
Part xc7k410tffv676-2 has an unsupported device xc7k410t
Part xc7k410tffv676-2L has an unsupported device xc7k410t
Part xc7k410tffv676-3 has an unsupported device xc7k410t
Part xc7k410tffv900-1 has an unsupported device xc7k410t
Part xc7k410tffv900-2 has an unsupported device xc7k410t
Part xc7k410tffv900-2L has an unsupported device xc7k410t
Part xc7k410tffv900-3 has an unsupported device xc7k410t
Part xc7k410tiffv900-2L has an unsupported device xc7k410ti
Part xc7k410tiffv676-2L has an unsupported device xc7k410ti
Part xc7k410tiffg900-2L has an unsupported device xc7k410ti
Part xc7k410tiffg676-2L has an unsupported device xc7k410ti
Part xc7k410tifbv900-2L has an unsupported device xc7k410ti
Part xc7k410tifbv676-2L has an unsupported device xc7k410ti
Part xc7k410tifbg900-2L has an unsupported device xc7k410ti
Part xc7k410tifbg676-2L has an unsupported device xc7k410ti
Part xc7k420tffg1156-1 has an unsupported device xc7k420t
Part xc7k420tffg1156-2 has an unsupported device xc7k420t
Part xc7k420tffg1156-2L has an unsupported device xc7k420t
Part xc7k420tffg1156-3 has an unsupported device xc7k420t
Part xc7k420tffg901-1 has an unsupported device xc7k420t
Part xc7k420tffg901-2 has an unsupported device xc7k420t
Part xc7k420tffg901-2L has an unsupported device xc7k420t
Part xc7k420tffg901-3 has an unsupported device xc7k420t
Part xc7k420tffv1156-1 has an unsupported device xc7k420t
Part xc7k420tffv1156-2 has an unsupported device xc7k420t
Part xc7k420tffv1156-2L has an unsupported device xc7k420t
Part xc7k420tffv1156-3 has an unsupported device xc7k420t
Part xc7k420tffv901-1 has an unsupported device xc7k420t
Part xc7k420tffv901-2 has an unsupported device xc7k420t
Part xc7k420tffv901-2L has an unsupported device xc7k420t
Part xc7k420tffv901-3 has an unsupported device xc7k420t
Part xc7k420tiffv901-2L has an unsupported device xc7k420ti
Part xc7k420tiffv1156-2L has an unsupported device xc7k420ti
Part xc7k420tiffg901-2L has an unsupported device xc7k420ti
Part xc7k420tiffg1156-2L has an unsupported device xc7k420ti
Part xc7k480tiffv901-2L has an unsupported device xc7k480ti
Part xc7k480tiffv1156-2L has an unsupported device xc7k480ti
Part xc7k480tiffg901-2L has an unsupported device xc7k480ti
Part xc7k480tiffg1156-2L has an unsupported device xc7k480ti
Part xc7k480tffv901-1 has an unsupported device xc7k480t
Part xc7k480tffv901-2 has an unsupported device xc7k480t
Part xc7k480tffv901-2L has an unsupported device xc7k480t
Part xc7k480tffv901-3 has an unsupported device xc7k480t
Part xc7k480tffv1156-1 has an unsupported device xc7k480t
Part xc7k480tffv1156-2 has an unsupported device xc7k480t
Part xc7k480tffv1156-2L has an unsupported device xc7k480t
Part xc7k480tffv1156-3 has an unsupported device xc7k480t
Part xc7k480tffg901-1 has an unsupported device xc7k480t
Part xc7k480tffg901-2 has an unsupported device xc7k480t
Part xc7k480tffg901-2L has an unsupported device xc7k480t
Part xc7k480tffg901-3 has an unsupported device xc7k480t
Part xc7k480tffg1156-1 has an unsupported device xc7k480t
Part xc7k480tffg1156-2 has an unsupported device xc7k480t
Part xc7k480tffg1156-2L has an unsupported device xc7k480t
Part xc7k480tffg1156-3 has an unsupported device xc7k480t
Find pins for xc7k325tfbg676-1
Find pins for xc7k325tfbg900-1
Find pins for xc7k325tfbv676-1
Find pins for xc7k325tfbv900-1
Find pins for xc7k325tffg676-1
Find pins for xc7k325tffg900-1
Find pins for xc7k325tffv676-1
Find pins for xc7k325tffv900-1
Find pins for xc7k70tfbg484-1
Find pins for xc7k70tfbg676-1
Find pins for xc7k70tfbv484-1
Find pins for xc7k70tfbv676-1
`
Perhaps there is a problem getting the license for non-webpack parts ?

The CI system only currently has access to WebPack parts.

Is the xc7k325 part a non-webpack part?

Looks like only Kintex®-7 FPGA: • XC7K70T, XC7K160T are supported by the WebPack version.

Perhaps we need two X-Ray databases, one that can be generated from WebPack, another (for larger parts) that has to be generated by owners of suitable boards (that come with an unlimited number of node-locked and device-locked threads). Another way would be to affiliate with an academic institution that has access to bulk licenses. Meanwhile, I will try and get a  bitstream running on a real board to see if we have something usable.

I committed my new database here (https://github.com/jrrk2/prjxray-db/tree/k325). I suppose that means that eventually anybody can use this part license-free.

jrrk2 makes a good point here; for example, the 7k325t and 7k420t dev kits are actually cheaper on aliexpress than the 7k160t. I suspect this because the larger parts are more popular in the supply chain, and 160t can fetch a higher price as the latest webpack can be used.
Using these parts without a vivado license is actually a major reason I am interested in the open source FPGA tooling.

Furthermore the 7k325t part is used in the frequently encountered KC705 board from Xilinx and the Genesys2 board from Diligent. The latter comes with a Vivado node/device-locked license which may be used to generate the open-source database.

I'm very close to being able to access our license server on our Kokoro system.

Thanks for the heads-up but this will not take us much further forward unless support is added for kintex7 in the symbiflow-arch-defs repo. I'm not familiar with CMake config files nor can I find any documentation on how to do this. In addition the Python support seems to be Ubuntu/Debian specific. I tried it on Fedora-35 and it didn't work.

@jrrk2 Did you log a bug about it not working on Fedora-35 somewhere?

Kintex 7 failed really fast. First make --dry-run and source settings/kintex7.sh && make -C fuzzers ran without actually running any fuzzers, then source settings/kintex7_325t.sh && make -C fuzzers part_only failed, which seems to be caused by make failing to find xc7k325t amongst a list consisted only of xc7k70t parts. My initial guess is some Makefiles aren't written correctly, haven't looked into it yet though.

Kintex 7 failed really fast. First make --dry-run and source settings/kintex7.sh && make -C fuzzers ran without actually running any fuzzers, then source settings/kintex7_325t.sh && make -C fuzzers part_only failed, which seems to be caused by make failing to find xc7k325t amongst a list consisted only of xc7k70t parts. My initial guess is some Makefiles aren't written correctly, haven't looked into it yet though.

You need to run make db-prepare-kintex7 before source settings/kintex7_325t.sh as per the documentation (mutatis mutandis). Also the whole thing will fall flat if you don't have the Vivado license for this device. I just checked out mithro's patches and it seems to run OK (probably will take a day or so on 12 cores).

I spoke too soon, XRAY_ROI_TILEGRID is commented out which I took to mean (based on the comment above) that the whole device would be considered. Mea culpa.

@jrrk2 No I was referring to Kokoro's build log. I f you scroll down to the end you'll find that no fuzzy tests ran at all, with main error being:
source settings/kintex7_325t.sh && make -C fuzzers part_only
Traceback (most recent call last):
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/utils/create_environment.py", line 66, in <module>
    main()
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/utils/create_environment.py", line 59, in main
    environment = get_environment_variables()
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/utils/create_environment.py", line 37, in get_environment_variables
    part_info = get_part_information(db_root, part)
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/prjxray/util.py", line 48, in get_part_information
    assert part, "Part {} not found in {}".format(part, part_mapping)
AssertionError: Part None not found in {'xc7k70tfbg484-1': {'device': 'xc7k70t', 'package': 'fbg484', 'speedgrade': '1'}, 'xc7k70tfbg484-2': {'device': 'xc7k70t', 'package': 'fbg484', 'speedgrade': '2'}, 'xc7k70tfbg484-2L': {'device': 'xc7k70t', 'package': 'fbg484', 'speedgrade': '2L'}, 'xc7k70tfbg484-3': {'device': 'xc7k70t', 'package': 'fbg484', 'speedgrade': '3'}, 'xc7k70tfbg676-1': {'device': 'xc7k70t', 'package': 'fbg676', 'speedgrade': '1'}, 'xc7k70tfbg676-2': {'device': 'xc7k70t', 'package': 'fbg676', 'speedgrade': '2'}, 'xc7k70tfbg676-2L': {'device': 'xc7k70t', 'package': 'fbg676', 'speedgrade': '2L'}, 'xc7k70tfbg676-3': {'device': 'xc7k70t', 'package': 'fbg676', 'speedgrade': '3'}, 'xc7k70tfbv484-1': {'device': 'xc7k70t', 'package': 'fbv484', 'speedgrade': '1'}, 'xc7k70tfbv484-2': {'device': 'xc7k70t', 'package': 'fbv484', 'speedgrade': '2'}, 'xc7k70tfbv484-2L': {'device': 'xc7k70t', 'package': 'fbv484', 'speedgrade': '2L'}, 'xc7k70tfbv484-3': {'device': 'xc7k70t', 'package': 'fbv484', 'speedgrade': '3'}, 'xc7k70tfbv676-1': {'device': 'xc7k70t', 'package': 'fbv676', 'speedgrade': '1'}, 'xc7k70tfbv676-2': {'device': 'xc7k70t', 'package': 'fbv676', 'speedgrade': '2'}, 'xc7k70tfbv676-2L': {'device': 'xc7k70t', 'package': 'fbv676', 'speedgrade': '2L'}, 'xc7k70tfbv676-3': {'device': 'xc7k70t', 'package': 'fbv676', 'speedgrade': '3'}}
Makefile:205: recipe for target 'db-part-only-kintex7_325t' failed
make: *** [db-part-only-kintex7_325t] Error 1

[ID: 8383546] Build finished after 221 secs, exit value: 1

From the log, it seems that Kokoro halted at the preparation stage, hence why we can see it failed mere minutes after the build test started. Note how 'make db-part-only-kintex7_325t' when looking for 325t parts in that list of 70t parts. I suspect something is wrong with the files that set up the CI process, since you have successfully created your own prjxray-db repo that has 325t parts' info in it, though unfortunately I can't help you test it (don't have a paid Vivado license for 325t).
Just to reduce potential confusion, "Kokoro" is a CI system used by Google internally. In this case it automatically compiles and runs all of X-Ray's fuzzers for all of X-Ray's supported parts (on Google's servers, I think). In my experience the files that tells Kokoro how to run the tests are here but that's about all I know. I'm pretty new to SymbiFlow too, and I'm not a Googler either. Maybe you can ask @mithro @acomodi @kgugala for more  details.

It seems to me the automated process preferably should be modified to include utils/prjxray_form_channels.py and friends, this is also a lengthy process which takes a number of hours per part. Also there is a potential boostrapping issue with prjxray/database directory that gets replaced by the newly created prjxray-db repo. I'm not sure how to solve either of these problems at the moment. I think there might be a temporary way to generate the channels for kintex7 using 'make -n -B' on a working artix7 part, capturing the output as a shell script, and then globally replacing the old part with the new one. However that's not good except as a proof of concept to see if the scripts can run. Another issue is that the   xc7k325t has six global clock regions and I don't know if there is a general solution to the problem of how to connect them in VPR. There are references to _bottom and presumably _top in smaller devices but this needs to be made scalable and documented by somebody familiar with the flow. I don't know if there are designs that cross clock regions but it would make sense that there would be a timing penalty and the possibility of hold violations to be taken into account.

So, the Kintex 7 build failed after ~5 hours and I'm unclear why.
make[1]: Leaving directory '/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers'
Makefile:181: recipe for target 'db-kintex7' failed
make: *** [db-kintex7] Error 2

.....

2021-12-21T19:45:51 - xc7k325tffg900-2/074-dump_all              - 5h15m: make[3]: Leaving directory '/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/074-dump_all'

2021-12-21T19:45:51 - xc7k325tffg900-2/074-dump_all              - 5h15m: # Clean up intermediate files after successful pushdb.

2021-12-21T19:45:51 - xc7k325tffg900-2/074-dump_all              - 5h15m: find build_xc7k325tffg900-2 -name "*.json5" -delete

2021-12-21T19:45:51 - xc7k325tffg900-2/074-dump_all              - 5h15m: touch run.xc7k325tffg900-2.ok

2021-12-21T19:45:51 - xc7k325tffg900-2/074-dump_all              - 5h15m: make[2]: Leaving directory '/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/074-dump_all'

2021-12-21T19:45:51 - xc7k325tffg900-2/074-dump_all              - 5h15m:

2021-12-21T19:45:51 - xc7k325tffg900-2/074-dump_all              - 5h15m: --------------------------------------------------------------------------

2021-12-21T19:45:51 - xc7k325tffg900-2/074-dump_all              - 5h15m: Succeeded! @ 2021-12-21T19:45:51.395225

make[1]: Leaving directory '/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers'

Script done, file is /tmpfs/tmp/tmp.eaE7a4s3sK
+ DATABASE_RET=2
+ set +x -e
Packing failing test cases
----------------------------------------
A failure occurred during Database build.
----------------------------------------
========================================
 Disk space in failure path
----------------------------------------
108G	.

[ID: 3389105] Build finished after 29135 secs, exit value: 2

Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[13:18:20] Executor<PHONE_NUMBER> is unresponsive. Build terminated.

I'm wondering if it is running out of disk space...
The current run is working on;
2021-12-22T03:22:39 - xc7k325tffg900-2/074-dump_all              - 3h12m: Still running (1m:1.98%, 5m:3.75%, 15m:16.27% Mem:3.5Gi used, 349.0Gi free).

2021-12-22T03:22:39 - xc7k325tffg900-2/074-dump_all              - 3h12m: make---make---bash---generate_after_---python3-+-48*[python3]

2021-12-22T03:22:39 - xc7k325tffg900-2/074-dump_all              - 3h12m:                                                `-sh---vivado.sh---vivado---loader---vivado

2021-12-22T03:22:49 - xc7k325tffg900-2/074-dump_all              - 3h12m: Still running (1m:2.15%, 5m:3.73%, 15m:16.12% Mem:1.2Gi used, 351.0Gi free).

2021-12-22T03:22:49 - xc7k325tffg900-2/074-dump_all              - 3h12m: make---make---bash---generate_after_---python3

2021-12-22T03:22:59 - xc7k325tffg900-2/074-dump_all              - 3h12m: Still running (1m:2.12%, 5m:3.67%, 15m:15.98% Mem:1.6Gi used, 350.0Gi free).

I also logged https://github.com/SymbiFlow/prjxray/issues/1810

Ahh ha!
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: ++ md5sum
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: ++ echo /tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/051-pip-imuxlout-bypalts/build/1/specimen_008
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: + echo '`define SEED 32'\''h84da4d6f'
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: + make -f /tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/utils/top_generate.mk
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: Traceback (most recent call last):
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:   File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/int_generate.py", line 103, in <module>
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:     segmk.compile(bitfilter=get_bitfilter(os.getenv('XRAY_DATABASE'), 'INT'))
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:   File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/prjxray/segmaker.py", line 391, in compile
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:     assert this_tile_tags == 0, "Tile %s does not have bitstream info but %s tags" % (
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: AssertionError: Tile INT_L_X24Y151 does not have bitstream info but 50 tags
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: make[4]: *** [generate.ok] Error 1
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: make[3]: *** [/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/051-pip-imuxlout-bypalts/build/1/specimen_003/OK] Error 2
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: make[3]: *** Waiting for unfinished jobs....
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: Traceback (most recent call last):
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:   File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/int_generate.py", line 103, in <module>
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:     segmk.compile(bitfilter=get_bitfilter(os.getenv('XRAY_DATABASE'), 'INT'))
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:   File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/prjxray/segmaker.py", line 391, in compile
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:     assert this_tile_tags == 0, "Tile %s does not have bitstream info but %s tags" % (
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: AssertionError: Tile INT_L_X24Y151 does not have bitstream info but 50 tags
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: make[4]: *** [generate.ok] Error 1
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: make[3]: *** [/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/051-pip-imuxlout-bypalts/build/1/specimen_007/OK] Error 2
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: Traceback (most recent call last):
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:   File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/int_generate.py", line 103, in <module>
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:     segmk.compile(bitfilter=get_bitfilter(os.getenv('XRAY_DATABASE'), 'INT'))
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:   File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/prjxray/segmaker.py", line 391, in compile
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:     assert this_tile_tags == 0, "Tile %s does not have bitstream info but %s tags" % (
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: AssertionError: Tile INT_L_X24Y151 does not have bitstream info but 50 tags
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: make[4]: *** [generate.ok] Error 1
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: make[3]: *** [/tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/051-pip-imuxlout-bypalts/build/1/specimen_004/OK] Error 2
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: make[2]: *** [run] Error 2
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s:
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: --------------------------------------------------------------------------
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: !Failed! @ 2021-12-21T13:54:03.214741 with exit code: 2
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: --------------------------------------------------------------------------
2021-12-21T13:54:03 - xc7k70tfbg676-2/051-pip-imuxlout-bypalts  - 58m03s: WARNING: Fuzzer failed!

I haven’t tried to build the xc7k70 support and I’m not aware of any cost-effective boards that use it. My recollection is that the xc7k325 support peaks at around 250GB disk space but difficult to be sure because the clean up when it comes takes a matter of seconds on SSD disk.

@jrrk2 - Was the xc7k70t not building before?

It looks like that part is building successfully on master at the moment? https://source.cloud.google.com/results/invocations/bc5ac76c-546a-4899-8cf6-384ca0d127a8/targets

Could the change to https://github.com/SymbiFlow/prjxray/pull/1794/files#diff-2aea961e96499d96f980a17a1110978f3bf66bcc1e4ec1e772aaec71f6932614 be breaking the xc7k70t part? All the other changes should only affect the xc7k325 if I understand them correctly?

I'm a little unclear here, but it seems like the xc7k70 needs the orphan_int_column fuzzer but is not compatible with the dsp_int fuzzer, while the xc7k325 does work with the dsp_int fuzzer?

(FYI - I sent https://github.com/SymbiFlow/prjxray/pull/1811 to be a bit clearer about this section of the Makefile.)

Perhaps this anomaly would be explained by the DSP location limits being wrong in the environment configuration file ?

In order to test if the DSP fuzzer really is broken on Kintex7 (apart from the new part addition), I created https://github.com/SymbiFlow/prjxray/pull/1816

As #1811 was passing, it has been merged. I rebased this pull request onto the merge and are now rerunning the CI to see if we still get the xc7k70tfbg676-2/051-pip-imuxlout-bypalts failure....

With the current limited set of changes, I don't see any reason that this change should affect the runs for the xc7k70tfbg676-2 parts which currently should pass.

I've been lurking here and am interested in helping. I wonder if the following line is breaking the run for xc7k70tfbg676-2 parts? Admittedly though, it's not obvious to me how run for Kintex parts ever happen with KINTEX_PARTS= on master.
https://github.com/jrrk2/prjxray/blob/330f7984a1e3e217258f44f4a46d9031e900bc51/Makefile#L188

I've been lurking here and am interested in helping. I wonder if the following line is breaking the run for xc7k70tfbg676-2 parts? Admittedly though, it's not obvious to me how runs for Kintex parts ever happen with KINTEX_PARTS= in that file on master.
https://github.com/SymbiFlow/prjxray/pull/1794/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R188

2022-01-07T04:12:35 - xc7k325tffg900-2/005-tilegrid              - 1h15m: + /tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/build/tools/bitread --part_file /tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/database/kintex7/xc7k325tffg900-2/part.yaml -F 0x00000000:0xffffffff -o design.bits -z -y design.bit

2022-01-07T04:12:35 - xc7k325tffg900-2/005-tilegrid              - 1h15m: + python3 /tmpfs/src/github/symbiflow-prjxray-presubmit-db-kintex7/fuzzers/005-tilegrid/clb/../fuzzaddr/generate.py --oneval 1 --design params.csv --dword 0 --dframe 0

2022-01-07T04:12:35 - xc7k325tffg900-2/005-tilegrid              - 1h15m: make[2]: *** [run] Error 2

2022-01-07T04:12:35 - xc7k325tffg900-2/005-tilegrid              - 1h15m:

2022-01-07T04:12:35 - xc7k325tffg900-2/005-tilegrid              - 1h15m: --------------------------------------------------------------------------

2022-01-07T04:12:35 - xc7k325tffg900-2/005-tilegrid              - 1h15m: !Failed! @ 2022-01-07T04:12:35.091602 with exit code: 2

2022-01-07T04:12:35 - xc7k325tffg900-2/005-tilegrid              - 1h15m: --------------------------------------------------------------------------

2022-01-07T04:12:35 - xc7k325tffg900-2/005-tilegrid              - 1h15m: WARNING: Fuzzer failed!

Makefile:91: recipe for target '005-tilegrid/run.xc7k325tffg900-2.ok' failed

make[1]: *** [005-tilegrid/run.xc7k325tffg900-2.ok] Error 2

make[1]: *** Waiting for unfinished jobs....

This PR is obsolete now and can be closed.
The changes in this pull request already have been merged in https://github.com/f4pga/prjxray/pull/2046

The Kintex 325T support isn't really experimental anymore and has proven to work quite reliably.
So the title should be changed.
Unfortunately it looks like @jrrk2 currently does not actively develop this.
So we are waiting for the merge.

@jrrk2 Thanks!

is there any further progress on this? I'd love to see support for it! thanks

I think this PR might be stale. AFAIK, I made a PR for this which got merged, so you might just look into master to see if it is there.

@erikarn I just had a look, and it really seems it is not there yet.
So yes, this would need to be merged

@erikarn Now I remember, I also put in a PR for the K420/K480T parts,
and what is holding both those and this one back is that prjxray has not yet got the license server for those parts
working reliably, and that's why they CI cannot pass.
Nevertheless, these parts are working, and if you
want a ready made toolchain which supports there, look no further than https://github.com/openXC7/


Updated dependencies.

Description
npm audit resulted in over 5000 vulnerabilities prior to these changes. Despite the fact, that probably none of those actually pose a threat to this project, it'll be better to fix the vulnerabilities when possible.

Codecov Report

Merging #5 (fa07e7f) into development (4483b05) will not change coverage.
The diff coverage is n/a.

@@              Coverage Diff              @@
##           development        #5   +/-   ##
=============================================
  Coverage       100.00%   100.00%
=============================================
  Files               25        25
  Lines              486       486
  Branches            26        26
=============================================
  Hits               486       486

Flag
Coverage Δ

unittests
100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4483b05...fa07e7f. Read the comment docs.


Training the biencoder

I was trying to reproduce the code. However, I am facing issues with training the biencoder. Could you please tell how to finetune the biencoder on the datasets mentioned in the paper? Also, can you provide the WikilinksNED Unseen-Mentions dataset which you used? @fabiopetroni

@anshks We will update with an instruction on how to fine-tune the model shortly. For the WikilinksNED Unseen-Mentions dataset, we obtained it from the authors of this paper: https://ojs.aaai.org/index.php/AAAI/article/view/6380

@anshks We will update with an instruction on how to fine-tune the model shortly. For the WikilinksNED Unseen-Mentions dataset, we obtained it from the authors of this paper: https://ojs.aaai.org/index.php/AAAI/article/view/6380

@anshks We will update with an instruction on how to fine-tune the model shortly. For the WikilinksNED Unseen-Mentions dataset, we obtained it from the authors of this paper: https://ojs.aaai.org/index.php/AAAI/article/view/6380

+1
Waiting for fine-tuning note :)

@anshks We will update with an instruction on how to fine-tune the model shortly. For the WikilinksNED Unseen-Mentions dataset, we obtained it from the authors of this paper: https://ojs.aaai.org/index.php/AAAI/article/view/6380

@ledw any progress on this? I'm trying to train the bi-encoder, but can't find the details of how you trained it (e.g. scripts, hyperparameters).

It is indeed possible to train a model on your own custom dataset. I would recommend first training a zeshel (Zero-shot learning) model, then bring your data in the same format as zeshel and train your model.
Scripts to download and pre-process zeshel data set are in the repository.


Update requirements

Responding to @richrines1's comment on #70, I tested behavior with and without the cirq dev requirement. All of the checks still pass so creating this PR to remove the explicit dependency.

@richrines1 @perlinm yesterday I removed the explicit cirq dependence in SupermarQ and instead we just require css and qss. (@richrines1 and I checked that all the checks still pass with the versions of cirq that css requires) and then all of the checks were passing here, but once I merged into main it failed the Windows pytest -- why does this happen?

@teaguetomesh the same test passed when i reran it, so might have just been a weird fluke

Yeah it's a fluke. Let's be on the lookout if it happens again.


Ad Gitlab icon in the footer.

Please add gitlab icon in the footer like the github one and please add Gitlab pages support.

Is this commit to help you in my branch?
I try to write code for your issue.
https://github.com/xinolinx/xinolinx.github.io/commit/10866d9cd740197f0051f7985349980612d65a60
look like this.
https://xinolinx.github.io/
best regard,
Shinobu HUYUGIRI.

Did it, thanks mate!
https://epocopa.github.io/
You should write a PR ;)

I'm happy to help you :)