Rt tracking by state

Is your feature request related to a problem? Please describe.
Showing just case counts is informative but doesn't get at the root of the issue.
Describe the solution you'd like
It would be nice to see regional Rt estimates. We have an approach for doing this by region (https://github.com/epiforecasts/covid-regional/blob/b5cd746537dfc1f2cb15bef233342c4bfc1987d4/united-kingdom/update_nowcasts.R; https://github.com/epiforecasts/EpiNow/blob/master/R/regional_rt_pipeline.R) with an example here (https://epiforecasts.io/covid/posts/national/united-kingdom/)
Describe alternatives you've considered
Could roll your own implementation but there are some tricky details it is important to consider and naive implementations can have quite a few issues
Additional context
I am the lead author on the linked project. We are struggling with capacity at the moment so support may be an issue but I would be happy to review any dev work using our tooling.
Great work on the site - very clear, very useful.

Our national estimate may be of interest (note uses ECDC data and an international report delay): https://epiforecasts.io/covid/posts/national/india/

This is quite interesting work. Potential candidate for the deep-dive section. @gautam1858 please do have a look.

There is a website live to track daily rt estimates for US. https://rt.live/
They have opensourced the Jupyter book they are using to arrive at these values. https://github.com/k-sys/covid-19/blob/master/Realtime R0.ipynb
This can be a good starting point.

If this is based on the Instagram analysis then there are some statistical/epi issues with their approach. The website looks excellent however.

Is this being taken up? This could be the single most important metric to track and visualize current transmission rates both state wise and at a national level. http://systrom.com/blog/the-metric-we-need-to-manage-covid-19/
As pointed reference code is already available. This would be required to be calculated in the backend working on the state wise daily confirmed data.
Would love to contribute to fast track any effort if this can be prioritized/assigned.

happy to help consult from my prvs six sigma and stats experience!

I am not an statistics expert but, wanted to add what is being done in US. If anyone takes up that would be great. There is a data set at state-level in deep-dive repo of the organization. covid-19-india.csv.
One can use this data to build a model for Rt estimation. The same model can be extended to district-level if shape of data is similar.

thanks. data is very good though would have loved as many attributes to it
eg. city and even locality else it will be rework? Any idea on that
With the raw data a lot of stats hypothesis testing can be done and I can
help suggest visualise. HAve just got started wit hGithub on Udemy so a
long way to go to do myself :( any one willing to collaborate?
On Sun, 19 Apr 2020 at 07:04, Praveen T<EMAIL_ADDRESS>wrote:

I am not an statistics expert but, wanted to add what is being done in US.
If anyone takes up that would be great. There is a data set at state-level
in deep-dive repo of the organization. covid-19-india.csv
https://github.com/covid19india/deep-dive/blob/master/data/dataset/covid_19_india.csv
.
One can use this data to build a model for Rt estimation. The same model
can be extended to district-level if shape of data is similar.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/covid19india/covid19india-react/issues/1084#issuecomment-616002887,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APEWZWLBZMCR44XXJOT5VCTRNJILXANCNFSM4MHXDWXQ
.

If this is based on the Instagram analysis then there are some statistical/epi issues with their approach. The website looks excellent however.

@seabbs   Could you elaborate on the statistical issues?
Plugged in India data but haven't really done a deep dive on the underlying issues with the model.
Linked here - https://github.com/covid19india/covid19india-react/issues/1275#issuecomment-616025618

So, unfortunately, I have some work I have to finish today but here is a brief summary of the issues that leap out at me:

Use of smoothed reported cases. They use a gaussian smoother which includes future and past information to smooth current reported cases. This means future information is included when estimating the Rt and biases the curve. An alternative approach to this is to impute the symptom onset date using multiple samples.
Related issue. Presenting Rts by date of report. Interventions impact the Rt on the date of infection therefore interventions can only be compared to Rts by date of infection not report or symptom onset. A simplistic way to do this is to left shift with a more advanced method being to again sample from a distribution.
Another related issue. If you correctly map reported cases -> cases by date of onset then you have issues of right truncation (as cases may have onset but not reported). Again several approaches to dealing with this.
Use of a fixed 7-day smoothing window. This is arbitrary and window choice has been shown to introduce bias (people adjust to get the curve shape they want). The downside to a longer window is that real changes are delayed with the upside being to protect against statistical noise. An improved approach is to optimize the window (ideally within a sampling step) using some kind of proper scoring rule (for example RPS). This allows you to reduce the amount of smoothing and see changing dynamics earlier.
Fixed serial interval. The serial interval has a big impact and ideally, the level of uncertainty here needs to be fully propagated. The simplest approach is to use a distribution. However, this still doesn't not include all of the uncertainty. A better approach is to fit the serial interval using a bayesian approach and then use multiple samples from the posterior.

I probably missed something in there as well. We have dealt with all of the above in our analysis (that is by no means perfect) linked above.

@seabbs

So, unfortunately, I have some work I have to finish today but here is a brief summary of the issues that leap out at me:

Use of smoothed reported cases. They use a gaussian smoother which includes future and past information to smooth current reported cases. This means future information is included when estimating the Rt and biases the curve. An alternative approach to this is to impute the symptom onset date using multiple samples.
Related issue. Presenting Rts by date of report. Interventions impact the Rt on the date of infection therefore interventions can only be compared to Rts by date of infection not report or symptom onset. A simplistic way to do this is to left shift with a more advanced method being to again sample from a distribution.
Another related issue. If you correctly map reported cases -> cases by date of onset then you have issues of right truncation (as cases may have onset but not reported). Again several approaches to dealing with this.
Use of a fixed 7-day smoothing window. This is arbitrary and window choice has been shown to introduce bias (people adjust to get the curve shape they want). The downside to a longer window is that real changes are delayed with the upside being to protect against statistical noise. An improved approach is to optimize the window (ideally within a sampling step) using some kind of proper scoring rule (for example RPS). This allows you to reduce the amount of smoothing and see changing dynamics earlier.
Fixed serial interval. The serial interval has a big impact and ideally, the level of uncertainty here needs to be fully propagated. The simplest approach is to use a distribution. However, this still doesn't not include all of the uncertainty. A better approach is to fit the serial interval using a bayesian approach and then use multiple samples from the posterior.

I probably missed something in there as well. We have dealt with all of the above in our analysis (that is by no means perfect) linked above. There are also multiple limitations that we explore in depth in our methods so I won't go into them here. The key is that it does not account for bias due to changing testing efforts.

Appreciate the response! I believe they actually use a 9-day smoothing window which would amplify some of the error you mentioned. I was looking through the model you linked https://epiforecasts.io/covid/ - really nice work btw.  How do you handle changing rate of testing affecting the number of positive cases? Is this at all captured in the uncertainty? To me this sticks out as a missing piece in a lot of r_t models.
Also I was looking at S.Korea - https://epiforecasts.io/covid/posts/national/south-korea/ does this r_t number make sense given their number of cases is pretty low (<40 on the last date you computed for)

Can someone from project kind of map-out how this can be executed? While my knowledge of statistics is poor, I am very good at figuring out languages and getting things to run. I can work on updating the R pipeline shared by @seabbs to work with India data that is available in deep-dive repo. Or, I can do the same with Jupyter notebook from rt.live website.
Can someone volunteer in building UI for the same? If there are some requirements regarding how data should be reported so, it can be embedded into website.

I can join hands as have reasonable experience on stats and hypothesis testing. Nube on coding and UI. Happy to speak to coordinate better. My no. is<PHONE_NUMBER> and can coordinate over watsapp to speak.
Rgds
Saurabh

From: Praveen T<EMAIL_ADDRESS>Sent: Wednesday, April 22, 2020 4:21:50 PM
To: covid19india/covid19india-react<EMAIL_ADDRESS>Cc: Saurabh Sharma<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [covid19india/covid19india-react] Rt tracking by state (#1084)
Can someone from project kind of map-out how this can be executed? While my knowledge of statistics is poor, I am very good at figuring out languages and getting things to run. I can work on updating the R pipeline shared by @seabbshttps://github.com/seabbs to work with India data that is available in deep-dive repo. Or, I can do the same with Jupyter notebook from rt.live website.
Can someone volunteer in building UI for the same? If there are some requirements regarding how data should be reported so, it can be embedded into website.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/covid19india/covid19india-react/issues/1084#issuecomment-617703871, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APEWZWJWQX7PW4BNYOPOR7DRN3D4NANCNFSM4MHXDWXQ.

ok. I will start model from @seabbs. I will let you know if you I run into any issues that I encounter.

As mentioned, I had already done the modelling as per rt.live's model. And the results did have features which make qualitative sense. It's not needed to wait for a lot more data, though the model will remain sensitive to testing ramp ups even in future. I'm not an expert but I don't think it is a bad model to start with. We can incorporate changes and corrections suggested by experts, later subject to verification by peers.
For the time being I'm uploading some of the calculated data to covid19rt.in
I'm basing the calculations on daily state wise csv from this site. I'd love for it to be verified/cross checked, and corrections suggested. Also I'm myself not involved in frontend a lot so have just come up with a basic template. Would appreciate any help.

@gauravbgp Can we not get this integrated with covid19india website? Ideally in state page.

And can we also put the district level pls in any format. I am from Mumbai and guess some detailed analysis will help this city!

As mentioned, I had already done the modelling as per rt.live's model. And the results did have features which make qualitative sense. It's not needed to wait for a lot more data, though the model will remain sensitive to testing ramp ups even in future. I'm not an expert but I don't think it is a bad model to start with. We can incorporate changes and corrections suggested by experts, later subject to verification by peers.
For the time being I'm uploading some of the calculated data to covid19rt.in
I'm basing the calculations on daily state wise csv from this site. I'd love for it to be verified/cross checked, and corrections suggested. Also I'm myself not involved in frontend a lot so have just come up with a basic template. Would appreciate any help.

@gauravbgp, I noticed that you are plotting reproduction numbers for 16 states. I was wondering why you haven't included the remaining states. Also, the first plot is for 'TT', and I am having a hard time figuring out what that is. Lastly, while this project figures when and where to include Rt tracking, could you arrange yours in alphabetical order so that it is easier to navigate.
Thanks for your great work!

@seabbs @gauravbgp @praveentiru
Apologies for the delay in getting back to this issue. This is an imperative statistic as learnt from the above discussion, and rt.live has visualized it very well. We can do the same too, however, for someone who gets confused with a lot of numbers, I'd really appreciate a simple summary on the status of this issue, and how much accurate we are/can be with the data at hand.
I also see that @seabbs has given his suggestions and potential problems that we may run into when visualizing this. I'd also like to know how much of them can be circumvented. After that we can start with the implementation. I'm also bumping the priority of this issue to get things rolling.
We can sort the frontend part (rt.live uses d3 and we use them on the website too). We can display this for each state in the state page, and then also have a dedicated page that would have all the states for the purpose of comparison.

@jeremyphilemon Thanks for the reply. The current state is that @gauravbgp has implemented rt.live model from which he has built a site for India covidrt.in. @seabbs belongs to a team base at London School of Hygiene and Tropical Medicine. They manage site Epiforecasts where they track epidemics. He believes that there are flaws in model of rt.live
Which is better model? I do not have expertise in this domain to make a call. The way I see it we have following tasks.

[ ] Decide on model to use for rt prediction
[ ] Update the model to work with covid19india's database
[ ] Integrate the model into covid19india's update cycle (Ideally this model should be run daily after the information for that day is collated).
[ ] Update state's detail front-end to add visualization

People needed to complete task:

Person to assess and pick a model. Make it run with site's data model.
Person to deploy the model on scheduled basis to update data daily
Person to update site's front-end.

I can own up the second role. This is my view.

@seabbs @gauravbgp looks like there is a model update from rt.live on 23rd April.

@praveentiru thanks for succinctly summarizing the status. Sorry, I've been busy and unable to reply. I'll just mention my rationale for putting up the website first.
Firstly, the model is understood to be deficient in terms of being epidemiologically inaccurate in certain regions and being sensitive to changes in testing rates. But, it converts the epi strength to easy to understand ranges of Rt for layman, and casual trend analysis. Also as a metric to track the current progression, it is very similarly useful. Although the estimated Rt values vary slightly highly, and also in cases of initial backlog, Rt even shoots above 3 depending on states to explain cases (this is clearly epidemiologically incorrect) , it is still very correctly indicative of the efficacy of controls in a region, at least qualitatively, and is easy to track. Please have a look at http://covid19rt.in/ and judge for yourselves.
To this effect, I have endeavored to make end to end reports available first, and connect it to people (layman at large) so that they become aware that there is such an apt. metric which even they can use to track, and visualizes the status much better than other graphs available to the user.
rt.live has updated their model to include epiforecast's suggestions of onset delay and serial interval as a distribution. I also plan to update my model, but in some time. Currently, this is the same model rt.live have used and published reports for the US till 23/04 and upto a week after significant media coverage. And right now, I think the more people that can see the report the better.
As for including reports on covid19india.org . It is my understanding that the model needs sufficient centralization of control and would need maintenance efforts. In my workflow what I could do is once I have my processes sorted a little (~maybe 2-3 weeks), I can provide an API to download the generated Rt csv everyday. This can then be included in the site and visualized anyway. (Note: I myself have very less frontend touch, and am using almost no js at my site for now. Plots are png exported from matplotlib).
Seeing as how that might take some time, and how I think this report would be very useful for people, I would love if you think it would be appropriate to link to my webpage http://covid19rt.in from your site, since there are already many users.
If not that, at least individual users, if you find the analysis useful, it would be great if you could forward it in your groups.

@gauravbgp If you can package your model into a python program. I can work on packaging it to process data from covid19india API once a day and publish the result as an csv or, json as needed for visualization. I can look into visualization part but, it is not really my area of expertise. We can get it to closure if we have someone who is more capable here.

@praveentiru as mentioned even in the webpage, I am already using data from covid19india API. But as I have pointed out in my comment, it is the model that would need to keep undergoing modifications/validation and would also need to clean up and collate data from other sources about other facts.
You can add my no<PHONE_NUMBER> on whatsapp/telegram for any other faster offline communication/co-ordination as reqd.

any update on this activity? i have created a python version on kaggle using data from this site (covid19india.org)
Tracking done only for states where case count > 100
realtime tracking of Rt for Indian states

Hi all,
I have built R pipeline for using epiforecasts model for generating state level predications.
It generates analysis only for states which have at least one day where there are 40 cases in a day. Which gives analysis for 14 states which are: "AP" "BR" "DL" "GJ" "JK" "KA" "MH" "MP" "PB" "RJ" "TG" "TN" "UP" "WB".
@jeremyphilemon if you can specify the format in which you want data. (json and csv) to enable visualization of the same. I will own up the task of running the pipeline on daily basis to provide with fresh data. You can see the work of model here. We can leverage visualization of rt.live. That is something I need assistance with.

Hi
We have worked on predicting estimated cases, deaths and recoveries for next week in India. We are achieving < 2% error rate till now wrt actual cases. Feel free to explore it at
https://coronaindia.github.io/views/researchcentre.html
I do look forward to improving such forecasts and any suggestions or contributions are welcome.

Hi. went through the details. Great work by you and team! Any possibility
of including at district level pls? For example Mumbai will be very diff
prediction vs Maharashtra. city level details are found at
https://www.covid19india.org/ as we know best.
On Sat, 9 May 2020 at 23:09, Siddharth Srivastava<EMAIL_ADDRESS>wrote:

Hi
We have worked on predicting estimated cases, deaths and recoveries for
next week in India. We are achieving < 2% error rate till now wrt actual
cases. Feel free to explore it at
https://coronaindia.github.io/views/researchcentre.html
I do look forward to improving such forecasts and any suggestions or
contributions are welcome.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/covid19india/covid19india-react/issues/1084#issuecomment-626210928,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APEWZWOM6OJYJYAW7OUR5XLRQWIL7ANCNFSM4MHXDWXQ
.


Implement PEP 691

This obviously needs tests prior to landing, and of course PEP 691 itself needs to be accepted, but just an example of what implementing this could look like.
This should fully implement PEP 691, the only deviation is it adds a _last-serial key to the meta dictionary because PyPI had that on it's HTML pages (though it was in an HTML comment there).

This needs https://github.com/python/pypi-infra/pull/89 to not massively inflate the number of cached objects.


[ERROR] [MainProcess] [DB:connect:117] Unable to connect to  CONFIG SERVERS

Running on : CentOS Linux release 7.3.1611
With MongoDB version: 3.0.11
With mongodb-consistent-backup  version:	mongodb-consistent-backup version: 1.2.0, git commit hash: unknown
Python version: 2.7.5
Python modules:
Fabric: 1.14.0
pymongo: 3.5.1
multiprocessing: 0.70a1
yaml: 3.12
boto: 2.48.0
filechunkio: 1.8
execution failing within first few connection steps:
MongoDB 3.0.11 and mongodb-consistent-backup running on same host  or
mongodb-consistent-backup running on a remote host with MongoDB shell version: 3.2.8 produce
    the same error

[root@]# ./mongodb-consistent-backup --config /root/patrick/mongodb-consistent-backup.yml --verbose

	[2017-10-27 14:08:12,645] [DEBUG] [MainProcess] [Lock:acquire:20] Acquired exclusive lock on file: /tmp/mongodb-consistent-backup.lock
	[2017-10-27 14:08:12,645] [INFO] [MainProcess] [Main:init:144] Starting mongodb-consistent-backup version 1.2.0 (git commit: unknown)
	[2017-10-27 14:08:12,646] [INFO] [MainProcess] [Main:init:145] Loaded config: {"archive": {"method": "tar", "tar": {"compression": "gzip"}, "zbackup": {"binary": "/usr/bin/zbackup", "cache_mb": 128, "compression": "lzma"}}, "authdb": "admin", "backup": {"location": "/opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe", "method": "mongodump", "mongodump": {"binary": "/usr/bin/mongodump", "compression": "auto"}, "name": "default"}, "configPath": "/root/patrick/mongodb-consistent-backup.yml", "environment": "production", "host": "host11-mongo101.ubisoft.onbe", "lock_file": "/tmp/mongodb-consistent-backup.lock", "log_dir": "/var/log/mongodb-consistent-backup", "notify": {"method": "none"}, "oplog": {"compression": "none", "flush": {"max_docs": 100, "max_secs": 1}, "tailer": {"enabled": "true", "status_interval": 30}}, "password": "******", "port": 27017, "replication": {"hidden_only": true, "max_lag_secs": 10, "max_priority": 1000}, "rotate": {"max_backups": 5}, "sharding": {"balancer": {"ping_secs": 3, "wait_secs": 300}}, "upload": {"method": "none", "retries": 5, "rsync": {"path": "/", "port": 22}, "s3": {"chunk_size_mb": 50, "region": "us-east-1", "secure": true}, "threads": 4}, "username": "admin", "verbose": true}
	[2017-10-27 14:08:12,646] [INFO] [MainProcess] [Stage:init:31] Notify stage disabled, skipping
	[2017-10-27 14:08:12,647] [DEBUG] [MainProcess] [DB:connect:111] Getting MongoDB connection to host11-mongo101.ubisoft.onbe:27017 (replicaSet=None, readPreference=secondaryPreferred, readPreferenceTags=False, ssl=False)
	[2017-10-27 14:08:12,653] [DEBUG] [MainProcess] [DB:auth_if_required:126] Authenticating connection with username: admin
	[2017-10-27 14:08:12,664] [INFO] [MainProcess] [State:init:145] Initializing root state directory /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default
	[2017-10-27 14:08:12,665] [INFO] [MainProcess] [State:load_backups:162] Found 0 existing completed backups for set
	[2017-10-27 14:08:12,666] [INFO] [MainProcess] [State:init:131] Initializing backup state directory: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/20171027_1408
	[2017-10-27 14:08:12,666] [DEBUG] [MainProcess] [Lock:acquire:20] Acquired exclusive lock on file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/mongodb-consistent-backup_META.lock
	[2017-10-27 14:08:12,666] [DEBUG] [MainProcess] [State:write:73] Writing StateRoot state file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/mongodb-consistent-backup_META/meta.bson
	[2017-10-27 14:08:12,666] [DEBUG] [MainProcess] [Lock:release:30] Releasing exclusive lock on file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/mongodb-consistent-backup_META.lock
	[2017-10-27 14:08:12,666] [DEBUG] [MainProcess] [Lock:acquire:20] Acquired exclusive lock on file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/20171027_1408/mongodb-consistent-backup_META.lock
	[2017-10-27 14:08:12,667] [DEBUG] [MainProcess] [State:write:73] Writing StateBackup state file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/20171027_1408/mongodb-consistent-backup_META/meta.bson
	[2017-10-27 14:08:12,667] [DEBUG] [MainProcess] [Lock:release:30] Releasing exclusive lock on file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/20171027_1408/mongodb-consistent-backup_META.lock
	[2017-10-27 14:08:12,668] [DEBUG] [MainProcess] [Stage:init:48] Loaded stage mongodb_consistent_backup.Archive with task Tar
	[2017-10-27 14:08:12,668] [INFO] [MainProcess] [Stage:init:31] Upload stage disabled, skipping
	[2017-10-27 14:08:12,668] [INFO] [MainProcess] [Main:run:299] Running backup in sharding mode using seed node(s): host11-mongo101.ubisoft.onbe:27017
	[2017-10-27 14:08:12,671] [INFO] [MainProcess] [Sharding:get_start_state:68] Began with balancer state running: False
	[2017-10-27 14:08:12,674] [DEBUG] [MainProcess] [DB:connect:111] Getting MongoDB connection to shard1/host11-mongo101.ubisoft.onbe:10001,host11-mongo102.ubisoft.onbe:10001 (replicaSet=None, readPreference=primaryPreferred, readPreferenceTags=False, ssl=False)
	[2017-10-27 14:08:12,678] [DEBUG] [MainProcess] [DB:auth_if_required:126] Authenticating connection with username: admin
	[2017-10-27 14:08:12,691] [DEBUG] [MainProcess] [DB:connect:111] Getting MongoDB connection to shard2/host11-mongo201.ubisoft.onbe:10001,host11-mongo202.ubisoft.onbe:10001 (replicaSet=None, readPreference=primaryPreferred, readPreferenceTags=False, ssl=False)
	[2017-10-27 14:08:12,700] [DEBUG] [MainProcess] [DB:auth_if_required:126] Authenticating connection with username: admin
	[2017-10-27 14:08:12,714] [INFO] [MainProcess] [Sharding:get_config_server:188] Found sharding config server: host11-mongo101.ubisoft.onbe:10000,host11-mongo102.ubisoft.onbe:10000,host11-mongo103.ubisoft.onbe:10000
	[2017-10-27 14:08:12,714] [DEBUG] [MainProcess] [DB:connect:111] Getting MongoDB connection to host11-mongo101.ubisoft.onbe:10000,host11-mongo102.ubisoft.onbe:10000,host11-mongo103.ubisoft.onbe:10000 (replicaSet=None, readPreference=primaryPreferred, readPreferenceTags=False, ssl=False)
	[2017-10-27 14:08:17,722] [ERROR] [MainProcess] [DB:connect:117] Unable to connect to host11-mongo101.ubisoft.onbe:10000,host11-mongo102.ubisoft.onbe:10000,host11-mongo103.ubisoft.onbe:10000! Error: No servers available
	[2017-10-27 14:08:17,723] [CRITICAL] [MainProcess] [Sharding:get_config_server:200] Unable to locate config servers using host11-mongo101.ubisoft.onbe:27017: No servers available!
	[2017-10-27 14:08:17,723] [CRITICAL] [MainProcess] [Main:exception:207] Problem getting shard/replica set info! Error: No servers available
	[2017-10-27 14:08:17,723] [INFO] [MainProcess] [Main:cleanup_and_exit:163] Starting cleanup procedure! Stopping running threads
	[2017-10-27 14:08:17,723] [INFO] [MainProcess] [Sharding:restore_balancer_state:141] Restoring balancer state to: False
	[2017-10-27 14:08:17,747] [DEBUG] [MainProcess] [Stage:close:59] Calling close on backup stage mongodb_consistent_backup.Archive with task Tar
	[2017-10-27 14:08:17,752] [DEBUG] [MainProcess] [DB:close:212] Closing connection to: host11-mongo101.ubisoft.onbe:27017
	[2017-10-27 14:08:17,753] [INFO] [MainProcess] [Main:cleanup_and_exit:193] Cleanup complete, exiting
	[2017-10-27 14:08:17,753] [INFO] [MainProcess] [Logger:rotate:81] Running rotation of log files
	[2017-10-27 14:08:17,753] [INFO] [MainProcess] [Logger:compress:60] Compressing log file: /var/log/mongodb-consistent-backup/backup.default.20171027_1332.log
	[2017-10-27 14:08:17,754] [DEBUG] [MainProcess] [Lock:release:30] Releasing exclusive lock on file: /tmp/mongodb-consistent-backup.lock
	[root@host11-mongo101 patrick]#

YML file is attached.

mongo-consistent-backup.txt

Hello @patricklarochelle,
This sounds related to issues people are seeing with non-replset-based Config Servers (SCCC mode) in 3.2 and older. There is an issue here to re-test the code with the old style config servers: https://github.com/Percona-Lab/mongodb_consistent_backup/issues/203.
Importantly, due to the nature of non-replset-based (SCCC mode) config servers not having any consistent changelog/oplog the support for SCCC mode may be removed in the future as we cannot guarantee a point of consistency without an oplog. See more about this here: https://github.com/Percona-Lab/mongodb_consistent_backup/issues/233.
As SCCC mode is supported as of today I will test and attempt to fix this problem, however I do suggest that you consider moving to 3.2+ with CSRS (replset-based) config servers as the support for SCCC may go away, plus there are tonnes of other benefits to using CSRS!
Stay tuned!
Tim


PEM read-side refactor

WIP partially since we still have to make up our mind how strict we want the parser to be (and with flags, we even can offer multiple options).
See also RT4698

My comments on RT4698 still apply - I don't quite understand the state change to "rejected but issue tracked on (this) PR".
Whitespace (anywhere) and odd line lengths are common UI issues.  Non-base64 chars have pluses and minuses.  For the former, consider an e-mail that gets quoted with '>' when forwarded.  The latter are described in the RFC.
As I mentioned in the RT, the OpenXPKI project has encountered this issue; see openxpki/openxpki/issues/437.
I maintain the Perl CPAN module Crypt::PKCS10, where I parse and output CSRs and public keys.
When parsing, I remove all whitespace, optionally remove non-base64 characters, and treat the result as one long string.   That's a lot simpler than insisting on 64 character lines (except the last).    And it accepts all valid input.  Of course, that's Perl.  But in C code it's easy to do the same thing by processing one character at a time.
Errors are best caught when parsing the asn.1 (and validating signatures), not by enforcing a strict punch-card era text format.  I recommend this.
For output, of course, I produce strictly compliant PEM.
In other applications, it's a bit more involved to detect the encryption header - but once past that, I use the same approach.
Bottom line, I encourage the Unix "accept liberal, produce conservative".  I'd settle for an option, but I don't think it's necessary - and OpenSSL has enough of those as is.

First off, I never said there weren’t interop concerns.  I am sorry if my quibbles/jokes over MUST sent the conversation down the wrong chute.
The ticket is closed because we’re moving to GitHub.  Please raise your concerns here, or perhaps Ben has addressed them.

I understood the spirit of your comments, no problem.  I literally didn't understand why 'rejected' was the chosen state for the RT - but that's why I commented here.  'rejected' is usually a strong dismissive statement from a maintainer.
The original purpose of this PR was addressing refactoring and choice of memory pools, not the issue I raised.  I'm certainly OK with merging them for convenience; I'm grateful that Ben volunteered.
At first glance, Ben has re-worked the existing code, but not liberalized the parsing.  His comment at the top of this thread says "WIP", "we still need to decide..."
The code seems to have a lot of legacy and complexity; I haven't done a detailed review but applaud the courage in touching it.  If you take my advice, it could well get simpler.
I think my concerns are adequately listed here (incorporating by reference the RT thread).
I'll watch this thread and am willing to participate if that's useful.

I suppose I will use "resolved" in closing the others.  Thanks!

It is WIP largely because we need to decide what we want the behavior to be.
Having flags gives us options; there could be a flag for bug-for-bug compatibility, for ignoring all whitespace, all non-b64 characters, for enforcing line lengths, etc.
I would like some feedback from the team about which features are desired and what the defaults should be.
Unfortunately, I injured my hand yesterday and cannot type out the full list of open questions as I see it.

Yes, flags provide options - but also complexity for users, maintenance and testing.
I settled on one option - whether to ignore non-b64.  While I don't think using them as a covert channel is a serious risk in OpenSSL data, it's a simple accommodation and sometimes catches corruption.  e.g. '=' in the middle of  a string can happen when someone copies quoted printable...
Whitespace is harmless.  So are random line lengths.
I don't have a case for bug-for-bug compatibility here.  Currently, valid data aborts processing.
I think I saw that OpenSSL will sometimes accept PEM data without the END label.  If someone relies on that, I'd have little sympathy.  That's the only case that comes to mind of OpenSSL's PEM processing being too liberal.
One could easily implement a separate tool for 'is this strict PEM?'.  Anyone who might want to validate that they're producing strict PEM would be better served by such a tool than by OpenSSL's idiosyncrasies...  And I think that reducing the options in OpenSSL code would be a good thing.
Of course once you accept liberal input, openssl x -in foo -out bar will produce strictly compliant PEM.
Hope your injury heals soon.

There is code in openssl that depends on some of the current behavior.  I forget details.  See the issue/PR from AGL that code a beer-mug in it.

#1209

Cute, but I know of no RFC that considers '-' as a comment character.  That's the 'covert channel' argument from the RFC.  Note that Google reports 'no ill effects' from its removal.
Most of the image could be retained with "ignore Base64" - only 7 distinct characters in the image are valid Base64.  oOCQCG0 There are plenty of other graphics with which to replace them.
Why would asn1parse care about the PEM?  It should get binary the same way that all PEM consumers do, and parse the binary.  If it doesn't, it should - if for no other reason than that as a debug parser, it should see the same bits as everything else.

Should doesn't always equal the way things work, and considerations of the deployed based.

By 'should', I meant that if necessary, simplifying asn1parse to use the new input routines becomes part of this work - or maybe it's a separate bug against asn1parse.
A diagnostic tool can be evil if it sees different bits from the real consumer.
Is the '-' comment line behavior documented?  Is that really what allows the image - it looks as though it may be relying on the 'if the line is not 64 characters long, the PEM is over' logic?
Anyhow, my 3 cents is that if someone relies on this, it's not a hardship to deal with it externally
e.g.  sed -e'/^-/d' file.pem | openssl foo
Compatibility is important - but it's always a trade-off.
I've said enough.  I'll yield to your team's judgement - my primary concern is that OpenSSL accept RFC-compliant input.

what's going on with this now?

I timed out on waiting for feedback about what flags the dev team was interested in seeing, and was going to default to implementing what @tlhackque wanted and the secure memory flag and remove the WIP label, but then started blocking on time to actually do that work.
It seems somewhat unlikely that time to do that work will appear this week while I'm at the IETF.

Might be worth trying to start discussion on the tls-dev list and perhaps close this.

Rebased to lateset master, with some updates: renamed flag from "weak eol" to "eay compatible", since that's what it's intended to be, and added a b64-only (i.e., "strict") option, with the default behavior being to treat control characters as whitespace (i.e., ignored) ... which means that ASCII non-base64 characters get passed through to EVP_Decode, for now.
This is all up for discussion, of course, but I removed the WIP label since I am actually proposing this change.

I've just had another look at this work (prompted by @kaduk's comment in #2746)...  and I wonder, are there thoughts of having the resulting EVP_PKEY et al from PEM_read_bio_PrivateKey and friends in secure memory as well?

are there thoughts of having the resulting EVP_PKEY et al from PEM_read_bio_PrivateKey and friends in secure memory as well?

Probably makes sense as a later addition ... there are several codepaths that could be dispatched to by that function and it's not clear that it would need to be done at the same time.  Thanks for pointing it out!

Later addition is fine, I was only asking about the intention going forward.

Rebased and fixed a couple needless behavior changes exposed by 04-test_pem.t.

Apart from my remarks of the changes, you need to fix the documentation further, as noted by find-doc-nits:
(cd .; /usr/bin/perl util/find-doc-nits -n ) >doc-nits
if [ -s doc-nits ] ; then cat doc-nits; rm doc-nits ; exit 1; fi
doc/man3/PEM_bytes_read_bio.pod:1: missing copyright
doc/man3/PEM_read_bio_flags.pod:1: PEM_FLAG_SECURE missing from NAME section
doc/man3/PEM_read_bio_flags.pod:1: PEM_FLAG_EAY_COMPATIBLE missing from NAME section
doc/man3/PEM_read_bio_flags.pod:1: PEM_FLAG_ONLY_B64 missing from NAME section
doc/man3/PEM_read_bio_flags.pod:1: missing copyright
doc/man3/PEM_read_bio_flags.pod:1: period in NAME section
make: *** [doc-nits] Error 1

Also, you need to rebase on a fresher master

Looking at the commits again caused me to feel uneasy about the lack of test coverage for PEM_FLAG_ONLY_B64, which turns out to have been justified.
As it stands, that flag is not usable with encrypted files, though it would take some thinking to determine if ther's a way to fix that that does not involve a second pass over things.

Looking at the commits again caused me to feel uneasy about the lack of test coverage for PEM_FLAG_ONLY_B64, which turns out to have been justified.
As it stands, that flag is not usable with encrypted files, though it would take some thinking to determine if ther's a way to fix that that does not involve a second pass over things.

Depends what you mean with "encrypted".  If "encrypted by PEM headers", then yes, you're right, but if it's encrypted in a PKCS#8 container, then PEM_FLAG_ONLY_B64 should be possible to use, no?

"encrypted with PEM headers", yes.

There are several ways you can do that...  one idea is that you can sanitize with flags & (found_header ? ~0 : ~PEM_FLAG_ONLY_B64), and you can also detect if your first body line is a header by looking for a colon (:), and if not, allow sanitize to see PEM_FLAG_ONLY_B64 already then.

You seem to have upset the CIs.  Most importantly, the Appveyor ones show this:
Error processing -s_key keyU.ss
804:error:0909006C:PEM routines:get_name:no start line:..\crypto\pem\pem_lib.c:776:Expecting: ANY PRIVATE KEY
804:error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:..\ssl\ssl_rsa.c:550:
ssltest_old.exe -s_key keyU.ss -s_cert certU.ss -c_key keyU.ss -c_cert certU.ss -bio_pair => 1

But only when building shared!

It may be hard to track down the AppVeyor issue without a windows machine; the file it's complaining about is generated during the build with "..\apps\openssl.exe req -config ....\test\Uss.cnf -out reqU.ss -keyout keyU.ss -new > err.ss".
The error indicates that libcrypto is getting an error or zero-length input when trying to read the first line of the file to look for the "-----BEGIN FOO-----" name line.
But it looks like this code is doing essentially the same operation as the historical code -- BIO_gets() and checking if the return value is less than or equal to zero.
I guess I'll push my other changes and give it another crack, at least.

I'll have a look on a VMS system, that can sometimes trigger the same kind of faults as Windows...

Thanks for trying it out on VMS.
FWIW, I put together a prototype for what it could look like to use ':' to detect the PEM header, at https://github.com/kaduk/openssl/commit/03434ffaec6c5cd7cddbfef147afc763c4c5df76 (my 'pem2' branch).  But I didn't dig back into RFC 1421 to confirm that it should always work, so it's not part of this pull request.

But I didn't dig back into RFC 1421 to confirm that it should always work

From RFC 1421:
       Pre-Encapsulation Boundary (Pre-EB)
           -----BEGIN PRIVACY-ENHANCED MESSAGE-----

       Encapsulated Header Portion
           (Contains encryption control fields inserted in plaintext.
           Examples include "DEK-Info:" and "Key-Info:".
           Note that, although these control fields have line-oriented
           representations similar to RFC 822 header fields, the set
           of fields valid in this context is disjoint from those used
           in RFC 822 processing.)

       Blank Line
           (Separates Encapsulated Header from subsequent
           Encapsulated Text Portion)

       Encapsulated Text Portion
           (Contains message data encoded as specified in Section 4.3.)

       Post-Encapsulation Boundary (Post-EB)
           -----END PRIVACY-ENHANCED MESSAGE-----

                   Encapsulated Message Format
                            Figure 1

Note "line-oriented representations similar to RFC 822", which indicates key:value form of lines.  The first line must have a colon to be such a header.

Unfortunately, VMS wasn't much of a help...
@dot-asm, do you have the possibility to check what goes wrong with this?
(@kaduk, you will have to be patient...  Easter is a 4-day holyday in Sweden, Friday through Monday, so don't expect too much from me)

Managed to scrounge up a windows VM with which to reproduce.
The error message is somewhat confusing, as it is generated after scanning through the entire file looking for a start line and hitting EOF.
Somehow, after sanitize_line() we are ending up with spurious whitespace at the end of the line, so that we are comparing "---- \n" to TAILSTR (i.e., "-----\n"), which unsurprisingly fails.  More investigation and (presumably) updated patch pending...

In retrospect, this is an unsurprising natural consequence of CRLF line endings, and conditionals that explicitly compare against '\n'.  The extra '\r' gets treated as a control character and forced to whitespace, which is then trailing whitespace at the end of the line and causes the check to fail.

Oh, appveyor needs a rebase before it will even try building?  Interesting.

It looks like I may need to push the squashing off until tomorrow, but we can let appveyor have a crack at the new code at least.

Hmm, the pyca tests are failing, in a way plausibly related to how this change makes the format requirements for private keys slightly more strict.  How much do we care?

We care very much. Breaking what used to work is definitely not allowed in this release.

Hmm, I think it's the same 3 PYCA tests that failed before and after adding PEM_FLAG_EAY_COMPATIBLE to the PrivateKey loading (via PEM_bytes_read_bio_secmem()), but didn't take notes on the original travis run.
Just to be clear, the failing cases look to be named "test_corrupt_traditional_format_der", "test_wrong_private_format", and "test_wrong_public_format", which is why I'm not 100% convinced that this is a case where we need to retain bug-for-bug compatibility.  But, I don't think I'll be able to say more about why these cases are failing without running them locally, which will take some time (unless @jon-oracle decides to jump in, but I do not plan to wait for that to happen).

Ah, just having the pyca source to poke at is helpful, in that it lets me understand what the travis output is trying to tell me.  The test runs
        with raises_unsupported_algorithm(
                _Reasons.UNSUPPORTED_PUBLIC_KEY_ALGORITHM):
>           load_pem_public_key(key_data, backend)
``` (the ```>``` indicates that's where we are in the call chain, and the other stuff is context), indicating that it expects to get a "Unsupported public key algorithm" exception, as this operation is in fact supposed to fail.  But, the code to handle errors from the OpenSSL call is doing:

    elif errors[0][1:] in (
        (
            self._lib.ERR_LIB_ASN1,
            self._lib.ASN1_F_ASN1_CHECK_TLEN,
            self._lib.ASN1_R_WRONG_TAG
        ),
        (
            self._lib.ERR_LIB_PEM,
            self._lib.PEM_F_PEM_READ_BIO,
            self._lib.PEM_R_NO_START_LINE
        ),
    ):
        raise UnsupportedAlgorithm(
            "Unsupported public key algorithm.",
            _Reasons.UNSUPPORTED_PUBLIC_KEY_ALGORITHM
        )

that is, inspecting the error library, function and reason code.  But, the function code has changed, since the function is now PEM_read_bio_flags(), so this check does not trigger, and things fall through to the else clause that just does ```raise ValueError("Could not deserialize key data.")```, which is not the expected exception in the test function, so is counted as a failure.

So, I guess it's time to address the elephant in the room: are error function codes part of the stability guarantee?  Attempting to make them be part of it would seem to run into direct conflict with ```make update```, which insists that the function code matches the name of the function it's being used in; that in turn would prevent us from making an existing function into a wrapper around a new function with extended functionatliy.  But probably this discussion is better held on openssl-dev than in an already-long github comment thread; I'll work on drafting an email.

So, I guess it's time to address the elephant in the room: are error function codes part of the stability guarantee? Attempting to make them be part of it would seem to run into direct conflict with make update, which insists that the function code matches the name of the function it's being used in; that in turn would prevent us from making an existing function into a wrapper around a new function with extended functionatliy. But probably this discussion is better held on openssl-dev than in an already-long github comment thread; I'll work on drafting an email.

Well, I think that most of the time, we've considered the library and reason code combo to be reasonably stable, but the function code can vary.  I'd say that the pyca stuff is overly sensitive in its checks of error data.

... on the other hand, who are we to say how others use our public stuff?

But yeah, all in all, I'd like to consider the check of the function code a bit over the top...

I just had a look at the pyca/cryptography, and it looks like they have updated their code, the checks in _handle_key_loading_error are different.  Maybe we should update our sub-module...

So how about this, either patch your branch with the contents of #3346 and see if that helps you, or wait 'til #3346 has been merged, rebase on top of that and try again?

#3346 is now merged, feel free to rebase on a fresher master and see if that helps matters...

#3346 is now merged, feel free to rebase on a fresher master and see if that helps matters...

Having compared the pyca sources before and after, it looks like it should help.
Running a local test build now before pushing...

Sigh, pyca/cryptography/vectors/cryptography_vectors/asymmetric/Traditional_OpenSSL_Serialization/key1.pem has trailing whitespace on the "END RSA PUBLIC KEY" line, so I cannot sneak in the lack of PEM_FLAG_EAY_COMPATIBLE for PEM_read_bio_secmem(), alas.

Yeah, enhanced strictness will have to wait... Also, not sure we should ever. "Be generous in what you accept" 'n all that.

Requiring a new flag to be used to parse old files that used to work is not acceptable.
A PEM_PARSE_STRICT flag would be okay, for example.

@richsalz with the latest commit, the behavior of all previously existing public APIs remains unchanged.  If you use the  new public API and do not specify PEM_FLAG_EAY_COMPATIBLE, you get the more-strict behavior.  Does that suffice for your acceptibility criteria?
(I would like to remove the EAY-compatible stuff in some future major release, but clearly that is not okay for 1.1.1.  Most likely path would be to drop the flag from the aforementioned previously existing public API's calls to the _flags variant for 1.2, so people needing the behavior could switch to calling the flags variant directly as a transition plan, then remove the compat behavior entirely in 1.3)

As for the elephant, remember when I tried to remove unused function codes and we decided we can't?

Removing the function codes entirely would break people's build.
Just not returning them anymore is somewhat different.

(The many red 'x'es were due to a change in my local copy that didn't make it into a commit.)
Rebased, squashed, and attempted to take all feedback so far.

And rebased again so the CI will actually take a look.

This should be taken out of WIP at this point, no?

This should be taken out of WIP at this point, no?

I could have sworn that I already had done so; maybe it reverted back somehow (on rebase?).
Regardless, removed WIP from the title.

Ah, the history feed has
 richsalz changed the title from PEM read-side refactor to WIP PEM read-side refactor 6 days ago

Shrug.

@levitte it seems that Rich is happy now, and the WIP is gone; can you take another look?

@richsalz, wanna merge or shall I?

go  ahead.

Merged.  204afd81b12c71d625e89599c0eef33588afc1f0 to fa3ed5b2c2b508a6444124fdf12ecbb4898007ed


dashboard stat numbers white on yellow is hard to see

When I go to my dashboard
And I have files statistics
I should be able to easily read them

You know that light gray could be darkened, too.

@mtribone good idea!


Program, if left untouched for a while, will freeze when attempting usage

I have created a program that uses a SQL query to populate a dropdown list with records from the database. It allows the user to print a label from the database to a label printer by running another query using the item they selected and writing the results to a printable text file. However, if I start the program and do not use it for approximately 10 minutes and decide to use it later, the program freezes immediately when I choose to click any button or other interactive object. I'm not sure where the issue is rooted. Is this just an issue with InForm or is it more likely an issue with my code? My code also fires PowerShell scripts for an open file dialog, a print dialog, and two data entry scripts. If I use it constantly without taking long breaks, the program works fine. It only freezes if I choose to wait about 10-15 minutes between interactions.

You could try creating a simple program with a few controls without all the bulk of your full-blown program and let it rest on your desktop, to check whether it's on InForm's end or not. Please let me know your OS and QB64 version as well. I haven't had it happen to me. One of my programs is a simple database but it doesn't use SQL, so it doesn't crash on me (not saying it's that, but something could be interacting weirdly there). Please let me know.

Thank you for your response. I'll check it out on my end here and see what is causing it.


update qiskit 1 0 0

Description
Reopen https://github.com/CQCL/pytket-qiskit/pull/279 to run the tests
Checklist

[ ] I have performed a self-review of my code.
[ ] I have commented hard-to-understand parts of my code.
[ ] I have made corresponding changes to the public API documentation.
[ ] I have added tests that prove my fix is effective or that my feature works.
[ ] I have updated the changelog with any user-facing changes.

Needed to change the backbend in the not working test. Hope it works now.


feat(matomo): add dev matomo

:tada: This PR is included in version 1.44.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


SAN-73 Genericise lfp specific logic in api

Genericise lfp specific logic
Add yaml file to load different penalty types

I will create a new pull request for this


glibc: Missing mtrace script

https://command-not-found.com/mtrace
https://man7.org/linux/man-pages/man1/mtrace.1.html

mtrace is a perl script that comes with glibc that processes the output of the mtrace() function, showing malloc/free invocations.
It seems that mtrace is missing from the nixos packaging of glibc.
It would be great to have this packaged.

Thank you @Ma27 !


[UTILS-114] Add abstract duplicate method to IntervalArray.

Resolves #114.

Coverage decreased (-0.07%) to 54.273% when pulling 3435376befeaddc6e814e2b60268673b43077f23 on fnothaft:issues/114-copy into 2431d6c4bb1f2690d57321a43f16f6f9ec13eead on bigdatagenomics:master.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/bdg-utils-prb/128/
Test PASSed.

I tested this with https://github.com/bigdatagenomics/adam/pull/1440 and it works for me locally, but of course https://github.com/bigdatagenomics/adam/pull/1360 also worked for me locally.
How should we sequence these changes?  Do we need bdg-utils newer than 0.2.11 for ADAM release 0.22.0?

How should we sequence these changes? Do we need bdg-utils newer than 0.2.11 for ADAM release 0.22.0?

We don't need it, but it'd be a nice to have due to https://github.com/bigdatagenomics/utils/pull/113.
WRT sequencing, I'd say let's merge this in here. For 0.22.0, I'm guessing you don't want to take all of the changes in https://github.com/bigdatagenomics/adam/pull/1360? If so, what I'd propose is I could then take https://github.com/bigdatagenomics/adam/pull/1440 and cut out everything except for the utils bump.

Agreed, #113 is worth having in 0.22.0. How about

merge this,
cut bdg-utils 0.2.13,
create a new pull request on ADAM with commit https://github.com/bigdatagenomics/adam/pull/1440/commits/041b4f201d1f3dfedec206c1588d27f71d006e1a updated for bdg-utils 0.2.13,
and then after that gets merged to ADAM I'll cleanup and squash https://github.com/bigdatagenomics/adam/pull/1360 separately for possible inclusion in 0.22.0.

I'mma slightly reorder that to:

merge this,
update adam#1440 as you said, against the snapshot
if all is good, cut bdg-utils 0.2.13
then update adam#1440 to the release
and then after that gets merged to ADAM @heuermh will cleanup and squash bigdatagenomics/adam#1360 separately for possible inclusion in 0.22.0.

but otherwise SGTM.

+1


Components / Templates - tweak captions, should be black and grey text, no bold

captions need a little fixing. should match mocks, i can also try to play with this one myself.

I think i fixed some of them with the Typography classes. Let me know if this still needs more fixing


Next export takes a lot of time for ~240 pages

We use next.js for building our website (obviously).
Our website generates dynamic landingpages. This data comes from contentful.
We now count over 240 different pages.
Each page is staticly generated in our deployment build using the exportPathMap.
Our problem now is that each page uses time to generate.
So each new page is adding up to our build time.
We started with a few pages and about 1:30min build time in travis.
Now we're at about 6min 😕
The exporting part of the build is for ~240 pages 181s
To Reproduce
Steps to reproduce the behavior, please provide code snippets or a repository:

Have about ~240 pages
next build && next export

Expected behavior
We'd like that the pages would not been been generated serial but parallel to reduce the time to export
System information

Node: 10.9.0
Next.js: 7.0.2

Feel free to revive this PR: https://github.com/zeit/next.js/pull/4765

Besides that there's this PR: https://github.com/zeit/next.js/pull/5597 that I merged yesterday

@timneutkens
I saw the multi-threading PR but could't find it while writing this issue ^^
#5597 sounds really good. Can I use it in the lates canary?

#5597 sounds really good. Can I use it in the lates canary?

Yes, you'll have to cache node_modules/.cache and .next/records.json

Fixed in #5870


Add __toString() method to File validation rule

In case #46849 is actually something that can be fixed, this is a possible solution for it.

I'm not sure this is really the right solution? We don't really want to convert it to a string. We want it to be preserved as a rule object so that it's passes method is called, etc. Probably better to fix the rule parsing logic.

Hey @taylorotwell thanks for the review.
From my understanding you will be getting the string only when you require a string representation of the class.
As far as I understood is how other validation rules work.
Anyway do you plan to work on this or should I attempted the fix on the rule parsing logic?
Cheers


need station anchors

physical machine or structure that when powered/activated makes the grid way more massive and very hard to move
good excuse for big space anchor as well, maybe some sort of animation where it lowers down into space or something whatever
this is needed so shitters in shuttles dont giga move the station around but also can be toggled for sabotage or whatever else
ive been requesting this for years but never amde an issue on it proper

Could this just be added to the big Gravity Generator?

I agree, station gets annoying to be near as both crew and nukies
Would thematically make sense to tie this to grav gen, but probably better to map multiple subspace anchors for redundancy

im excited to map these when they exist

Wonder if this is as simple as adding a structure with an absurdly large mass number...wonder if it passes it through to the grid or if grid weight is purely based on tiles. Not checking but nothing is ever simple :'(

Wonder if this is as simple as adding a structure with an absurdly large mass number...wonder if it passes it through to the grid or if grid weight is purely based on tiles. Not checking but nothing is ever simple :'(

Motion damping for has already been added to ShuttleComponent on #18697 so I think this would only need to create a new device that can override that value for a specific grid

Also ideally you'd split out the gravity generator UI as I think that sort of "delayed turn-on / off" functionality is useful for here + IFF.
Will also need for the trade station.

okay after trade station got merged, we REALLY need this. People are literally driving the outpost to the station and then not even using the cargo shuttle to travel.

People have already started to dock them to the station permanently by adding a shuttle console the the outpost and extending the cargo dock.. omg.

Suppose we add an anchor to the trade outpost. Won't the same people just turn it off? If this is no longer a convenience feature but a gameplay limitation, it needs to be either added directly to the map prototypes (this can be done right now, I think), be a result of the station grid's "mass",  or there needs to be some compelling downside to turning off the anchors

You could set the station anchor on the trade station to require centcomm access 😄

Make sure that the Station's anchor also requires Captain access so the QM can't turn it off :D

We have anchors now.


ubuntu 24.04 ipv6 waitforx: Timed out waiting for RandR outputs

xrdp version
0.10.80
Detailed xrdp version, build options
xrdp 0.10.80
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-ipv6
      --enable-fuse
      --enable-jpeg
      --enable-rfxcodec
      --enable-mp3lame
      --enable-vsock
      --enable-painter

  Compiled with OpenSSL 3.0.13 30 Jan 2024

Operating system & version
Ubuntu 24.04.1 LTS
Installation method
Doesn't matter
Which backend do you use?
xorgxrdp
What desktop environment do you use?
No response
Environment xrdp running on
LG gram14 notebook，Intel® Core™ i5-8265U × 8，Software Rendering
What's your client?
MS offical client
Area(s) with issue?
Session manager (sesman)
Steps to reproduce
The operating system is ubuntu 24.04, compiled and installed the latest xrdp software. 24.04 has automatic ipv6 enabled by default, and the xrdp connection is unsuccessful. xrdp-sesman.log reports an error: waitforx: Timed out waiting for RandR outputs. this time, use the following command in /usr/local /libexec/xrdp/waitforx -d :10 command, sometimes the connection is successful.
Use the command sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1 to turn off ipv6 on all interfaces and connect successfully without restarting the xrdp service.
✔ Expected Behavior
Would like to be able to use xrdp in an ipv6 environment.
❌ Actual Behavior
Under ubuntu 24.04, IPv6 must be turned off to use xrdp.
Anything else?
xrdp-sesman.log:
[2024-11-01T12:11:56.463+0800] [INFO ] Socket 13: connection accepted from AF_UNIX
[2024-11-01T12:11:56.470+0800] [INFO ] Received system login request from xrdp for user: jackwang IP: <IP_ADDRESS>
[2024-11-01T12:11:56.474+0800] [DEBUG] Closed socket 13 (AF_UNIX)
[2024-11-01T12:11:56.474+0800] [DEBUG] Calling exec (excutable: /usr/local/libexec/xrdp/xrdp-sesexec, arguments: xrdp-sesexec)
[2024-11-01T12:11:56.485+0800] [INFO ] starting xrdp-sesexec with pid 18114
[2024-11-01T12:11:56.527+0800] [INFO ] TerminalServerUsers group tsusers doesn't exist. Access granted for jackwang
[2024-11-01T12:11:56.530+0800] [INFO ] Access permitted for user: jackwang
[2024-11-01T12:11:56.532+0800] [DEBUG] Closed socket 9 (AF_UNIX)
[2024-11-01T12:11:56.532+0800] [INFO ] Received sys login status for jackwang : logged in
[2024-11-01T12:11:56.549+0800] [INFO ] Received request from xrdp to create a session for user jackwang
[2024-11-01T12:11:56.553+0800] [DEBUG] session_list_get_bydata: search policy=UB type=Xorg U=1008 B=24 D=(2560x1440) I=<IP_ADDRESS>
[2024-11-01T12:11:56.556+0800] [DEBUG] session_list_get_bydata: No matches found
[2024-11-01T12:11:56.560+0800] [DEBUG] Did not find a running X server at /tmp/.X11-unix/X10
[2024-11-01T12:11:56.564+0800] [DEBUG] Did not find a running X server at /tmp/.X10-lock
[2024-11-01T12:11:56.567+0800] [DEBUG] Closed socket 15 (<IP_ADDRESS>:5910)
[2024-11-01T12:11:56.570+0800] [DEBUG] Did not find a running X server at 5910
[2024-11-01T12:11:56.573+0800] [DEBUG] Closed socket 15 (<IP_ADDRESS>:6010)
[2024-11-01T12:11:56.576+0800] [DEBUG] Did not find a running X server at 6010
[2024-11-01T12:11:56.579+0800] [DEBUG] Closed socket 15 (<IP_ADDRESS>:6210)
[2024-11-01T12:11:56.582+0800] [DEBUG] Closed socket 13 (AF_UNIX)
[2024-11-01T12:11:56.636+0800] [DEBUG] Waiting for X server to start on display :10
[2024-11-01T12:11:56.637+0800] [DEBUG] Calling exec (excutable: /usr/local/libexec/xrdp/waitforx, arguments: /usr/local/libexec/xrdp/waitforx -d :10)
[2024-11-01T12:11:56.640+0800] [INFO ] Starting X server on display 10: Xorg :10 -auth .Xauthority -config xrdp/xorg.conf -noreset -nolisten tcp -logfile
.xorgxrdp.%s.log
[2024-11-01T12:11:56.644+0800] [DEBUG] Calling exec (excutable: Xorg, arguments: Xorg :10 -auth .Xauthority -config xrdp/xorg.conf -noreset -nolisten tcp
-logfile .xorgxrdp.%s.log)
[2024-11-01T12:11:56.644+0800] [DEBUG] waitforx: Opening display :10. Attempt 1 of 10
[2024-11-01T12:12:26.645+0800] [ERROR] waitforx: Timed out waiting for RandR outputs
[2024-11-01T12:12:26.650+0800] [DEBUG] waiting for pid 18130 to exit
[2024-11-01T12:12:26.653+0800] [ERROR] Timed out waiting for X server
[2024-11-01T12:12:26.668+0800] [DEBUG] Closed socket 9 (AF_UNIX)
[2024-11-01T12:12:26.671+0800] [INFO ] Session on display 10 has finished.
[2024-11-01T12:12:26.673+0800] [DEBUG] Closed socket 15 (AF_UNIX)
[2024-11-01T12:12:26.675+0800] [DEBUG] Closed socket 14 (AF_UNIX)
[2024-11-01T12:17:39.952+0800] [INFO ] sesman_main_loop: sesman asked to terminate
[2024-11-01T12:17:39.978+0800] [DEBUG] Closed socket 12 (AF_UNIX)

Problem found, it was my problem, turned off ipv6 on the loopback interface via sysctl.
net.ipv6.conf.lo.disable_ipv6=1
After turning on ipv6 on the loopback interface, the xrdp connection worked fine.

please close issue

@jackwang713 - thanks for raising this, and also for providing a solution.
Given you're the third person to report something similar, I'm going to try to reproduce this so I understand it further. At the moment I can't see a reason why this should happen.

I can't reproduce this as reported on Ubuntu 22.04
As it is, disabling loopback on the lo0 interface results in an unstable system. Sometimes I get a desktop and sometimes I don't. However, the waitforx executable always manages to connect to the X server.
Looking at the source code for libX11 and XCB, if I'm passed a DISPLAY of the form :n, waitforx will always try to make a connection to the X server via the UNIX socket. Furthermore, by default we pass -nolisten tcp to the X server, so the X server won't be listening on TCP anyway.
However, looking into this has given me an insight into how this could potentially happen. I'll open a PR for discussion and reference this issue.

PR #3297 now links to this issue. I'll close it as requested.


Can't scroll in the terminal

OS Version: macOS

Can you repro in stable? Are you using zsh or oh-my-zsh?

Can also repro in stable. Yes, using oh-my-zsh. Happens to both me and @egamma, we both use it.

https://github.com/Microsoft/vscode/issues/7817 https://github.com/sourcelair/xterm.js/issues/151


Fix cast memory usage by using map on subtables

The cast operation on a pyarrow Table may create new arrays in memory.
This is an issue since users expect memory mapped datasets to not fill up the RAM.
To fix that I used map to write a new arrow file on disk when cast is used.
To make things more convenient I introduced the arrow formatting of a dataset, to make it return pyarrow tables instead of python dicts. This way one can use pyarrow transforms directly when using map.
edit: we'll use the same mechanism for filter

I addressed your comments about the docstrings and the output validation :)

I updated the bleurt mocking method and bleurt test is passing now.
I also ran the slow tests and they are passing for bleurt.

Thanks @lhoestq and @albertvillanova !


Cancellation of FtpConnection.ReadFromStreamAsync causes UnobservedTaskException

In FtpConnection.ReadFromStreamAsync, readTask is started using the cancellation token passed into the method, but tcs is also created which has its result set when the cancellation token is cancelled:
protected override async Task<int> ReadFromStreamAsync(byte[] buffer, int offset, int length, CancellationToken cancellationToken)
{
	var readTask = Stream
	   .ReadAsync(buffer, offset, length, cancellationToken);

	var tcs = new TaskCompletionSource<object?>();
	using var registration = cancellationToken.Register(() => tcs.TrySetResult(null));
	var resultTask = await Task.WhenAny(readTask, tcs.Task)
	   .ConfigureAwait(false);

	if (cancellationToken.IsCancellationRequested)
	{
		Logger?.LogTrace("Cancelled through CancellationToken");
		return 0;
	}
...

This can cause resultTask to complete when tcs.Task is complete, and the exception from the cancellation of readTask is never observed. causing an UnobservedTaskException.

Nice catch!


Will not accept code, keeps saying that a semicolon is missing.

Challenge Increment a Number with JavaScript has an issue.
User Agent is: Mozilla/5.0 (X11; CrOS x86_64 8530.96.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.154 Safari/537.36.
I was simply running my test and the shell would say, "Syntax error: Unexpected Number." Additionally the content checks for the test disappeared.

My code:

var myVar = 87;

// Only change code below this line

myVar++1;

Try changing myVar++1 to myVar++

Thank you, it works 👍

Please use the help button on the challenge page or the forum for these types of issues in the future. Thanks.

you dont need the "1" after it.  the "++" operator adds 1 automatically.


Fixes #34 and #39 parsing nicknames and user hosts

On some networks it's possible to have color codes in vhosts. I think it's pointless to add more and more exceptions to RFC, when we can simply accept all characters except spaces.
Ex<EMAIL_ADDRESS>JOIN :#/b/\r\n

On Quakenet such message can appear:
:*.quakenet.org MODE #example +o mike1256\r\n
It's also not valid according to standards, because it has asterisk inside hostname.

Bouncers like ZNC use virtual nicknames starting with asterisk for internal communication. Parser seems to doesn't recognize them:
<EMAIL_ADDRESS>PRIVMSG thebot :Error from Server [Closing Link: <IP_ADDRESS> (Killed (linear (youtube loop)))]\r\n

This pull request fixes all those issues and removes one test, which is unlikely to happen in real environment.

@svpernova09 Based on my experiences with this issue, I would tend to concur with @stil. His change looks solid to me.

Thanks @stil
If IRC servers aren't going to conform to the RFCs I agree we shouldn't hold too  hard and fast to them. We should support what the users are using as best we can while still holding close to the RFCs if it makes sense.


Short not null notation

Sorry for my English.
Example:
@computed get someName() {
	const tags = (store.get('Contact', id) as Contact).tags as Array<Tag>;
	// ...
}

These  as Contact,  as Array<Tag>, etc. already give me nightmares.
Why not add to TS a short notation for not null value? Example:
@computed get someName() {
	const tags = store.get('Contact', id)!.tags!;
	// ...
}

https://kotlinlang.org/docs/reference/null-safety.html#the--operator

Did you mean the non-null-assertion?

Excellent!

:confused:

@RyanCavanaugh you mean:❗ 😱 ❗ right?

Suggestion: It'd be useful if pressing the center of the steering wheel made a beeping noise


Programmatic schedule for scheduler node

Add possibility to programmatically configure the schedule of the scheduler node via input message.

Feature request has been moved to discussion forum as idea #31.


cors annotation does not work at module level

Describe the bug
Module-level CORS config is not having an effect
To Reproduce
Steps to reproduce the behavior:

Install Ambassador from helm chart, changing the default values.yaml as follows:
https://gist.githubusercontent.com/digressionary/09e3419b5d3d9b24ef3e6ab001a387c4/raw/f9a9383034820593ff1a41dfd2be4eb020e6553f/values.yaml

Configure an AuthService with whitelisted headers and a single mapping to route everything to an echo service.
https://gist.githubusercontent.com/digressionary/bc7d1e94d75ee8719751e3be8802f099/raw/27b0528a2d78825d48996b3769294b2e6dcf8ab1/service.yaml

Deploy an echo service for debugging similar to this:
https://gist.githubusercontent.com/digressionary/6a2a328672ed484d3c12215c6c20a72a/raw/6713851a2006cafaff13e39480853e47e23df639/echo-service.yaml

POST a request and examine the headers that the Echo service sees:
curl -X POST \ http://localhost/v1/partner/token \ -H 'Content-Type: application/json' \ -H 'Origin: http://apis.google.ca' \ -H 'Postman-Token: c2675a19-c894-48b4-a69b-ba79ce14fe28' \ -H 'cache-control: no-cache' \ -d '{  "username": "redacted",  "password": "redacted" }'

Expected behavior
The echo service response should include an Access-Control-Allow-Origin header.
Actual behavior
Request served by echo-deployment-b6b6878c-5t477
HTTP/1.1 POST /v1/partner/token
Host: localhost
Origin: http://apis.google.ca
User-Agent: PostmanRuntime/7.6.0
X-Envoy-Internal: true
Content-Type: application/json
Accept: /
Content-Length: 69
X-Forwarded-Proto: http
Cache-Control: no-cache
Postman-Token: 2fbdd64d-fad4-420d-9be4-ae771132d426
X-Request-Id: 3aa01529-e784-4e25-acee-8187e05899eb
X-Envoy-Expected-Rq-Timeout-Ms: 3000
Accept-Encoding: gzip, deflate
X-Forwarded-For: <IP_ADDRESS>
X-Envoy-Original-Path: /v1/partner/token
{}
Versions (please complete the following information):

Ambassador: [0.73]
Kubernetes environment [Docker-For-Desktop ]
Version: Engine 19.03.1, Kubernetes 1.14.3, version <IP_ADDRESS>

Additional context
If you use the same cors: stanza but at the mapping level, the expected behaviour of the CORS header being present occurs.
Echo service response when CORS is enabled on the mapping level:

mapping change:
https://gist.githubusercontent.com/digressionary/8b79ebe536c969de78c03a180a3887bb/raw/1664f4eabadfd1fb81df1194cefed07441d6d0d8/service.yaml

change in Echo Service response:
Request served by echo-deployment-b6b6878c-5t477

HTTP/1.1 POST /v1/partner/token
Host: localhost
Origin: http://apis.google.ca
User-Agent: PostmanRuntime/7.6.0
X-Envoy-Internal: true
Content-Type: application/json
Accept: /
Content-Length: 69
X-Forwarded-Proto: http
Cache-Control: no-cache
Postman-Token: 2fbdd64d-fad4-420d-9be4-ae771132d426
X-Request-Id: 3aa01529-e784-4e25-acee-8187e05899eb
Access-Control-Allow-Origin: *
X-Envoy-Expected-Rq-Timeout-Ms: 3000
Accept-Encoding: gzip, deflate
X-Forwarded-For: <IP_ADDRESS>
X-Envoy-Original-Path: /v1/partner/token
< redacted body>

Hello! I sorted out the issue by whitelisting some more headers when defining the AuthService.  For anyone else having this problem, here are the headers I whitelisted:
  allowed_request_headers:
    - requested-status
    - requested-body-info
    - content-type
    - access-control-allow-origin
    - access-control-allow-headers
    - access-control-request-method
    - access-control-request-headers
    - max-age
  allowed_authorization_headers:
    - content-type
    - max-age
    - auth-request-body
    - access-control-allow-origin
    - access-control-allow-headers
    - access-control-request-method
    - access-control-request-headers


qa: Fix typos

In case you wonder, I found it using codespell. And also I found that bitcoin/bitcoin has had codespell linter since 2018: https://github.com/bitcoin/bitcoin/pull/13954.


Add the project name before the application name when perspective is grouped by app labels

Version: main
UI: dev-debfa1a
Commit ID: 91c2f0ef5ff2e0bb7c7f1b8ae3407526bcabce7f
Env: MacOS 13.1, Apple M2
Prerequisite:

Create two app with the same name in two different projects.
Deploy instances and wait for cost data to be fetched.

Test Steps:

Create a perspective grouped by Label/ Application.

View the perspective created in step 1.

Test Result:

The data of the two apps (project1/appa and project2/appa) is aggregated. (Issue 1 in https://github.com/seal-io/seal/issues/390)

Suggestion:
Please add the project name before the application name to distinguish applications with the same name.

No more valid in the latest version


Firebase analytics is not initialized on iOS

Describe the bug
Documentation states to use initializeFirebase only for web devices, but i'm getting an initilization problem in iOs too.
To Reproduce
Steps to reproduce the behavior:

Launch the application.
Go to Firebase Analytics debug mode, you won't see any new event incoming.
Go to the Safari's web inspector and open a debugger for the device.
Go to the console and see the error.

Expected behavior
As Android, it should be initialized itself without to do it manually in code.
Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: Mac OsX 10.15.7
Browser Safari
Version 14.0.1 (156<IP_ADDRESS>.10, 15610)

Smartphone (please complete the following information):

Device: iPhone 8 Plus
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Do you have the Google services file for ios installed? It works on my project...

Yes, I have it.

Could you post some code? That would be helpful to debug

This is the original initilization funtion:
async initAnalytics() {
    const deviceInfo = await Device.getInfo()
    if (deviceInfo.platform === "web") {
      FirebaseAnalytics.initializeFirebase(appEnv.firebaseConfig)
    }

    this.setProperty({
      name: "platform",
      value: deviceInfo.platform,
    })
    this.setProperty({
      name: "appVersion",
      value: deviceInfo.appVersion,
    })
    this.setProperty({
      name: "appBuild",
      value: deviceInfo.appBuild,
    })
    this.setProperty({
      name: "osVersion",
      value: deviceInfo.osVersion,
    })
  }

I added ios as platform to initialize too and i don't get the warning, but i don't get any log too.
I don't understand what is the problem.
Do i need the certification for Firabese as we need it for CLoud messages?

Did you:

Add the iOS App to Firebase in Project settings -> My Apps
Sync the Capacitor Native Code with npx cap sync ios

Yeah, both of them we did.

You have to do it like this: https://firebase.google.com/docs/analytics/debugview

Thank you for your response.
In fact, my problem was that my Podfile wasn't up to date after my npx cap sync so I had to add this :
  pod 'CapacitorCommunityFirebaseAnalytics', :path => '../../node_modules/@capacitor-community/firebase-analytics'

Sounds like this issue has been resolved - if it is still a problem for you, please reopen.

@cyril-colin  after more than a year I spent a day trying to figure this out. In the end I just needed to add the pod manually thanks!


Feature/bluetooth architecture refactoring

Hi @ajdevy!
Consider I want to write own app for Ruuvi devices from scratch. Which steps shall I do in order to reuse Ruuvi Station app Bluetooth code?

@

Hi @ajdevy!
Consider I want to write own app for Ruuvi devices from scratch. Which steps shall I do in order to reuse Ruuvi Station app Bluetooth code?
Let's discuss in #81

You need the Interactor, Gateways, models and RuuviNotifier

@

Hi @ajdevy!
Consider I want to write own app for Ruuvi devices from scratch. Which steps shall I do in order to reuse Ruuvi Station app Bluetooth code?
Let's discuss in #81

You need the Interactor, Gateways, models and RuuviNotifier

Is it possible to extract them to library? Keeping in mind that we need to register for permissions in Android Manifest?

@

Hi @ajdevy!
Consider I want to write own app for Ruuvi devices from scratch. Which steps shall I do in order to reuse Ruuvi Station app Bluetooth code?
Let's discuss in #81

You need the Interactor, Gateways, models and RuuviNotifier

Is it possible to extract them to library? Keeping in mind that we need to register for permissions in Android Manifest?

Yes, it is possible.

@ajdevy is the requirement for user notification handled in the current codebase?
If I remember correctly, if you are scanning for bluetooth, you must inform user starting with SDK > 26.
https://github.com/rinat-enikeev/BTLib/blob/master/btlib/src/main/java/io/btkit/btlib/BTScanner.kt#L71

@rinat-enikeev yes, but it still should be refactored, there are currently 3 implementations that can be made into one


'xdpi' and 'ydpi'

DisplayMetrics metrics = getContext().getResources().getDisplayMetrics();
mDensity = (metrics.xdpi + metrics.ydpi) / 2f;
Could you tell me are 'xdpi' and 'ydpi' in these two pieces of code equal to  the width and the height in ios？ Thank you.

No. Xdpi/ydpi are density values, not dimensions. With Android, you never
know for sure what pixel density the screen will have, so careful measures
have to be taken to ensure the size of the stroke appears approximately the
same between devices.
For iOS, your screen densities are more finite (retina/non-retina). So you
would only need to account for those cases when drawing.
On Fri, Nov 13, 2015, 2:07 AM youyou5920<EMAIL_ADDRESS>wrote:

DisplayMetrics metrics = getContext().getResources().getDisplayMetrics();
mDensity = (metrics.xdpi + metrics.ydpi) / 2f;
Could you tell me are 'xdpi' and 'ydpi' in these two pieces of code equal
to the width and the height in ios？ Thank you.
—
Reply to this email directly or view it on GitHub
https://github.com/simplifycom/ink-android/issues/5.


<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.

Hi,
i am unable to install the plugin because:

<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.

But i use<EMAIL_ADDRESS>
Any tip? :-)

Thanks @HeikoMamerow, for the most part you can ignore this message, it should work without performing the below, but it's probably best to do this anyway.
It's because of a weird issue with how NPM installs, or in this case no longer installs peer dependencies, when installing packages globally that depend upon one another they need to be installed at the same time /shrug
So, if you run: npm uninstall -g stylelint stylelint-config-wordpress and follow that up by running  npm install -g stylelint stylelint-config-wordpress both packages should be installed without error.

Ha, works like charme! Thank you!!!!! :-)


Prevent panel hover header crop

Fixes #84830
Issue:
When hoverHeader is enabled PanelMenu is being cropped. There are 2 issues:

crop on the first panel due to hidden overflow of scroll wrapper

crop on subsequent panels due to descending z-index on the panels

Fixed by #675

Solution:
Set hoverHeaderOffset prop on PanelChrome to prevent default offset of -32 in HoverWidget
Definitely not ideal because it covers viz data but its quite tricky to portal HoverWidget and keep accessibility
Open to comments on this one.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hm, in the old architecture the grid was rendered withing a scroll wrapper too, what's the difference then between the DOM of the old and new arch?
I don't think we should change the offset.

Uhm, isn't that a matter of canvas-content not having top padding anymore?

Uhm, isn't that a matter of canvas-content not having top padding anymore?

@dprokop  Yes indeed, old arch had top padding of 16px and offset of 16 (to fill that padding). Currently we don't have padding on the canvas content and padding comes from controls above (which are sticky with high zIndex).
We could add top padding of 16px and offset of 16 to mimic old arch would look like that:

@Sergej-Vlasov noticed one more thing, in the old arch the offset is smaller for the panels at the top:

So there must be a collision detection or sth?

hoverHeaderOffset

Check is based on position so offset if -16 if panel at the top and -32 if anything else
const hoverHeaderOffset = (panel.gridPos?.y ?? 0) === 0 ? -16 : undefined;

@dprokop Thinking of just passing through a hoverHeaderOffset prop in scenes and defining everything in main repo.   Would this be the right place buildGridItemForPanel ?

Not sure we can fix this only in buildGridItemForPanel without a behavior as you can move panels around and the offset would need to change

Exactly. @Sergej-Vlasov  we would need to add a behavior to DashboardGridItem that would observe the changes in y position and update VizPanel accordingly. But the VizPanel API would have to be updated to allow offset configuration and I personally don't like this. VizPanel should not care about any UI-specific props. It's the responsibility of the PanelChrome I think.
That's why maybe there's an easier solution that does not require us to observe grid position. The same problem may appear in layouts other than Grid and then the current method won't work. I think we may want not to rely on the y-position, but rather the constraints of the scroll container. We already use https://floating-ui.com/docs/usefloating to position Tooltips and I think that's what we possibly would may want to use to position the hover header.
Thoughts?

@dprokop @torkelo Due to complications with portaling using FloatingUI  I decided to proceed with temporary fix using hoverHeaderOffset and behaviour that updates it for the panels at the top only. So here I only pass in hoverHeaderOffset prop.


create handler to write logs to a file (w/setting in options dialog) (Trac #24)

Create handler to write logs to a file and make the corresponding options dialog.
Migrated from http://microfluidics.utoronto.ca/ticket/24
{
    "status": "closed",
    "changetime": "2015-01-06T12:57:37",
    "description": "Create handler to write logs to a file and make the corresponding options dialog.",
    "reporter": "cfobel",
    "cc": "",
    "resolution": "fixed",
    "_ts": "1420549057189277",
    "component": "microdrop",
    "summary": "create handler to write logs to a file (w/setting in options dialog)",
    "priority": "major",
    "keywords": "",
    "version": "0.1",
    "time": "2012-01-04T03:26:10",
    "milestone": "Microdrop 1.0",
    "owner": "cfobel",
    "type": "enhancement"
}

Trac update at 20120104T21:13:12:

<EMAIL_ADDRESS>commented:

commit 4f52176fb9dcc3ba12d20202d316826a8abd295a
Author: Christian Fobel<EMAIL_ADDRESS>Date:   Wed Jan 4 15:25:06 2012 -0500
Add logging file handler to options dialog...

Add logging file handler to options dialog. Fixes #28.

<EMAIL_ADDRESS>changed resolution from "" to "fixed"
<EMAIL_ADDRESS>changed status from "new" to "closed"

Trac update at 20120104T21:13:31: cfobel changed type from "defect" to "enhancement"

Trac update at 20140314T05:59:17:

ryan commented:

Logging to a file functionality seems to be broken.

ryan changed resolution from "fixed" to ""
ryan changed status from "closed" to "reopened"

Trac update at 20140417T19:39:43: ryan changed component from "core" to "microdrop"

Trac update at 20150105T20:48:00:

cfobel changed resolution from "" to "fixed"
cfobel changed status from "reopened" to "closed"

Trac update at 20150106T12:57:37: Christian Fobel<EMAIL_ADDRESS>commented:

In changeset 4d36ef49316e9ad71c60a755fcd27c6a9a8e7d97:
#!CommitTicketReference repository="microdrop" revision="4d36ef49316e9ad71c60a755fcd27c6a9a8e7d97"
Fix #28, logging to file

Prior to this commit, logging to file was not working since the log-file
settings were incorrectly being loaded from the
`'microdrop.gui.main_window_controller'` app options.  This commit corrects the
mistake and now loads the log file settings from the app options of the
`'microdrop.app'` plugin.


feat: trim image id to 12 characters

@DOO-DEV Thanks for your contribution, I made some more changes,

Have a look, the first part will be copied by clicking on it, what do you think about it? 🤔
Also, we may need That long ID in future integrations, but we wanted to trim it only for telegram formatting.

Yeah it is a good idea to send a trim version of id but with option to click on it and get full id


feat: enhanced server update

:tada: This PR is included in version 2.1.0-beta.14 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Error with the target when paired with zendesk tap

Hello. I've run into an issue when trying to use this target alongside the zendesk tap, when the satisfaction_ratings stream provided by the latter is selected. Two errors appear to be happening at the same time, but only one happens within this target's files.
INFO Querying for satisfaction ratings between 2020-10-19T23:59:59Z and 2020-11-11T19:13:02Z
CRITICAL 'properties'
Traceback (most recent call last):
  File ".virtualenvs/target-postgres/bin/target-postgres", line 8, in <module>
    sys.exit(cli())
  File "~/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/__init__.py", line 45, in cli
    main(args.config)
  File " ~/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/__init__.py", line 39, in main
    target_tools.main(postgres_target)
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/target_tools.py", line 28, in main
    stream_to_target(input_stream, target, config=config)
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/target_tools.py", line 77, in stream_to_target
    raise e
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/target_tools.py", line 58, in stream_to_target
    _line_handler(state_tracker,
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/target_tools.py", line 123, in _line_handler
    buffered_stream = BufferedSingerStream(stream,
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/singer_stream.py", line 47, in __init__
    self.update_schema(schema, key_properties)
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/singer_stream.py", line 75, in update_schema
    properties = self.schema['properties']
KeyError: 'properties'

This is the first error that occurs. After a second error with the tap, this is what it's displayed:
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

I tried the same tap with the same config with the csv target, and it worked without a problem. I've tried other streams and they work fine, it's the only one I've run into a problem with.
PD: I'm running it from the terminal with
.virtualenvs/zendesk-tap/bin/tap-zendesk --config zendesk_config.json --catalog zendesk_test.json | .virtualenvs/targetpostgres/bin/target-postgres --config postgres_test_config.json

@j-oflaherty sorry for just now getting back to you. Github's new notifications somehow buried this for me 😅
Are you still experiencing this problem? I don't know that we've had long lasting issues with Zendesk as of late, so I'd be a bit shocked if we can't get this working.

@j-oflaherty sorry for just now getting back to you. Github's new notifications somehow buried this for me 😅
Are you still experiencing this problem? I don't know that we've had long lasting issues with Zendesk as of late, so I'd be a bit shocked if we can't get this working.

Hi alexander,
Thanks for responding. It worked fine in a co-worker's computer so i've
haven't tried again in a while. I think there's a problem with some of my
python libraries or something like that.
If you want to take a look anyway ask for whatever info you need, i'm glad
to help.
On Mon, Jan 4, 2021, 1:23 PM Alexander Mann<EMAIL_ADDRESS>wrote:

@j-oflaherty https://github.com/j-oflaherty sorry for just now getting
back to you. Github's new notifications somehow buried this for me 😅
Are you still experiencing this problem? I don't know that we've had long
lasting issues with Zendesk as of late, so I'd be a bit shocked if we can't
get this working.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/datamill-co/target-postgres/issues/194#issuecomment-754072247,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AO3MQAOC4XHFECFFU3Z3P2TSYHTORANCNFSM4TSMCROA
.

Hi alexander,
Thanks for responding. It worked fine in a co-worker's computer so i've
haven't tried again in a while. I think there's a problem with some of my
python libraries or something like that.
If you want to take a look anyway ask for whatever info you need, i'm glad
to help.
On Mon, Jan 4, 2021, 1:23 PM Alexander Mann<EMAIL_ADDRESS>wrote:

@j-oflaherty https://github.com/j-oflaherty sorry for just now getting
back to you. Github's new notifications somehow buried this for me 😅
Are you still experiencing this problem? I don't know that we've had long
lasting issues with Zendesk as of late, so I'd be a bit shocked if we can't
get this working.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/datamill-co/target-postgres/issues/194#issuecomment-754072247,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AO3MQAOC4XHFECFFU3Z3P2TSYHTORANCNFSM4TSMCROA
.

Hi @AlexanderMann,
Sorry for the delay,
The problem still persists. I've opened the issue in the zendesk tap and they pointed out it's probably a problem on the target's end. Is there any extra information i can provide to help you sort the problem out?

@j-oflaherty that's very odd...at this point, the best thing would be a test which reproduces this. You can check out our https://github.com/datamill-co/target-postgres/blob/master/tests/unit/test_sandbox.py for examples for how to add one.
Aside from that, redacted streams, schemas, and config are welcomed if you can share them.

@AlexanderMann I've managed to find and ''fix'' the problem. On the file singer_stream.py on line 69 the function json_simplify(schema) gives different structures for different streams. With the stream tickets the result is the following:

This way, in line 76, self.schema['properties'] does't raise a key error. When doing json_scheam.simplify on the satisfaction_ratings schema the result is the following structure:

In this case, on line 77 we get the KeyError. When changing line 70 to self.schema = json_schema.simplify(schema)['anyOf][0] it works fine, but now the rest of the streams fail. Adding a try/except seems to work fine.

I've been experiencing this same problem lately

@zopf @j-oflaherty do you have a link to these schemas? This definitely seems like a bug 🐛 we should be handling better for everyone. Additionally if you just have the work ready to merge, I'd be happy to entertain a pr contribution!

@AlexanderMann I was able to make it work implementing a try and except to catch the error. I don't know if it's the best solution, but it hasn't given any errors so far...
I only added this on the file singer_stream.py on line 75:
try:
    properties = self.schema['properties']
except KeyError:
    self.schema = self.schema['anyOf'][0]
    properties = self.schema['properties']

I don't know what you mean by a link to the schemas. If it's catalogs you are refering to I just get it with the discover function from the zendesk tap.


Canal + Typha do not work together in v3.28.1

Canal with Typha will cause felix to go into a restart loop due to an incorrect IpInIpTunnelAddr config change.
Succinct calico-node:v3.28.1 logs to show the problem
...
# This is a correctly detected value on start
2024-09-12 19:32:34.231 [INFO][59604] felix/config_params.go 635: Parsing value for IpInIpTunnelAddr: <IP_ADDRESS> (from datastore (per-host))
...
# With typha that previously detected value gets overwritten with an empty string
2024-09-12 19:32:34.265 [INFO][59604] felix/daemon.go 1345: Configuration value changed; change DOES require Felix to restart. key="IpInIpTunnelAddr" newValue="" oldValue="<IP_ADDRESS>"
...
# Which causes felix to go into a restart loop until eventually sleeping
2024-09-12 19:32:34.265 [WARNING][59604] felix/daemon.go 754: Felix is shutting down reason="config changed"

Expected Behavior
Expect the IpInIpTunnelAddr value to remain "<IP_ADDRESS>"
Current Behavior
IpInIpTunnelAddr gets updated to "" which causes felix to go into a restart loop. This also causes the readiness probe to fail with 503.
curl localhost:9099/readiness
+---------------------------+---------+----------------+---------------------+--------+
|         COMPONENT         | TIMEOUT |    LIVENESS    |      READINESS      | DETAIL |
+---------------------------+---------+----------------+---------------------+--------+
| CalculationGraph          | 30s     | reporting live | reporting ready     |        |
| FelixStartup              | -       | reporting live | reporting ready     |        |
| InternalDataplaneMainLoop | 1m30s   | reporting live | reporting non-ready |        |
+---------------------------+---------+----------------+---------------------+--------+

Possible Solutions

You can work around this by manually annotating nodes

kubectl annotate node newnode1 projectcalico.org/IPv4IPIPTunnelAddr=<IP_ADDRESS>

Possibly revert this change originally scoped for BGP? https://github.com/projectcalico/calico/commit/5b3a6806fe609a98a62b872da1cb25d2499573cb

Steps to Reproduce (for bugs)

Take the canal manifest: https://github.com/projectcalico/calico/blob/v3.28.1/manifests/canal.yaml
Add typha: https://docs.tigera.io/calico/latest/getting-started/kubernetes/hardway/install-typha
Start/restart/look at a calico-node container logs on any host that doesn't have the projectcalico.org/IPv4IPIPTunnelAddr annotation

Here is a manifest I've made from the steps above to simplify a reproducible environment.

canal-typha.txt

Context

For canal upgraded k8s clusters it will prevent adding new nodes without manually adding the projectcalico annotations.
For newly created k8s clusters it will prevent canal from going into a ready state until manually adding projectcalico annotations.

Your Environment

Canal version: v3.28.1
Orchestrator version: Kubernetes 1.28
Operating System and version: Ubuntu 22.04

This was my mistake that was kind of masked away with older versions. In older versions automatic node annotations where hiding a misconfiguration between canal + typha.  Since v3.27.0 (I believe) it stopped annotating the nodes which exposes what I ran into.
Just as USE_POD_CIDR is defined for felix / calico-node:
https://github.com/projectcalico/calico/blob/v3.28.1/manifests/canal.yaml#L4895-L4896
Typha needs to have the same configuration on its deployment as well.
  - name: USE_POD_CIDR
     value: "true"

Once both have that there is no resetting for IPIPTunnelAddr to zero length string.


Updated podcast:value specification for custom records

This commit makes the usage of customKey and customValue more specific.
The change to the spec was made based on the known users for these attributes (LNPay and HIVE gateway), but it is not necessarily backwards compatible.
Why is this change necessary?
Client applications that use the information on the podcast:value tag must know what to with its content.
Imagine one receiver sets customValue to raw binary data encoded in hex, and expects it to be sent in the payment as raw binary data.
Imagine another receiver sets customValue to raw binary data encoded in base64, and expects it to be sent in the payment as raw binary data.
Imagine now that other receiver sets customValue to a string, and expects it to be sent in the payment as a string encoded in UTF-8 (which currently known users do).
There is no way a client could know what do to with customValue (how to decode it an re-encode it), without specific knowledge about how each receiver uses this field.
Although this is possible, it is counter productive for client development, as clients should have to know as little as possible
about different application specific protocols used by the receivers.

@brianoflondon , @tkijewski , which encoding do you expect when decoding the value from the custom records?
utf-8 works, as, to my best knowledge you use characters within the ASCII range - within this range UTF-8 is compatible with other encodings such as Windows-1252 and ISO 8859–1, so in general it should not be a problem, but in case you have anything against using UTF-8 in the spec, please comment.