Integrate with other WebGL engines

There's been some interest in integrating Cesium with other WebGL engines.
On the Cesium side, this should be pretty straightforward.  The changes will require Cesium to only own the context during a call to Scene.render, not forever.  This is already mostly there since objects only access the context in update functions called from within Scene.render, but we also keep some state between frames that we'll need to change:

Render State #638
Clear states (if I don't remove them sooner)

Also, when we do the post processing framework, we need to be mindful of when we are rendering to texture vs. the main framebuffer.

Hi,
I've done a first test with cesium + three.js here:
http://potree.org/demo/experimental/2015.08.19_cesium_test/examples/cesium_test.html
The basic idea is this:

Copy "cesium.camera.viewMatrix" and "cesium.camera.inverseViewMatrix" to "three.camera.matrixWorldInverse" and "three.camera.matrixWorld"
The potree point cloud is given in EPSG:21781(swiss). This is first transformed to WGS84(degrees) and from that to Cesium.Cartesian3.fromDegrees().
pointcloud.position is set to the Cartesian3 value of the bounding box minimum
pointcloud.lookAt(bottomLeftHigh) is called to align the point cloud to align the point cloud on the ground
potree.pointcloud.up.copy(latDir) is used to rotate the point cloud to align with longitude and latitude

Issues:

This example is specific to a point cloud in the EPSG:21781 coordinate system
three.js is rendered in its own canvas over the cesium canvas. Idealy, both rendering engines should use the same context and render into the same buffers, if possible.
Precision issues with the three.js camera, especially noticeable when the camera is moving close to the point cloud. Possibly because three.js uses floats? Does cesium use double precision values for transformations?

Does cesium use double precision values for transformations?

Yes, see the link I sent on twitter: http://blogs.agi.com/insight3d/index.php/2008/09/03/precisions-precisions/
There will also be depth buffer compositing and perhaps precision issues with this approach.

Thanks, I'll take a look at it!

Found out how to render into the same canvas with the same context:
First, create a three.js renderer and pass the canvas as argument. The canvas/context was already initialized by cesium. Three.js will call getContext() for the same canvas and therefore, according to the spec, receive the same context as cesium.
three.renderer = new THREE.WebGLRenderer({alpha: true, canvas: cesium.viewer.canvas});

In the render loop, cesium has to be rendered first.
During the rendering with three.js, WebGLRenderer._currentProgram has to be set to null before using the three.js render function. Since _currentProgram is a private variable, this can only be done through WebGLRenderer.resetGLState(). resetGLState(), however, also calls state.reset which causes errors. This workaround therefore temporarily changes the state.reset function to an empty function
var resetFunc = three.renderer.state.reset;
three.renderer.state.reset = function(){};
three.renderer.resetGLState();
three.renderer.state.reset = resetFunc;

Now it's probably not that important for most cases but it would still be interesting to use the same depth buffer in threejs as well. Do you think that will be possible with the way cesium handles depth buffers? At https://cesiumjs.org/massiveworlds/ there are slides about rendering with multiple frustums in which case reusing the depth buffer is probably out of question.

Thanks for the info.

Now it's probably not that important for most cases but it would still be interesting to use the same depth buffer in threejs as well.

I expect most apps will want this, e.g., depth testing points vs. Cesium's terrain.  I expect this can be done by setting Three.js' camera near and far planes and explicitly rendering the Three.js scene for each frustum in Cesium.  See executeCommands in Scene.js.
The jittering issue in Three.js we discussed on twitter would also need to be addressed.

We're doing a project to mix Cesium and Three, like potree. Cesium is rendered firstly,then Three's elements rendered on the Cesium's canvas.But the problem is Three's elements always block Cesium's. e.g.: there is a geometry created by Three in front of the Cesium‘s earth,it can block the Cesium‘s earth and other elements correctly, but when the scene is rotated to the backward, Three's object still block the Cesium,maybe depth buffer is the main reason,but we cannot find the solution to solve. Do you have any methods?We wanna make Cesium the part of the Three’s scene,like an ordinary geometry. Is it possible?
Here is the code.But if using three.renderer.context.enable(three.renderer.context.DEPTH_TEST); below,the whole scene is shining.
`

three.js & Potree & Cesium

`

@brucefuqiming this is because Cesium and Three have two different depth buffers and Three is being rendered second.  To merge the scenes correctly, this will require core engine support in Cesium as described in this issue.

For a prototype of integrating Cesium and three.js, see Section 6.2 here: https://www.cg.tuwien.ac.at/research/publications/2015/Adorjan-2015/Adorjan-2015-thesis.pdf

One possibility could be to render color and depth buffers in one lib and then use the depth buffer(s) in the other lib to compose the renderings of both.
e.g.:

User renders his stuff in three.js and provides color buffer, depth buffer, and projection parameters to cesium.
Cesium does a screen pass (or one for each frustum) with the three.js color and depth map as input and discards all fragments that do not pass a custom depth test. The custom depth test would consist of transforming the three.js depth values to cesium depth values.

Vice-versa, cesium could provide a color buffer and all of its depth maps and someone can use this to do a custom compose step to merge the cesium renderings with his three.js or other renderings.

Hello sir, I would like to ask you about the cesium+three.js how to load the.Gltf model, and let the three.js camera with cesium camera rotation, there is no good examples of reference, thank you.

@gaojunxiaowwww8888 you might want to ask the three.js community or try the Cesium forum: http://cesiumjs.org/forum.html.  Just a heads up that the Cesium team doesn't yet have any three.js experience.

Hello sir, I want to ask is whether in the development of cesium+three by three cesium light to improve the illumination effect, three light can affect cesium light; or what is the way to change the cesium in the light effect?

@m-schuetz Hello sir, I want to ask is whether in the development of cesium+three by three cesium light to improve the illumination effect, three light can affect cesium light; or what is the way to change the cesium in the light effect?

Hello, sir. I don't know how to improve the lighting effect of cesium. Is there a good example for reference?

@gaojunxiaowwww8888 please do not post the same question over and over again in this issue.  As explained in CONTRIBUTING.md, the Cesium forum is the preferred place to ask question; we use GitHub issues for bugs and feature requests.

For the cesium+three.js development process, the three model jitter problem is solved? Now encountered this problem, I hope you can help, thank you

@gaojunxiaowwww8888 please do not post the same question to the Cesium forum and this repo.  Your question was answered in https://groups.google.com/forum/#!topic/cesium-dev/gCWi3BpT47U.
For jitter in three.js, your best bet is to ask the three.js community as I mentioned to you above.

Related blog post: https://cesium.com/blog/2017/10/23/integrating-cesium-with-threejs/

Just adding a note to myself that we would also consider down-scoping this if it makes it easier to implement, e.g., support three.js geometries but not post-processing, etc.

Related: luma.gl's WebGL state tracker

一种可能性是在一个库中渲染颜色和深度缓冲区，然后使用另一库中的深度缓冲区来组合两者的渲染。
例如：

用户在 Three.js 中渲染他的东西，并向 cesium 提供颜色缓冲区、深度缓冲区和投影参数。
Cesium 使用 Three.js 颜色和深度图作为输入进行屏幕传递（或每个平截头体一个），并丢弃所有未通过自定义深度测试的片段。自定义深度测试包括将 Three.js 深度值转换为 cesium 深度值。

反之亦然，cesium 可以提供颜色缓冲区及其所有深度图，并且有人可以使用它来执行自定义合成步骤，以将 cesium 渲染与他的 Three.js 或其他渲染合并。

我很认同，但是大佬有没有代码啊


Is dynamic dictionary case sensitive?

I wonder if the documentation could be specific about whether the injection of dynamic dictionary term is case sensitive or not?  I believe it is case sensitive, but I would like a confirmation.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1648784c-c385-4120-a2a6-3414a0d752e1
Version Independent ID: 1f90c698-a146-3345-40dd-eb5dabe56cf3
Content: Translator Translate Method - Azure Cognitive Services
Content Source: articles/cognitive-services/Translator/reference/v3-0-translate.md
Service: cognitive-services
Sub-service: translator-text
GitHub Login: @laujan
Microsoft Alias: lajanuar

@mikethetexan
Thanks for your feedback! We will investigate and update as appropriate.

@mikethetexan yes, it is case sensitive.


Tag the chrome version too?

This is not trivial, as we only know the chrome version when building the Docker image.
Might be a nice to have i.e. not needed.  If we are pinning to a Docker digest successfully when pulling the image then we’ll have reproducibility in our builds, which is the main thing.

One way to do this would be to install Chrome on the CircleCI image that is used to build the Docker image.  That would give us the chrome version (google-chrome --version) which we could then use as the tag, after correlating that it is the same version that the Docker build installed.

Alternatively if there is a way to capture the output of the RUN command in the Dockerfile then that would be an option too - with the advantage that we wouldn’t have to install Chrome on the CircleCI image just to get the version, and also that we’d be certain to be tagging the right version.

Or could use a dedicated Docker image just for Chrome.  That might be overkill though.  More overhead to maintain.  Worse performance potentially. Nice separation of concerns mind you.  So it’s a tradeoff.

Maybe could do something using separate Docker layers or a multistage build.  Again, might be overkill.

Turned out to be a much easier way to do it
Closed by #13


added some example code

Summary

changed package name of example app to avoid name conflicts with library
added example code

Looks awesome! :+1:
SQLite integration coming soon!


SwitchAccountAsync function is missing

Hi, I'm an ID@Xbox developer using C# and MonoGame to program a UWP Xbox One game. I'm using the latest version of XSAPI.UWP from NuGet. There is a requirement that we need an option on the main menu to let the player switch profiles... I see no actual way to do this in XSAPI. If you look in the CHM documentation there is a function in XboxLiveUser called SwitchAccountAsync() but when trying to use this function in XSAPI it's nonexistent.
I've posted a thread here detailing how I'm currently trying to switch users but I can't get anything working: https://social.msdn.microsoft.com/Forums/en-US/31835e63-c766-4d7a-8588-f2fd2f1f693b/uwp-changing-users-during-game?forum=xboxlivedev
Microsoft told me via email that UWP games are not exempt from this requirement, so either I'm not seeing how to do this or the functions I need are missing.
Would really appreciate some assistance!
Thanks,
Michael

Hello @michaelarts,
This is not supported at this time.  You can switch users using the windows Xbox app but there is not currently a way to do this from within your game.

Thanks, for anyone who reads this in the future: the way to switch users is to use the UserPicker class. I've posted my code here: https://gist.github.com/michaelarts/2ada67de3ca4295b1ea57df00fd230ca


Bump to Crossplane v1.11.3

This PR bumps the Crossplane version of the workflows in this repo to v1.11.3, the latest release of Crossplane.
https://github.com/crossplane/crossplane/releases/tag/v1.11.3

One of the provider upgrade tests failed, but it has been failing from before this PR.  So proceeding with this bump.
https://github.com/crossplane/test/actions/runs/4600416459/jobs/8127467194?pr=40


fundchannel should return a txid inside the json respose

We are missing the more important information that is the txid
➜  ~ lampo-cli --network signet fundchannel --node_id 026b92e6cc3e0f24b42494672ecf65f3d147b9deb95186bf024da7a7745008ee75  --amount 200000 --public true
{
  "node_id": "026b92e6cc3e0f24b42494672ecf65f3d147b9deb95186bf024da7a7745008ee75",
  "amount": 200000,
  "public": true,
  "push_mst": 0,
  "to_self_delay": 2016,
  "tx": {
    "version": 2,
    "lock_time": 0,
    "input": [
      {
        "previous_output": "ac9ee361e0ec07f919f6b9dc07c2fd21f05ef559326b45e6f889c6e37bb6f32b:0",
        "script_sig": "",
        "sequence":<PHONE_NUMBER>,
        "witness": [
          "55846fb5c7ef20f0ba055a8cc9c3d9e97c4323d02bbf9b9a88490d4fd24cacde5afeac503ca26a163089c3b3a4246340cfffa2dcc7388c5055d139871e101449"
        ]
      },
      {
        "previous_output": "9891b12bb6e608bada7296bdc47bd424df16ad4ac14330c0f003335df18b9b03:0",
        "script_sig": "",
        "sequence":<PHONE_NUMBER>,
        "witness": [
          "f6e0d011c6052ccf697ed5a5c48cee62acf5f2aed548bf27d71b31951072b5ef150a49897ec260a0c80dc4246903affb62bb21ad80036b2a0ec551c3b1ccc6fa"
        ]
      }
    ],
    "output": [
      {
        "value": 1059880,
        "script_pubkey": "51209a9177c972471352f58c4b9b8fedcaea9b0230edc1565c671644119e84059caf"
      },
      {
        "value": 200000,
        "script_pubkey": "0020ad50c2d4bbcc6db44e70bdf30acc123bd5b62652df250bd7fb29641d8083cd70"
      }
    ]
  }
}

Trying to bump this in the following release

I'll look at this tomorrow, sorry for the delay!

I'll look at this tomorrow, sorry for the delay!

No problem it is my fault that I just noted that we do not have this


Barkskin Overrides AC instead of creating a Minimum

I am guessing this is the case because there is no way currently to have Active Effects create a "minimum AC", is there?

The barkskin spell should be set to grant an effect that just has system.attributes.ac.min | Override | 16 now.


unmapped gene srs1

Antonia:
add srs1 PMID:9651503
to https://www.pombase.org/status/unmapped_genes
I'm unsure if this is https://www.pombase.org/gene/SPAC29A4.15 (can't find any evidence for it) have asked Val, will close if she says they are the same.

Val:
A different srs1...I'm not sure what srs is on the genetic map (something 'sensitive'?),
but it's not related to the tRNA synthase name....

Antonia:
suppressor of ras and ste9 :-)

from https://github.com/pombase/curation/issues/2034

done


Consider future.spawned() syntax

I proposed the same idea to tokio-rs/tokio#2652. I'm glad if async-std could have it.

Hi, thanks for opening this! Following @taiki-e's comment in https://github.com/tokio-rs/tokio/issues/2652 I'm not quite sure what the benefits would be for integrating this into async-std. If Tokio decides to adopt a variant of this it'd be interesting to see how it would work out, but for the time being I don't think we can take on more surface area for spawning tasks beyond what we already expose. Thanks!


Add bal executor support for 1.2.x branch

Description:
Add both ballerina and bal command support for 1.2.x branch since the version 2.1.14 supports bal command by default.
Suggested Labels:

Suggested Assignees:

Affected Product Version:
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:

Fixed with https://github.com/ballerina-platform/ballerina-lang/pull/29858


[PAG-1362] Add documentation for sub selecting elements

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Plan for effectively scaling to handle increased service setup

From #9:

Add to this list, re: infrastructure:
Authenticode signing services;
Certificates for said signing;
.NET Foundation CLA bots;
Static / security analysis tools per the maturity ladder requirements;
Hosting / deployment for websites; and
Access control + ability to delegate to other project members for all of the above.
Today, in order for me to give another Akka.NET member the ability to setup CI for one of our sub-projects or plugins - is there a way I can do that without having to wait on a human being at the .NET Foundation?
What are the plans for scaling this?

My thoughts on high level plan is to scale the things that can't be automated is through the Project Support Action Group.
Organizationally, the action groups are run by board members (community elected) and staffed by action group volunteers (.NET Foundation voting members). It may require scaling through paying a vendor, but a decent amount of this can probably be handled by empowering volunteers by documenting processes and giving the right people access to service logins. We've had members volunteer and ask when they'll be given something they can help with, and this is a good example of that. We do have some funds we could use to pay for admin help through corporate sponsorship and member dues, it'll be up to the board to decide on how to spend the budget since paying for operational expenses takes away from money we could be donating to community events, projects, etc.
We were planning to get a better idea of the workload and how well the action group is able to handle it by extrapolating from our pilot project onboarding.
Thoughts?

Here's what we been considering on ReactiveUI to try and eliminate some need of getting new users onto Azure DevOps.

We are considering having a "Release" label. We would write a .ts azure devops script that would read the github PR and then add it to the build as a build tag. Azure DevOps has a "build tag" filter on the release system and then release when that tag was applied. This would limit the number of requests for users to the DevOps server.
We been considering GitHub actions as a CI. One issue with this approach is we don't control the sign client certificate and we'd have to integrate it into GitHub secret store. We definitely been working out ways of leveraging GitHub Actions more with automated Tweets when releases go out and stuff. We basically have been not wanting to go down this route because we annoyed @onovotny a bunch on previous issues.

With the GitHub actions approach it would allow us to be based on GitHub permissions to run. It does have some built in actions by the GitHub team for keystore storage and we could leverage the secrets store in the project/organisation.

For this to scale it's not fair to go rest on just @jongalloway and @onovotny shoulders, for a decent SLA we need a proper support channel.
And it's not fair to oppose more friction than needed upon maintainers, I would love this to be a positive thing where there's a recipe for success, where we offer something with hopefully better and with less maintenance if they would've done it them selves.
We shouldn't lock out possibility for projects to do their own infra, so important that the process for becoming certified should be documented and transparent.

I think relying on @jongalloway and @onovotny is problematic. I know on our project we know that @onovotny is especially busy so we've avoided poking him too much. People usually know when people are really busy and hate to interrupt them.


SINGA-487 Add Sparsification Algorithms

This PR implements some sparsification schemes, we transfer only gradient elements which are significant. When we make use of cuda thrust parallel algorithm to convert the dense array into sparse array, the overhead is relatively low.
It supports two mode, controlled by the flag topK:

When topK is False, it transmits the gradient elements which are greater than an absolute threshold value.
When topK is True, it transmits the K largest gradient element, where K equals the total number of elements multiplies the spars factor.
Moreover, there is a flag corr to use the local accumulate gradient for correction. The flag is true by default, because it is common to use the local accumulate gradient correction in sparsification.

Some reference papers for the Sparsification:
[1] N. Strom. Scalable distributed dnn training using commodity gpu cloud computing. In Proceedings of gpu cloud computing. In Proceedings of the InterSpeech 2015. International Speech
Communication Association (ISCA), September 2015.
[2] A. F. Aji and K. Hea
eld. Sparse communication for distributed gradient descent. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pages 440{445. Association for Computational Linguistics (ACL), September 2017.
I have added an examples file sparsification_mnist.py to test the accuracy. The following results is based on a 8 GPUs AWS instance p2.x8large of the GPU model K80.
ubuntu@ip-172-31-18-216:~/singa/examples/autograd$ python3 sparsification_mnist.py
Starting Epoch 0:
Training loss = 1237.824951, training accuracy = 0.537627
Evaluation accuracy = 0.831209, Elapsed Time = 1.364238s
Starting Epoch 1:
Training loss = 468.859161, training accuracy = 0.835053
Evaluation accuracy = 0.931229, Elapsed Time = 0.687484s
Starting Epoch 2:
Training loss = 329.488220, training accuracy = 0.887604
Evaluation accuracy = 0.949424, Elapsed Time = 0.713595s
Starting Epoch 3:
Training loss = 220.463303, training accuracy = 0.925731
Evaluation accuracy = 0.955592, Elapsed Time = 0.686450s
Starting Epoch 4:
Training loss = 171.178146, training accuracy = 0.942141
Evaluation accuracy = 0.961760, Elapsed Time = 0.686534s
Starting Epoch 5:
Training loss = 149.635681, training accuracy = 0.950237
Evaluation accuracy = 0.974198, Elapsed Time = 0.686791s
Starting Epoch 6:
Training loss = 124.092453, training accuracy = 0.958300
Evaluation accuracy = 0.973376, Elapsed Time = 0.686136s
Starting Epoch 7:
Training loss = 115.288582, training accuracy = 0.961205
Evaluation accuracy = 0.968647, Elapsed Time = 0.686174s
Starting Epoch 8:
Training loss = 99.048584, training accuracy = 0.966864
Evaluation accuracy = 0.981188, Elapsed Time = 0.685848s
Starting Epoch 9:
Training loss = 84.038574, training accuracy = 0.972239
Evaluation accuracy = 0.981188, Elapsed Time = 0.685568s

I will need more time to study it, so I close it first and reopen when it is ready. I will do more important thing first.

I have improved the code.
A test using AWS p2.x8large up to 4 instances or 32 K80 GPUs:


Socket.io client implimenation

Socket.io client
Description
Would it be possible to get socket.ios client implemented as an optional module that you have the choice of including within your project?
Version

0.9

Platform

Windows [X]

Yes, right now if you want to you can import it into the editor by using the Project/ExecuteScript option inside the editor and including it along with the nunuruntime when you deploy your application.
But i will implement that when i start  working on the NPM module import implementation (maybe this weekend)

Ahhh wow that's a great feature to hear about,thank you

Any eta on node modules implementation?

(There is not a tutorial for socket based communication using WebSockets)


Add autoLog

Adds autoLog on health.
Disconnects with message: "Health was: amount of health here"
Disables at logging in until health gets above threshold again.

seems fine


Parsed data format not sufficient for round-trip

For beautification, the parsing output should provide sufficient information to round-trip from input file to parsed data to output file with no changes to the displayed text.
This means that each parsed token needs to retain the number of newlines between it and the previous parsed token and the specific whitespace characters that occur before the token - between the start of the line and the current token if it is the first token on the line, or between the previous token and the current token if it is not.
Round-tripping with full-fidelity, including whitespace after text, whitespaces on empty lines, and specific/mixed newline characters, might be a nice to have but is not required.

To help me better appreciate the nature of this problem could you provide a use case example?  In most languages the specific white space immediately preceding a token isn't really important.  In markdown, though, the specific white space preceding a particular token, especially at the start of a line, could be critically important, but I am accounting for this in the lexer logic.

@prettydiff - The whitespace isn't important to the language but it is often important to the user and for formatting.
Consider the case where the user has formatted an javascript array literal in some non-standard but useful way (see the preserve-array-indentation setting is js-beautify).  In that case, tracking the whitespace before the token matters.

I think the way to handle that would be to specify preservation around a block of code so that the code structure in question is parsed as a single token with all white space.  I already have this for markup and I could extend into the script and style lexers using an opening and closing comment pair.

The solution is published with documentation (though I just realized I copy/pasted a JavaScript code sample into the CSS documentation).  It works like this:

There must be a comment that matches this pattern: /^(\/(\/|\*)\s*parse-ignore-start)/ which is essentially a line or block comment that opens with optional white space immediately followed by the word parse-ignore-start.
Everything including and following the opening comment will be included as a single parsed token of type ignore and presv of true.
This ignored token will contain everything from the start of the opening comment through either the end of the closing comment or end of code sample.
Ending code sample will be another comment nearly identical to the opening comment except the word is parse-ignore-end.

This feature is included in both the script and style lexers with identical effect.  Here is a brief code example:
if (b[a] === "\n") {
    /* parse-ignore-start */
    if (options.lang === "apacheVelocity" && lex[0] === "#") {
        a = a - 1;
        break;
    }
    /* parse-ignore-end */
    parse.lineNumber = parse.lineNumber + 1;
}


Make Document content_id not nullable

It was the intent that all Documents should have content_ids: ab71140021975b474f8804b67538f1b7414109fd
There's one Document record in production which doesn't have one - a News
Article. Let's preempt problems migrating it by filling in the missing value.

:+1:


Lost 23 hours of notes b/c it incorrectly indicated that the draft was saved

I was pressing cntr-s and believed that I was getting positive feedback that it was saving but after a short network problem, I found that the last saved draft was 23 hours old.
I suspect that you need to change the browser side code to require positive confirmation of a save rather than taking the absense of an error as indication of success.
Also, you could save each cntr-s draft as a separate row and instead of deleting them, set a deleted column to true. Then use a cron job to remove deleted row that are over N days old.
Steps To Reproduce
Steps to reproduce the behavior:

I was taking notes in a page while researching. I was pressing cntr-s often and I was aware of the "draft saved" msg and believed at the time that I was getting positive feedback that cntr-s was working. During that time I put me notebook to sleep several times. I actually thought that cntr-s was the same as pressing "save page" followed by "edit" to keep editting. I did not understand draft vs revision at the time. Its hard to say what exactly I saw after pressing cntr-s but I know that I got some feedback and it was probably the same that I see now -- a checkbox appears for a time. I did not notice at teh time whether the timestamp updated.
After a wifi network blip, I got a popup near the "save page" button saying that the page could not be saved because the server could not be reached. (did not record the exact text).
I fixed the wifi issue by connecting to a different SSID which would have changed my local IP.
I confirmed that connectivity was back by using a new browser window.
I pressed "save page" and was greeted with a "419|Page Expired" Error.
Page was then empty, but pressing "edit" filled in 23 hour old content.

Expected behavior
Not to have my hard work pissed away!
Your Configuration (please complete the following information):

BookStack v0.29.3
PHP Version: 7.4.3 (cli) (built: May 26 2020 12:24:22) ( NTS )
Hosting Method (Nginx/Apache/Docker):  NGINX on Ubuntu 20.04 server

Additional context
Discord transcript:
bobgToday at 11:56 AM
I just lost a day of research notes!  I was hitting cntrl-s often, and I know that I saw a msg about the page being saved, yet when I just had a network blip, my page went back to empty!  I queried page_revisions table and found only the initial save with a fraction of the content. Is there anywhere else I can look to find a more recent version of the content?
the current page is actually blank so I suspect that when I hit 'back' in the browser it resulted in the browser having an empty page and then submitting that empty page
any ideas?
been using bookstack for a week and was loving it up to now :frowning:
bobgToday at 12:07 PM
what is the concept of a "draft"? Is that stored in a different place than a "revision"?  Experimenting on a different page, the message I remeber seeing was "draft saved"
bobgToday at 12:16 PM
weel, I dumped the entire DB and serached for keywords -- the draft is not in the DB anywhere.
I assume thats it. no where else it could be. I would appreciate any confirmation of that before I move on.
bobgToday at 12:34 PM
So now I do see that initial commit of the page with some data in the page_revisions table but when I go to revisions in the UI, even though It lists one revision clicking on the "changes" and on "current" results in no content being shown? I can get the content string directly from the DB, how do I get that back into my page?
ssddanbrownToday at 12:45 PM
@bobg A revision is a published version of the page. A draft is user-specific draft content for a page.
What type is the row in the DB where you've found your content?
bobgToday at 12:46 PM
I think its a update_draft
ssddanbrownToday at 12:47 PM
Okay, There should be a user_id or something on that row, does it match the id of your user?
bobgToday at 12:47 PM
(I am using the mysql cmdline and the formatting of columns in hard to line up)
yes -- created_by = 3 which is me
ssddanbrownToday at 12:52 PM
I'd expect that content to be restored into the editor when you go to edit the page again, Should pop up a message like "Here's content you were editing earlier" .
bobgToday at 12:52 PM
oh. maybe if I just hit "edit" in the UI it will populate with my draft?
ssddanbrownToday at 12:53 PM
Should do, Make sure you have a copy of that DB first, just in case
bobgToday at 12:53 PM
good idea:) -- will do
that 'worked"
this draft was from 23 hours ago it says:(
so why was it letting me press cntr-s for a day without actually saving the draft?
I can not say for sure that I was seeing the "draft saved" message but I am pretty sure that I was -- I know it registered that cntr-s was doing something
after the network blip, I got a message about not saving so after network was back, I hit "save page" and that got a "4xx|Page Expired" msg. (I think it was 417)
I can not remember the last time I had a content lost error like this. It is painful because I was keeping really good notes!
but anyway, thanks for your help
ssddanbrownToday at 1:37 PM
@bobg Yeah, That's a harsh situation , apologies. I think someone else did report something similar but I had trouble re-producing it. Might need to double check the autosave logic to ensure we're always throwing errors when needed, think some cases have appeared where the save fails but a garbled success message pops up.
Autosave can always be tricky there as there's the risk of autosaving bad content off the back of another issue.
If you can re-produce the scenario in any manner make sure to file a bug so we can address it.
bobgToday at 1:44 PM
yes, after I understood the draft  vs versions, I assumed that the problem would be that it had been saving but the last save was empty. I am very surprised to find that it had not had a succesfull save in 23 hours
in what ever IPC you are using. the browser side thought it was a success when the server side was not successfull.  That should be solved in the protocol so that it can not happen.  the browser needs a positive confirmation of success as opposed to a lack of an error.
I actually thought that cntr-s was the same as "page save" + "edit". There was no indication that saving in that manner was more risky.
bobgToday at 2:00 PM
each draft save could produce an update_draft+(countCol++) revision and instead of deleting them on "page save",  you just toggle a deleted column on all those rows. Then a cron job removes deleted drafts older than N days.

I found a partial reproducible issue. This is not exactly what happened to me because mysql did not stop running during the time of my problem, but it illustrates an important flaw in the application that a failure to save draft content is not conveyed to the user. If this issue is fixed, it would prevent the loss of data that I suffered.

start editing a page
make changes, cntr-s to save a draft
on server stop, stop mysql
make changes, (which is still being edited ) and cntr-s save draft
observe the draft UI element at the top of the page edit area.

What happens is that a green check momentarily appears at the top of the UI, which seems to indicate that the page is saved. The text next to the checkmark is "Undefined NaaN.NaN" instead of the last saved time but its formatted the same as the time stamp would have been and its easy not to notice. What catches your eye is the green check mark appearing and then disappearing.
What should happen is a prominent UI indication that the content was not saved. For example, that "last saved" element should be red and loud with the text "could not save".  It would also make sense to inhibit further changes to the content but that's just a nice to have.
That checkmark should only appear in response to positive round trip confirmation of the save.
(BTW, this actually cost me two days because I could not let go of this loss and spent a lot of time looking into it)
--BobG

I've been editing the page over time and it got into that state again. I was away for a couple hours and the computer was on the whole time. It was working before but when I came back I started entering additional content and soon noticed that it was not saving anymore.

The draft save the UI  element  now displays "undefined 0NaN:0NaN" instead of the typical ""
cntr-s still causes the check checkmark to appear which seems to be indicating a sucessful save
on the server I query the row of the 'update_draft' for this page and i see that its not updating -- the updated_at no longer changes and the content does not repleft my edits.
the DB is still operable and I can go to another tab and create new pages, edit save, etc...

The work-a-round when you see "undefined 0NaN:0NaN" is to...

start editing the same page in a new tab. In my case it said "you are editing a draft that was last saved 4 hours ago"
then copy the content from the first tab (that has your current content) into the new tab and save.

When I press "save page" on the original tab I get "419|Page Expired"
My guess is the the server expires the page after a fixed period of time that it is open for editing. At that time any further request in that page context fails but the XMLHttpRequest that implements the draft save ignores the error and reads response data that is not there so when it updates the draft element, the null data renders as "undefined 0NaN:0NaN".
The "save page" is apparently implemented differently and correctly displays the page expired error.
If the save draft is fixed to notice the page expire error, it can just display the page expired message similar to the "save page" request. There might be a little unsaved data at that time but the really important thing is to prevent the user from continuing to enter content. Ideally you could freeze editing but still leave the page so that the user could copy and paste the data.
Of course, its possible to get the server to renew the page session and continue to save but the more important fix is to make sure that if the save fails for any reason, the page will be in a state that it no longer accepts new content from the user or at least prominently warns the user that the page is not in a good way.
I have also observed that it seems to be autosaving frequently so that the cntr-s is not really necessary. That means that even if the user looses the unsaved changes, its not that much.
BTW, I really like having the manual cntr-s save and the positive feed back of the green checkmarkin response to saving the draft -- as long as it really is a positive confirmation that it was saved.
--BobG

Here is another scenario where you can loose edits in progress.

edit page, produce edits, draft is saved automatically or via cntr-s
shutdown OS without explicitly closing chrome. (in my case, linux got in a weird state and it would not resume when openning notebook lid so I had to force shutdown)
when chrome reopens it asks if you want to restore pages -- answer yes
observe that the tab is restored to the edit state but has the last revision loaded and no the draft content.
At that point it would be easy to continue and presumably if you start editing, it would save the new draft, overwiting your previous draft content.

Work-a-round:

close the tab being careful not to make any edits (which would save the new draft).
navigate back to the page
execute "edit" and it will load your previous draft and you can continue as normal.

It seems that what ever chrome does to restore the page ends up loading the last revision instead of your draft content. I suspect that the URL that chrome submits is either different than the "edit" action or the state of the authentication/session is not complete when it makes the request.
--BobG

Experienced a similar problem here. I suspect the session was expired, but no warning was displayed.
As written by others above, the page was marked as saved and then clicking on the save button gave the 419 Page expired error.

Experienced the same and also lost a lot of text in the process :disappointed: The UI told me that drafts have been saved and the draft timestamp was up to date. A couple of hours later, I clicked on Save page and got the 419 error. After that, all changes were gone.

Apologies to everyone that encountered this.
Some changes to how we make AJAX requests caused errors to not be thrown when expected leading to a misleading success message showing instead of a warning on error message. For v0.30 quite a lot of the JS has been touched upon which included updating the AJAX logic to properly throw errors when requested.
I also done further testing myself and added a safety layer in d9e2bddee4297eb71a17b277aba5e948663feb2d to always show an error on failed requests (Sometimes only the first error in a series would be shown before) in addition to storing the page content in the browser's LocalStorage upon error event to give a chance of possible manual recovery where important.
Again, My apologies and sorry this hasn't been addressed sooner, the last few months have been hard going leaving little time to work on the project. These changes will be part of v0.30 which I'm intended to release this weekend.

@ssddanbrown Thank you for the update and all the work! :+1:


esphome reports "unknown" when inverter shuts down or on its own sometimes

the setup is working fine.  i do not know why it happens but every often the esphome reports "unknown" but i am not sure if its always because the inverter is off or because of range issues.
anyway, the problem is, most of the time when inverter is turned off because grid power goes out, esphome reports "unknown" and not off.  same for all entities (52 for me)
i can go to reimann integration and create a new entity to convert unknown into 0 value for example but what if the esphome does that?
also, can it be that esp is loosing wifi connection? maybe that can be a problem? is there a way to detect that because sometimes the logs say apiconnection error, so what can be the reason or reasons?
INFO ESPHome 2023.12.7
INFO Reading configuration /config/esphome/esphome-web-5f9e4d.yaml...

INFO Updating https://github.com/syssi/esphome-pipsolar.git@pip8048
INFO Starting log output from esphome-web-5f9e4d.local using esphome API
INFO Successfully connected to esphome-web-5f9e4d @ <IP_ADDRESS> in 2.923s
INFO Successful handshake with esphome-web-5f9e4d @ <IP_ADDRESS> in 2.453s
[23:35:00][I][app:102]: ESPHome version 2023.11.4 compiled on Nov 28 2023, 23:52:53
[23:35:00][C][wifi:559]: WiFi:
[23:35:00][C][wifi:391]:   Local MAC: 48:3F:DA:5F:9E:4D
[23:35:00][C][wifi:396]:   SSID: 'JioFiber-rHnqt'[redacted]
[23:35:00][C][wifi:397]:   IP Address: <IP_ADDRESS>
[23:35:00][C][wifi:398]:   BSSID: 58:95:D8:24:49:81[redacted]
[23:35:00][C][wifi:400]:   Hostname: 'esphome-web-5f9e4d'
[23:35:00][C][wifi:402]:   Signal strength: -81 dB ▂▄▆█
[23:35:00][C][wifi:406]:   Channel: 1
[23:35:00][C][wifi:407]:   Subnet: <IP_ADDRESS>
[23:35:00][C][wifi:408]:   Gateway: <IP_ADDRESS>
[23:35:00][C][wifi:409]:   DNS1: <IP_ADDRESS>
[23:35:00][C][wifi:410]:   DNS2: <IP_ADDRESS>
[23:35:00][D][pipsolar:924]: Sending polling command : QPIGS2 with length 6
[23:35:00][W][component:214]: Component esphome.coroutine took a long time for an operation (0.07 s).
[23:35:00][W][component:215]: Components should block for at most 20-30ms.
[23:35:00][C][logger:416]: Logger:
[23:35:00][C][logger:417]:   Level: DEBUG
[23:35:00][C][logger:418]:   Log Baud Rate: 115200
[23:35:00][C][logger:420]:   Hardware UART: UART0
[23:35:00][D][pipsolar:860]: checking crc on incoming message
[23:35:00][D][pipsolar:863]: CRC OK
[23:35:00][C][uart.arduino_esp8266:102]: UART Bus:
[23:35:00][C][uart.arduino_esp8266:103]:   TX Pin: GPIO4
[23:35:00][C][uart.arduino_esp8266:104]:   RX Pin: GPIO5
[23:35:00][C][uart.arduino_esp8266:106]:   RX Buffer Size: 256
[23:35:00][C][uart.arduino_esp8266:108]:   Baud Rate: 2400 baud
[23:35:00][C][uart.arduino_esp8266:109]:   Data Bits: 8
[23:35:00][C][uart.arduino_esp8266:110]:   Parity: NONE
[23:35:00][C][uart.arduino_esp8266:111]:   Stop bits: 1
[23:35:00][C][uart.arduino_esp8266:115]:   Using software serial
[23:35:00][C][pipsolar:948]: Pipsolar:
[23:35:00][C][pipsolar:949]: used commands:
[23:35:00][C][pipsolar:952]: QPIGS
[23:35:00][C][pipsolar:952]: QPIGS2
[23:35:00][C][pipsolar:952]: QMOD
[23:35:00][C][pipsolar:952]: QFLAG
[23:35:00][C][pipsolar:952]: QPIRI
[23:35:00][C][pipsolar:952]: QBATCD
[23:35:00][C][pipsolar.switch:068]: Pipsolar Switch '${name} output_source_priority_utility'
[23:35:00][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[23:35:00][C][pipsolar.switch:090]:   Restore Mode: always OFF
[23:35:00][C][pipsolar.switch:068]: Pipsolar Switch '${name} output_source_priority_solar'
[23:35:00][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[23:35:00][C][pipsolar.switch:090]:   Restore Mode: always OFF
[23:35:00][C][pipsolar.switch:068]: Pipsolar Switch '${name} output_source_priority_battery'
[23:35:00][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[23:35:00][C][pipsolar.switch:090]:   Restore Mode: always OFF
[23:35:00][C][pipsolar.switch:068]: Pipsolar Switch '${name} input_voltage_range'
[23:35:00][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[23:35:00][C][pipsolar.switch:090]:   Restore Mode: always OFF
[23:35:00][C][pipsolar.switch:068]: Pipsolar Switch '${name} pv_ok_condition_for_parallel'
[23:35:00][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[23:35:00][C][pipsolar.switch:090]:   Restore Mode: always OFF
[23:35:00][C][pipsolar.switch:068]: Pipsolar Switch '${name} pv_power_balance'
[23:35:00][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[23:35:00][C][pipsolar.switch:090]:   Restore Mode: always OFF
[23:35:00][C][pipsolar.select:009]: pipsolar.selectPipsolar Controller Select 'inverter0_output_source_priority_select'
[23:35:00][C][pipsolar.select:009]: pipsolar.selectPipsolar Controller Select '${name} charging discharging control'
[23:35:00][C][pipsolar.select:009]: pipsolar.selectPipsolar Controller Select '${name} charger_source_priority_select'
[23:35:00][C][pipsolar.select:009]: pipsolar.selectPipsolar Controller Select '${name} current_max_ac_charging_current_select'
[23:35:00][C][pipsolar.select:009]: pipsolar.selectPipsolar Controller Select '${name} current_max_charging_current_select'
[23:35:00][C][captive_portal:088]: Captive Portal:
[23:35:00][C][mdns:115]: mDNS:
[23:35:00][C][mdns:116]:   Hostname: esphome-web-5f9e4d
[23:35:01][C][ota:097]: Over-The-Air Updates:
[23:35:01][C][ota:098]:   Address: esphome-web-5f9e4d.local:8266
[23:35:01][C][ota:101]:   Using Password.
[23:35:01][C][api:139]: API Server:
[23:35:01][C][api:140]:   Address: esphome-web-5f9e4d.local:6053
[23:35:01][C][api:142]:   Using noise encryption: YES
[23:35:01][D][pipsolar:924]: Sending polling command : QMOD with length 4
[23:35:01][W][component:214]: Component esphome.coroutine took a long time for an operation (0.05 s).
[23:35:01][W][component:215]: Components should block for at most 20-30ms.
[23:35:01][D][pipsolar:860]: checking crc on incoming message
[23:35:01][D][pipsolar:863]: CRC OK
[23:35:01][D][pipsolar:536]: Decode QMOD
[23:35:01][D][text_sensor:064]: '${name} device_mode': Sending state 'L'
[23:35:02][D][pipsolar:924]: Sending polling command : QFLAG with length 5
[23:35:02][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:02][W][component:215]: Components should block for at most 20-30ms.
[23:35:02][D][pipsolar:860]: checking crc on incoming message
[23:35:02][D][pipsolar:863]: CRC OK
[23:35:02][D][pipsolar:544]: Decode QFLAG
[23:35:03][D][pipsolar:924]: Sending polling command : QPIRI with length 5
[23:35:03][W][component:214]: Component esphome.coroutine took a long time for an operation (0.09 s).
[23:35:03][W][component:215]: Components should block for at most 20-30ms.
[23:35:04][D][pipsolar:860]: checking crc on incoming message
[23:35:04][D][pipsolar:863]: CRC OK
[23:35:04][D][pipsolar:479]: Decode QPIRI
[23:35:04][D][pipsolar.select:031]: got value: 2
[23:35:04][D][pipsolar.select:033]: found mapped option 2A for option 2
[23:35:04][D][select:015]: '${name} current_max_ac_charging_current_select': Sending state 2A (index 0)
[23:35:04][D][pipsolar.select:031]: got value: 60
[23:35:04][D][pipsolar.select:033]: found mapped option 60A for option 60
[23:35:04][D][select:015]: '${name} current_max_charging_current_select': Sending state 60A (index 5)
[23:35:04][D][pipsolar.select:031]: got value: 0
[23:35:04][D][pipsolar.select:033]: found mapped option Utility first for option 0
[23:35:04][D][select:015]: 'inverter0_output_source_priority_select': Sending state Utility first (index 0)
[23:35:04][D][pipsolar.select:031]: got value: 2
[23:35:04][D][pipsolar.select:033]: found mapped option Solar and utility for option 2
[23:35:04][D][select:015]: '${name} charger_source_priority_select': Sending state Solar and utility (index 1)
[23:35:04][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:04][W][component:215]: Components should block for at most 20-30ms.
[23:35:04][D][pipsolar:924]: Sending polling command : QBATCD with length 6
[23:35:05][D][pipsolar:860]: checking crc on incoming message
[23:35:05][D][pipsolar:863]: CRC OK
[23:35:05][D][pipsolar:758]: Decode QBATCD
[23:35:05][D][pipsolar.select:031]: got value: (000
[23:35:05][D][pipsolar.select:036]: could not find option (000 in mapping
[23:35:05][D][pipsolar:924]: Sending polling command : QPIGS with length 5
[23:35:06][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:06][W][component:215]: Components should block for at most 20-30ms.
[23:35:06][D][pipsolar:860]: checking crc on incoming message
[23:35:06][D][pipsolar:863]: CRC OK
[23:35:06][D][pipsolar:502]: Decode QPIGS
[23:35:06][D][sensor:093]: '${name} grid_voltage': Sending state 232.70000 V with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} grid_frequency': Sending state 49.90000 Hz with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} ac_output_voltage': Sending state 232.70000 V with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} ac_output_frequency': Sending state 49.90000 Hz with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} ac_output_apparent_power': Sending state 1210.00000 VA with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} ac_output_active_power': Sending state 1184.00000 W with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} output_load_percent': Sending state 24.00000 % with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} bus_voltage': Sending state 330.00000 V with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} battery_voltage': Sending state 0.00000 V with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} battery_charging_current': Sending state 0.00000 A with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} battery_capacity_percent': Sending state 0.00000 % with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} inverter_heat_sink_temperature': Sending state 19.00000 °C with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} pv1_input_current': Sending state 0.00000 A with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} pv1_input_voltage': Sending state 0.00000 V with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} battery_voltage_scc': Sending state 0.00000 V with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} battery_discharge_current': Sending state 0.00000 A with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} battery_voltage_offset_for_fans_on': Sending state 0.00000 V with 1 decimals of accuracy
[23:35:06][D][sensor:093]: '${name} pv1_charging_power': Sending state 0.00000 W with 1 decimals of accuracy
[23:35:06][W][component:214]: Component esphome.coroutine took a long time for an operation (0.17 s).
[23:35:06][W][component:215]: Components should block for at most 20-30ms.
[23:35:06][D][pipsolar:924]: Sending polling command : QPIGS2 with length 6
[23:35:06][W][component:214]: Component esphome.coroutine took a long time for an operation (0.07 s).
[23:35:06][W][component:215]: Components should block for at most 20-30ms.
[23:35:07][D][pipsolar:860]: checking crc on incoming message
[23:35:07][D][pipsolar:863]: CRC OK
[23:35:08][D][pipsolar:924]: Sending polling command : QMOD with length 4
[23:35:08][W][component:214]: Component esphome.coroutine took a long time for an operation (0.05 s).
[23:35:08][W][component:215]: Components should block for at most 20-30ms.
[23:35:08][D][pipsolar:860]: checking crc on incoming message
[23:35:08][D][pipsolar:863]: CRC OK
[23:35:08][D][pipsolar:536]: Decode QMOD
[23:35:08][D][text_sensor:064]: '${name} device_mode': Sending state 'L'
[23:35:09][D][pipsolar:924]: Sending polling command : QFLAG with length 5
[23:35:09][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:09][W][component:215]: Components should block for at most 20-30ms.
[23:35:09][D][pipsolar:860]: checking crc on incoming message
[23:35:09][D][pipsolar:863]: CRC OK
[23:35:09][D][pipsolar:544]: Decode QFLAG
[23:35:10][D][pipsolar:924]: Sending polling command : QPIRI with length 5
[23:35:10][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:10][W][component:215]: Components should block for at most 20-30ms.
[23:35:10][D][pipsolar:860]: checking crc on incoming message
[23:35:10][D][pipsolar:863]: CRC OK
[23:35:10][D][pipsolar:479]: Decode QPIRI
[23:35:10][D][pipsolar.select:031]: got value: 2
[23:35:10][D][pipsolar.select:033]: found mapped option 2A for option 2
[23:35:10][D][select:015]: '${name} current_max_ac_charging_current_select': Sending state 2A (index 0)
[23:35:10][D][pipsolar.select:031]: got value: 60
[23:35:10][D][pipsolar.select:033]: found mapped option 60A for option 60
[23:35:10][D][select:015]: '${name} current_max_charging_current_select': Sending state 60A (index 5)
[23:35:10][D][pipsolar.select:031]: got value: 0
[23:35:10][D][pipsolar.select:033]: found mapped option Utility first for option 0
[23:35:10][D][select:015]: 'inverter0_output_source_priority_select': Sending state Utility first (index 0)
[23:35:10][D][pipsolar.select:031]: got value: 2
[23:35:10][D][pipsolar.select:033]: found mapped option Solar and utility for option 2
[23:35:10][D][select:015]: '${name} charger_source_priority_select': Sending state Solar and utility (index 1)
[23:35:10][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:10][W][component:215]: Components should block for at most 20-30ms.
[23:35:11][D][pipsolar:924]: Sending polling command : QBATCD with length 6
[23:35:11][D][pipsolar:860]: checking crc on incoming message
[23:35:11][D][pipsolar:863]: CRC OK
[23:35:12][D][pipsolar:758]: Decode QBATCD
[23:35:12][D][pipsolar.select:031]: got value: (000
[23:35:12][D][pipsolar.select:036]: could not find option (000 in mapping
[23:35:12][D][pipsolar:924]: Sending polling command : QPIGS with length 5
[23:35:12][W][component:214]: Component esphome.coroutine took a long time for an operation (0.07 s).
[23:35:12][W][component:215]: Components should block for at most 20-30ms.
[23:35:12][D][pipsolar:860]: checking crc on incoming message
[23:35:12][D][pipsolar:863]: CRC OK
[23:35:12][D][pipsolar:502]: Decode QPIGS
[23:35:12][D][sensor:093]: '${name} grid_voltage': Sending state 232.70000 V with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} grid_frequency': Sending state 49.90000 Hz with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} ac_output_voltage': Sending state 232.70000 V with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} ac_output_frequency': Sending state 49.90000 Hz with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} ac_output_apparent_power': Sending state 1210.00000 VA with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} ac_output_active_power': Sending state 1186.00000 W with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} output_load_percent': Sending state 24.00000 % with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} bus_voltage': Sending state 330.00000 V with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} battery_voltage': Sending state 0.00000 V with 1 decimals of accuracy
[23:35:12][D][sensor:093]: '${name} battery_charging_current': Sending state 0.00000 A with 1 decimals of accuracy
[23:35:13][D][sensor:093]: '${name} battery_capacity_percent': Sending state 0.00000 % with 1 decimals of accuracy
[23:35:13][D][sensor:093]: '${name} inverter_heat_sink_temperature': Sending state 19.00000 °C with 1 decimals of accuracy
[23:35:13][D][sensor:093]: '${name} pv1_input_current': Sending state 0.00000 A with 1 decimals of accuracy
[23:35:13][D][sensor:093]: '${name} pv1_input_voltage': Sending state 0.00000 V with 1 decimals of accuracy
[23:35:13][D][sensor:093]: '${name} battery_voltage_scc': Sending state 0.00000 V with 1 decimals of accuracy
[23:35:13][D][sensor:093]: '${name} battery_discharge_current': Sending state 0.00000 A with 1 decimals of accuracy
[23:35:13][D][sensor:093]: '${name} battery_voltage_offset_for_fans_on': Sending state 0.00000 V with 1 decimals of accuracy
[23:35:13][D][sensor:093]: '${name} pv1_charging_power': Sending state 0.00000 W with 1 decimals of accuracy
[23:35:13][W][component:214]: Component esphome.coroutine took a long time for an operation (0.17 s).
[23:35:13][W][component:215]: Components should block for at most 20-30ms.
[23:35:13][D][pipsolar:924]: Sending polling command : QPIGS2 with length 6
[23:35:13][W][component:214]: Component esphome.coroutine took a long time for an operation (0.07 s).
[23:35:13][W][component:215]: Components should block for at most 20-30ms.
[23:35:13][D][pipsolar:860]: checking crc on incoming message
[23:35:13][D][pipsolar:863]: CRC OK
[23:35:14][D][pipsolar:924]: Sending polling command : QMOD with length 4
[23:35:14][W][component:214]: Component esphome.coroutine took a long time for an operation (0.05 s).
[23:35:14][W][component:215]: Components should block for at most 20-30ms.
[23:35:14][D][pipsolar:860]: checking crc on incoming message
[23:35:14][D][pipsolar:863]: CRC OK
[23:35:14][D][pipsolar:536]: Decode QMOD
[23:35:14][D][text_sensor:064]: '${name} device_mode': Sending state 'L'
[23:35:15][D][pipsolar:924]: Sending polling command : QFLAG with length 5
[23:35:15][W][component:214]: Component esphome.coroutine took a long time for an operation (0.09 s).
[23:35:15][W][component:215]: Components should block for at most 20-30ms.
[23:35:15][D][pipsolar:860]: checking crc on incoming message
[23:35:15][D][pipsolar:863]: CRC OK
[23:35:15][D][pipsolar:544]: Decode QFLAG
[23:35:16][D][pipsolar:924]: Sending polling command : QPIRI with length 5
[23:35:16][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:16][W][component:215]: Components should block for at most 20-30ms.
[23:35:17][D][pipsolar:860]: checking crc on incoming message
[23:35:17][D][pipsolar:863]: CRC OK
[23:35:17][D][pipsolar:479]: Decode QPIRI
[23:35:18][D][pipsolar.select:031]: got value: 2
[23:35:18][D][pipsolar.select:033]: found mapped option 2A for option 2
[23:35:18][D][select:015]: '${name} current_max_ac_charging_current_select': Sending state 2A (index 0)
[23:35:18][D][pipsolar.select:031]: got value: 60
[23:35:18][D][pipsolar.select:033]: found mapped option 60A for option 60
[23:35:18][D][select:015]: '${name} current_max_charging_current_select': Sending state 60A (index 5)
[23:35:18][D][pipsolar.select:031]: got value: 0
[23:35:18][D][pipsolar.select:033]: found mapped option Utility first for option 0
[23:35:18][D][select:015]: 'inverter0_output_source_priority_select': Sending state Utility first (index 0)
[23:35:18][D][pipsolar.select:031]: got value: 2
[23:35:18][D][pipsolar.select:033]: found mapped option Solar and utility for option 2
[23:35:18][D][select:015]: '${name} charger_source_priority_select': Sending state Solar and utility (index 1)
[23:35:18][W][component:214]: Component esphome.coroutine took a long time for an operation (0.07 s).
[23:35:18][W][component:215]: Components should block for at most 20-30ms.
[23:35:18][D][pipsolar:924]: Sending polling command : QBATCD with length 6
[23:35:18][D][pipsolar:860]: checking crc on incoming message
[23:35:18][D][pipsolar:863]: CRC OK
[23:35:18][D][pipsolar:758]: Decode QBATCD
[23:35:18][D][pipsolar.select:031]: got value: (000
[23:35:18][D][pipsolar.select:036]: could not find option (000 in mapping
[23:35:18][D][pipsolar:924]: Sending polling command : QPIGS with length 5
[23:35:18][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:18][W][component:215]: Components should block for at most 20-30ms.

also, what does coroutine took a long time for operation mean?
[23:35:13][W][component:214]: Component esphome.coroutine took a long time for an operation (0.17 s).
[23:35:13][W][component:215]: Components should block for at most 20-30ms.
[23:35:13][D][pipsolar:924]: Sending polling command : QPIGS2 with length 6
[23:35:13][W][component:214]: Component esphome.coroutine took a long time for an operation (0.07 s).
[23:35:13][W][component:215]: Components should block for at most 20-30ms.
[23:35:13][D][pipsolar:860]: checking crc on incoming message
[23:35:13][D][pipsolar:863]: CRC OK
[23:35:14][D][pipsolar:924]: Sending polling command : QMOD with length 4
[23:35:14][W][component:214]: Component esphome.coroutine took a long time for an operation (0.05 s).
[23:35:14][W][component:215]: Components should block for at most 20-30ms.
[23:35:14][D][pipsolar:860]: checking crc on incoming message
[23:35:14][D][pipsolar:863]: CRC OK
[23:35:14][D][pipsolar:536]: Decode QMOD
[23:35:14][D][text_sensor:064]: '${name} device_mode': Sending state 'L'
[23:35:15][D][pipsolar:924]: Sending polling command : QFLAG with length 5
[23:35:15][W][component:214]: Component esphome.coroutine took a long time for an operation (0.09 s).
[23:35:15][W][component:215]: Components should block for at most 20-30ms.
[23:35:15][D][pipsolar:860]: checking crc on incoming message
[23:35:15][D][pipsolar:863]: CRC OK
[23:35:15][D][pipsolar:544]: Decode QFLAG
[23:35:16][D][pipsolar:924]: Sending polling command : QPIRI with length 5
[23:35:16][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:16][W][component:215]: Components should block for at most 20-30ms.
[23:35:17][D][pipsolar:860]: checking crc on incoming message
[23:35:17][D][pipsolar:863]: CRC OK
[23:35:17][D][pipsolar:479]: Decode QPIRI
[23:35:18][D][pipsolar.select:031]: got value: 2
[23:35:18][D][pipsolar.select:033]: found mapped option 2A for option 2
[23:35:18][D][select:015]: '${name} current_max_ac_charging_current_select': Sending state 2A (index 0)
[23:35:18][D][pipsolar.select:031]: got value: 60
[23:35:18][D][pipsolar.select:033]: found mapped option 60A for option 60
[23:35:18][D][select:015]: '${name} current_max_charging_current_select': Sending state 60A (index 5)
[23:35:18][D][pipsolar.select:031]: got value: 0
[23:35:18][D][pipsolar.select:033]: found mapped option Utility first for option 0
[23:35:18][D][select:015]: 'inverter0_output_source_priority_select': Sending state Utility first (index 0)
[23:35:18][D][pipsolar.select:031]: got value: 2
[23:35:18][D][pipsolar.select:033]: found mapped option Solar and utility for option 2
[23:35:18][D][select:015]: '${name} charger_source_priority_select': Sending state Solar and utility (index 1)
[23:35:18][W][component:214]: Component esphome.coroutine took a long time for an operation (0.07 s).
[23:35:18][W][component:215]: Components should block for at most 20-30ms.
[23:35:18][D][pipsolar:924]: Sending polling command : QBATCD with length 6
[23:35:18][D][pipsolar:860]: checking crc on incoming message
[23:35:18][D][pipsolar:863]: CRC OK
[23:35:18][D][pipsolar:758]: Decode QBATCD
[23:35:18][D][pipsolar.select:031]: got value: (000
[23:35:18][D][pipsolar.select:036]: could not find option (000 in mapping
[23:35:18][D][pipsolar:924]: Sending polling command : QPIGS with length 5
[23:35:18][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:18][W][component:215]: Components should block for at most 20-30ms.

Your observation is correct: If the api connection to the ESP gets lost the ESPHome integration of Home Assistant changes all sensor states of the node to unavailable. If the connection recovers a few seconds later the ESP cannot provide/push new measurements because the inverter doesn't provide any data (as long its unavailable).

[23:35:18][W][component:214]: Component esphome.coroutine took a long time for an operation (0.08 s).
[23:35:18][W][component:215]: Components should block for at most 20-30ms.

See https://github.com/esphome/issues/issues/4717


Bundle object Extras is null all the time

Hello,
I am trying to figure out if app is being created by clicking on push notification via using the Bundle object which is sent to onCreate function as parameter. Depending on the parameters sent via push message I want to open different views of the application.  As you can understand I am using this plugin in a Xamarin forms application to receive push notifications. Notifications are being shown properly and when I click them it opens the app properly. But I could not figure out how to reach the same parameters from inside the opened app to decide what to do in next step.
Let's say push notification Title is Football and text is Barcelona, I want to receive that from inside the app and open directly the football category with Barcelona's details. But I can not receive this data at the moment. What ever I tried I couldn't figure it out.
    protected override void OnCreate(Bundle bundle)
    {
        //bundle.Extras param is always NULL
        base.OnCreate(bundle);

        global::Xamarin.Forms.Forms.Init(this, bundle);

But it is coming as NULL all the time. What should I do to fix that?
And also, if this approach is wrong what can I use instead?
Thanks in Advance

Try this prerelease: https://www.nuget.org/packages/Xam.Plugin.PushNotification/1.2.5-beta


CFP: Enhance L2 Announcement

Cilium Feature Proposal
Is your proposed feature related to a problem?
One of the problems with kubernetes is external load balancers used for services (ingress for example). Load balancers should care about both failover and throughput. One kind of setup is to have private L2 network for nodes. Each node have a static IP inside. Also there is one more L2 network (several IPs with a gateway) and they can be dynamically assigned to the nodes (currently we do it via keepalived). This is really looks like new beta feature 'L2 Announcement', but 'L2 Announcement' assigns all external ip's defined for service to juts only one node. For failover and performance (via dsr) we need to try to distribute evenly external ip's on different nodes and to reassign them as soon as node will be treated as 'down'.
i.e. if we have three ip's (<IP_ADDRESS>-4 for example) and three nodes and ingress service with load balancer, I would expect that:
node1: <IP_ADDRESS>, node2: <IP_ADDRESS>, node3: <IP_ADDRESS>.
If node3 goes down it may be: node1: <IP_ADDRESS>, node2: <IP_ADDRESS> and <IP_ADDRESS>
Describe the feature you'd like
Try to always evenly distribute all external ip's defined for service on all allowed nodes. Try to reassign ip as soon as possible whenever any node goes down.

Any chances to look into this ?

bump


create-react-app doesn't work

Gives following error -
npx : The term 'npx' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that
the path is correct and try again.
At line:1 char:1

npx create-react-app hello-world

  + CategoryInfo          : ObjectNotFound: (npx:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CommandNotFoundException

This doesn't look like a create-react-app problem because the errors says that "npx" command is not recognized/found. This probably means you don't have "npx" installed or it's not set in your PATH on your computer.
"npx" is included with NPM v5.2+ so you can check what version of NPM you have by running npm -v in your terminal. If you have an older version than 5.2 then it means you don't have "npx" command at all. If you have a version newer than 5.2 then it might mean that your NPM install is just somehow messed up and this could probably be resolved by just installing latest LTS release of node (https://nodejs.org/en/), just so that you are using more up to date tooling and so that node and npm are reinstalled on your computer.

I think this is an error with your local installation of node and cmlet

https://www.npmjs.com/package/npx

This might help.

create-react-app doesn't work on Windows 10 for me either.  I'm able to create a project using:
npx create-react-app my-app

I then tried:
npm cache clean --force
delete node_modules folder
yarn install

Here's the output from yarn start:
$ react-scripts start
error Couldn't find the binary react-scripts start
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

npm -v react-scripts
6.14.9

import React, {component} from 'react';
import ReactDOM from 'react-dom';
class Demo extends Component{
render() {
    return
Hello World
;
}

}
export default Demo;
simple 'hello world' not work

import React, {component} from 'react';
import ReactDOM from 'react-dom';
class Demo extends Component{
render() {
    return
Hello World
;
}

}
export default Demo;
simple 'hello world' not work

It's quite hard to help you just by looking at this without seeing any error message you get (if you got any). Only thing I see is that "component" import should be "Component", with a capital C. import React, { Component } from 'react';
Next time please ask for help on Stackoverflow  or open a new issue with more details if you think there is an actual issue with create-react-app. And please also properly format your code. You only put few lines inside the code block and the rest is just outside of any code block.

I solved this using the following command.
npm uninstall -g create-react-app
npm cache clean --force
then
npx create-react-app project_name --template all
Thanks for this! Worked for me too.

When I tried it I got this response
Aborting installation.
npm install --no-audit --save --save-exact --loglevel error react react-dom react-scripts cra-template-all has failed.

When I tried it I got this response
_Aborting installation.
npm install --no-audit --save --save-exact --loglevel error react react-dom react-scripts cra-template-all has failed._


Missing LICENSE file

Since you do not have a LICENSE file, the copyright is unclear. Am I or a company I may work for allowed to use this material and under what conditions?

Thanks, good point. I'll add one

Licensed under MIT, btw. Thanks for pointing this out!


Function denormalize throws RangeError: Maximum call stack size exceeded

I still haven't been able to figure out how or why, but sometimes the function denormalize in normalizers/denormalize.js, will go into an infinite recursion loop while loading data.
Uncaught RangeError: Maximum call stack size exceeded
    at denormalize.js:21
    at Array.reduce (<anonymous>)
    at denormalize (denormalize.js:21)
    at denormalize (denormalize.js:7)
    at denormalize.js:24
    at Array.reduce (<anonymous>)
    at denormalize (denormalize.js:21)
    at denormalize (denormalize.js:7)
    at denormalize.js:12
    at Array.map (<anonymous>)

Not sure where to start debugging it...
The trace starts in getQuerySelector, so I'm trying to use that to obtain the result of a normalized request.

I probably know what this is... see https://github.com/klis87/redux-saga-requests/issues/317
Probably you have very many items in data, as well as very nested data for this query. Above issue would solve it. This could be also a different problem, but I cannot help without seeing data.

Here is my data, I will continue debugging but if you see something that strikes your eyes. It might be that I'm trying to get the  normalized data directly from the normalizedData dict... I still haven't figured out a way to get the normalized objects out of there ;)

@zeraien do you mean that you are reading normalizedData from requestsReducer on your own? this is internal implementation detail and should not be read manually
getQuery denormalizes data automatically, you don't need to think about it
pasted data doesnt look scary, I tested this with 10 000 items + and it still worked and still was fast, so probably you are doing sth unexpected, some snippet with get the normalized data directly from the normalizedData  would probably help :)

@zeraien do you mean that you are reading normalizedData from requestsReducer on your own? this is internal implementation detail and should not be read manually
getQuery denormalizes data automatically, you don't need to think about it
pasted data doesnt look scary, I tested this with 10 000 items + and it still worked and still was fast, so probably you are doing sth unexpected, some snippet with get the normalized data directly from the normalizedData  would probably help :)

Well, sometimes I just need to obtain objects by ID from the cache, because it's not always that objects are in a request, maybe you just need a related object in a different context, but now of course I realize that it's not a good idea because the data in the normalizedData is normalized! I'll do some more digging and see if I can figure it out, it may very well not be an issue with your code, but with mine (most likely) hehe.

I'm starting to think there might be something weird going on with the denormalize function after all.
Here is what path looks like after a few iterations...
.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints.parcels.activities.waypoints
As you can see it's repeating. I will do more digging.

Ok, so I found the cause. It's a recursion bug alright
Here is an abridged version of my JSON that causes the recursion loop:
{
	"id": "zVid0ty",
	"date": "2019-12-27",
	"barcode": "49394dcd-cd1b-46e8-82a2-2544c828d199",
	"identifier": "rdT6T9uz",
	"activities": [{
		"id": "A1ibDFG2"
	}, {
		"id": "40i9vHdK"
	}],
	"parcels": [{
		"id": "xwiA3sO",
		"activities": [{
			"id": "A1ibDFG2"
		}, {
			"id": "40i9vHdK"
		}]
	}]
}

If I remove the activities key from parcels, the problem disappears. You can argue that activities have no business being in parcels since they are already up on the top level, but it should probably still not crash :)
This works:
{
	"id": "zVid0ty",
	"date": "2019-12-27",
	"barcode": "49394dcd-cd1b-46e8-82a2-2544c828d199",
	"identifier": "rdT6T9uz",
	"activities": [{
		"id": "A1ibDFG2"
	}, {
		"id": "40i9vHdK"
	}],
	"parcels": [{
		"id": "xwiA3sO",
		"activities": null
	}]
}

Finally I narrowed down that the following normalizedData causes recursive loop in denormalize...
{

    '@@xwiA3sO': {
      id: 'xwiA3sO',
      activities: [
        '@@A1ibDFG2',
        '@@40i9vHdK'
      ],
    },
    '@@zVid0ty': {
      id: 'zVid0ty',
      activities: [
        '@@A1ibDFG2',
        '@@40i9vHdK'
      ],
      parcels: [
        '@@xwiA3sO'
      ],
    },
    '@@40i9vHdK': {
      id: '40i9vHdK',
      parcels: [
        '@@xwiA3sO'
      ],
    },
    '@@A1ibDFG2': {
      id: 'A1ibDFG2',
      parcels: [
        '@@xwiA3sO'
      ],
    }
}

And now I realise that I guess that data structure simply won't work since it will indeed be infinitely recursive... my stupid.

@zeraien if getQuery works then everything is fine
What you try to do is symptom we need new feature -> get object by id, the problem is that we need usedKeys etc which you can inspect in requestsReducer, it is possible some stuff will need to be refactored.
For now I hope you will solve it on the app level, but I will add this to TODO list so this will be built-in.

@zeraien I thought about it and there is 1 issue.
Imagine object like this fetched at /users/1
{
  id: 1,
  name: 'name',
  bestFriend: {
      id: 2,
      name: 'name 2',
      bestFriend: {
        id: 1,
        name: 'name',
      }
    }
  }
}

As you can see denormalizing would indeed cause endless loop. For queries it is not a problem thanks to usedKeys, which are automatically calculated based on query response structure and stored in reducer.
So based on the example information that nested user has only id and name key prevents this issue.
Now we want to allow to get an orbitrary object. For this case we dont have usedKeys, normalized object contains all used keys from all queries referencing a given object merged in 1 place. Then we don't have protection and can end up with recursive reference issue and so on.
Do you have any idea how we could solve that?

Also, the question is why do you even need to do that? Perhaps what you need is just action.meta.requestKey, so that you could have more queries of the same type stored inside reducer, like with multiple ids.

Good point, so it can indeed become an issue!
Well, there are a few solutions I guess, one is to turn relationships into function calls, so you need to call a function to get the relationship data...
{
  id: 2,
  name: 'friend 2',
  bestFriend: () => get("@@1") -> {id:1, name:'friend 1', bestFriend:  get()}
},
{
  id: 1,
  name: 'friend 1',
  bestFriend: () => get("@@2")
}

But that brings with it other complications, in theory, whoever has to work with that data structure risks going into an infinite loop themselves...
Another alternative is to simply cut the feed at level 2.
{
  id: 2,
  name: 'friend 2',
  bestFriend: {
      id: 1,
      name: 'friend 1'
    }
  }
},
{
  id: 1,
  name: 'friend 1',
  bestFriend: {
      id: 2,
      name: 'friend 2'
    }
  }
}

I guess you can check if the parent has the same key and then simply not include that key. But that risks mangling someone's data, although you could mitigate with a console warning...
Alternatively, simply find a way to detect the infinite loop and warn the user so they can change the API...

Relations has to be stored as strings, this has to be serializable.
Introducing a warning is 1 idea, but in our example API really is just fine, it just has recursive relationship but for queries with usedKeys this is not a problem at all. So we should not ask people to change API if API is fine :)
Another idea is to limit nesting depth, but like you said this would mangling data.
I think about catching this error indeed and showing a message like data has recursive structure, please provide usedKeys for this data to denormalize it properly which could be also documented
So actually app user would provide proper usedKeys for a give object oneself. I cannot see any other way.
But the most important question is, why do you even need this? Please provide use case, maybe I will give you another way to achieve what you need.

I don't really need it, it was just my API was configured in this way and so I changed my API to compensate.
I am not quite sure about usedKeys, would we provide it in the action creator?

@zeraien usedKeys would be provided to new selector like getNormalizedObjectById
normally this selector would just get id and it would return proper object, but if leading to recursive error, this would be catched and warning would be shown in console.
then someone could do thing like:
const myObject = getNormalizedObjectById(state, { id: '1', usedKeys: {
  id: true,
  name: true,
  bestFriend: {
    id: true,
    name: true,
  }
}}});

So usedKeys would be provided in easy to work with way by human, while library would convert it to the same structure as stored in reducer. Probably this would be used very sparingly, only when recursive relationship, but we would have an escape hatch so I could work on getNormalizedObjectById

actually this could be represented as graphql query, like:
{
id
name
bestFriend {
id
name
}
}
in theory such parsing could be also allowed via graphql driver

if I think about it more, this looks like graphql fragment concept, but for any api, including REST


Change cmake command to include Release type

Fix the README to not generate a debug executable, else, the generate test example is extremely slow, roughly 14 times slower.

Thanks so much for this cleanup. Merged.


Move type_id out of DynResolver and into SerializeDyn

This will make all reference resolvers a usize, so we can restrict it to guarantee that constraint. This will help make SharedWriter more performant since it won't have to make allocations.

Fixed by a720fa4ec8bbf8f566c49b32935654c82de66bb5

Fixed by a720fa4ec8bbf8f566c49b32935654c82de66bb5


add installation commands and make Qt4 components optional

This patch adds installation commands into CMakeLists.txt and makes components that require Qt4 optional.

@lsolanka-flya thanks for the contribution


Add BenSampo/laravel-enum support (checkEnum / checkEnumNullable)

Hi, I added a support for BenSampo::Enum in this PR.
In my project currently working on using Enums for statuses / types of entities instead of using id/name table like admin_roles or admin_statues.
My client request me to add a check constraint for the database column, I found your repository and decided to add enum support on your code.
I would be happy if you could merge this. Please comment if there is anything.
thanks.

Hi,
thank you for your interest and your pull request! I'm sorry I haven't replied yet.
I looked at your code and think I won't merge it unfortunately. Personally, I do not use that package and more importantly: if I see it correctly, you might as well create an actual enum column like so:
$table->enum('role', ['draft', 'published', 'archived'])->nullable();

Both solutions would require another migration if the list of enum values changes at a later point, so I don't really see the benefit here. Or am I missing something?
Of course you can still use this package and simply register your additional macros in your own AppServiceProvider.
Thank you for your understanding and best regards!

@dbakan Thanks for reviewing and commenting on my PR.
It just because my project not permitted enum type on DB and the short string columns are defined instead.
And also there are php Enum classes are already defined in project.
So that I need the checkEnum macro for BenSampo::Enum like this.
$table->string('type', 191)->index()->comment('card type');
$table->checkEnum('type', CardType::class);

Anyway, I'll use this as macro which you suggested. thanks.

Of course you can still use this package and simply register your additional macros in your own AppServiceProvider


FLY BUG: MaxForce and MaxTorque

Location: Server.Dependencies.Assets.Fly
I fixed it by doing this here on the line where it has bPos.MaxForce and bGyro.MaxTorque. The same thing happens in function Stop()
local as = Vector3.new(math.huge, math.huge, math.huge)
bPos.Position = part.Position
bPos.MaxForce = as

bGyro.CFrame = part.CFrame
bGyro.MaxTorque = as

Do you happen to have any plugins?

Yes, I have but it's only for Discord Webhook and delivering tags.

This has been fixed (at least in the nightly) so the issue can be closed

Awesome


Migrate startup sample container project to .devcontainers

Describe the issue

Current version of container startup-sample-project-aws-containers supports local deployment across Windows, Linux and Mac.
During setup, make uses OS environment to determine appropriate setup script for detected OS

Additional context

There's the possibility to migrate this setup to use .devcontainer, eliminating the need to have OS specific scripts during setup
Investigate use of Docker in Docker.

Definition of done

All dependencies for the dev environment are pre-packaged in the .devcontainer
Build, run and debug the application in the .devcontainer.
out of scope: No codespaces work in this sprint

Closing against PR-68


Config system improvements: compositional config, added scenes_dir, removed config_dir, fixed eqa test

https://github.com/facebookresearch/habitat-api/issues/63: Added scenes_dir parameter to Config to have no requirement of data symlink or data directory in working directory. scenes_dir will be used as prefix for scene_id during  episodes dataset loading.
Added overwriting options argument to get_config that unblocks parameter sweeping and experimentation capabilities from command line
https://github.com/facebookresearch/habitat-api/issues/29: config_file path can be comma separated list of configs that will be merged in exact order. That will help to use separate configs for dataset specification and sensors setups. String with commas was chosen because that make it easy to pass list of configs as script argument.
To make habitat-api easier to as package we decided to get rid of forcing loading of configs from config_dir param
https://github.com/facebookresearch/habitat-api/issues/75: fixed unit test caused by latest habitat-sim update

Caution: with this change all config paths will be specified in new way: tasks/poingoal.yaml => configs/tasks/pointgoal.yaml
How Has This Been Tested
pytest
black . --line-length 79;mypy . --ignore-missing-imports;pycodestyle;flake8
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.6.4, pytest-4.4.1, py-1.5.2, pluggy-0.9.0
rootdir: /private/home/maksymets/habitat-api, inifile: setup.cfg, testpaths: test
plugins: xdist-1.25.0, forked-0.2
collected 37 items

test/test_baseline_agents.py ..                                                                                [  5%]
test/test_config.py .                                                                                [  8%]
test/test_dataset.py .........                                                                                [ 32%]
test/test_habitat_env.py .........s                                                                                [ 59%]
test/test_habitat_example.py ...                                                                                [ 67%]
test/test_install.py .                                                                                [ 70%]
test/test_mp3d_eqa.py ...                                                                                [ 78%]
test/test_pointnav_dataset.py ...                                                                                [ 86%]
test/test_sensors.py ...                                                                                [ 94%]
test/test_trajectory_sim.py ..                                                                                [100%]
========================================================================================================== short test summary info ===========================================================================================================
SKIPPED [1] test/test_habitat_env.py:318: unconditional skip

=================================================================================================== 36 passed, 1 skipped in 210.82 seconds ===================================================================================================

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have completed my CLA (see CONTRIBUTING)
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Addressed the comments. Added support of config list, corrected scene_path, added overwriting options, added EQA dataset changes, adopted quaternion suggestion.


Feature: multi-user branch PR 1 - create password settings screen

FYI, I am going to create a series of PRs that we can work against for adding the multi-user support.  These should all be worked as PRs off the feature/multi-user branch (ie I have checked in that branch, please branch from it and then make PRs against it, when all PRs are complete we will merge to main)
The first one is simple: to add a new panel/form that allows an admin-level user to specify the rules for creating passwords.  Eventually, only admin levels will see this; to start with since we only have admin users, the PR should just add the new panel/form.
Looking for input on UI from @kitajchuk here) but my thought was that was we already have a "Settings" panel where this could live, except that panel has Service Provider level settings, and this is an admin-level setting.  However, maybe we extend/enhance that Settings panel to have two sections (or tabs) labeled 'Admin' and 'Service Provider'?  (In a later PR we can worry about hiding the Admin tab from a service provider user, for this PR we dont need to).
I'm assigning to @kitajchuk first for UI thoughts (and do we need an actual UI worked up in Figma, or is that overkill?).  When completed and agreed, this issue should then be assigned to @xquanluu for implementation

Looking for input on UI from @kitajchuk here) but my thought was that was we already have a "Settings" panel where this could live, except that panel has Service Provider level settings, and this is an admin-level setting. However, maybe we extend/enhance that Settings panel to have two sections (or tabs) labeled 'Admin' and 'Service Provider'? (In a later PR we can worry about hiding the Admin tab from a service provider user, for this PR we dont need to).

I think the tabs could work. The only other option would be to have another link in the nav for an entirely new route for Admin settings and I'm not really into that notion lol. For the tabs would we default to Admin tab in first position or Service Provider tab in first position? Note that this only matters for admins long-term as the Service Provider user role would only see the one tab for them anyway. Also note that my intention would be to place the tabs in the form section (same as Carrier form) so the API keys section would be visible no matter which tab you are on—is this okay?

I would say admin in first position, then service provider.  And yes, seeing the API keys would be fine -- but maybe we relabel them to "Service Provider API Keys" to be clear?

@kitajchuk  @xquanluu  I am keen to get started on this...the multi-user work needs to be the top priority now.  With the decision above, what else do we need to get Hoan started on this?

Start working on this. I will try to work with above discussion and then we can review and change like other PR we did when seeing something bad

I want to get confirmation that the settings should be like this layout.

The tabs look good.  But for the admin settings, I want something easier to use than regex.  It should be simple checkboxes and a dropdown for min password length.  Here is an example -- not exactly on target since it is for generating a password vs password rules -- but look at how intuitive it is:

Can you come up with something like that?  e.g.

Minimum password length (dropdown, defaulting to 8, choices from 6 to 20)
Require at least one digit (checkbox)
Require at least one special character: !@#$%^&*()+ (checkbox)

@davehorton yes sure, I will work on that soon.

yes, maybe work on that first, actually, while we wait for @kitajchuk  to chime in on the UI

I want to get confirmation that the settings should be like this layout.

@xquanluu We discussed putting the tabs inside the form section similar to how we did on the carrier view. That would be a desirable change since as they are now they look/feel broken or out of place.

Thanks @kitajchuk sure fixing it now
@davehorton api-server PR created https://github.com/jambonz/jambonz-api-server/pull/65/files

I just made this change to the UI, please help to check if this is fine?

Or the Tabs should be like this?

Or the Tabs should be like this?

In my prior comment I said “ In this case we can render the tabs right below the first field set with the message about required fields.” That’s what we would like to see here so none of these are doing that 🤔

ha ha, I'm sorry, my bad again...

Is this what you means?

Close but there’s double padding around the tabs. You may have wrapped the tabs in a field set which you don’t need to do. We have some styles that target the UI tabs in the forms already:
https://github.com/jambonz/jambonz-webapp/blob/052ea4d6cb79b2bd55218bfcb738624d28ae471a/src/styles/_forms.scss#L86
And if you look at the carrier form you’ll see we don’t wrap the tabs in a field set:
https://github.com/jambonz/jambonz-webapp/blob/052ea4d6cb79b2bd55218bfcb738624d28ae471a/src/containers/internal/views/carriers/form.tsx#L615

Thanks a lot @kitajchuk , I fixed it,
here is first draft PR: https://github.com/jambonz/jambonz-webapp/pull/130
It's really great if I can have your review and give comment if there is issue, anti-pattern ..etc in the PR so I can fix it.

@EgleHelms I think all this PR needs is to be rebased off the latest main branch.  Could you do that?  Also, make sure to pull the latest main branch of jambonz-api-server for testing, as I just merged in the API (/PasswordSettings) that this feature uses.


Determine DMA functions to understand each sections of the game [many calls in "SCPS_101.05"]

The game uses a number of DMA functions that are common to the PSX library, and will be our best shot to narrow down functions created/used specifically in the game.
This is a LARGE investigation, that will need to be done using Ghidra and the PSX loader.
Below are the functions we are aware of, but there are more that need to be found.

FUN_80020170(decompress)
FUN_8007d334(LoadImage)

GOAL:
Determine function that handle the following:

Font display
Text Decompression
Image Decompression
Overlay [the system that handles the displaying of text in text boxes]
TBA

A number of new functions have been identified & tracked in the wiki specific page for this file here.
A short list so far of functions that Ghidra does not immediate rename correctly are:

FUN_8007d2cc = DrawSync
FUN_8007d234 = SetDispMask
FUN_8007cf50 = ResetGraph
FUN_8007d0c4 = SetGraphDebug
FUN_8007d334 = StoreImage
FUN_8007d7b8 = ClearOTagR
FUN_8007d1d4 = DrawSyncCallback
FUN_8001f74c = SystemError
FUN_800872a4 = GetVideoMode
FUN_800976f0 = SpuVoiceAttr
FUN_8009b830 = SpuVolume
FUN_80099480 = SpuInitHot
FUN_8007ba20 = SetGeomOffset
FUN_8007ba40 = SetGeomScreen
FUN_8009d0c0 = SsSetStereo

You should also keep an eye on the following functions, since we think they will play a role in other areas:

FUN_8001bc9c - Main ASCII Font Controller(?)
FUN_8001fb94 - Related to "Ovr Prog" (Prgressive Overlay)
FUN_8001fd8c - Related to "CnstOvr" (Constant Overlay)
FUN_8002a4b8 - Mentions of "DEMO PLAY" "DEMO REC" etc.


Could spring stream add a annotation "StreamFunction" ?

Add a annotation StreamFunction, indicate some function is used in stream. Avoid centralized configuration of spring.cloud.function.definition
For example:
`
@Bean
@StreamFunction
public Function<String, String> toUpperCase() {
return s -> s.toUpperCase();
}
@Bean
@StreamFunction
public Supplier date() {
return () -> new Date(12345L);
}
`

I wonder what value would it add other then bring us back to where we once were with @StreamListener and mixed configuration. The spring.cloud.function.definition is just one of the many configuration properties, so even if we did what you propose you would still need to configure the rest externally, hence mixed configuration which is never a good idea as it is hard to follow.
Please clarify. . .