[Android] path Must not be null

Crashes on Android. iOS works fine.
To get the image I use image_picker and pick the image from gallery. If I pick from camera it works fine, so the issue is only when getting the image from gallery.
I/flutter (19871): The following ArgumentError was thrown:
I/flutter (19871): Invalid argument(s) (path): Must not be null
I/flutter (19871):
I/flutter (19871): When the exception was thrown, this was the stack:
I/flutter (19871): #0      ArgumentError.checkNotNull (dart:core/errors.dart:181:27)
I/flutter (19871): #1      new _File (dart:io/file_impl.dart:210:19)
I/flutter (19871): #2      new File (dart:io/file.dart:255:18)
I/flutter (19871): #3      FlutterImageCompress.compressAndGetFile (package:flutter_image_compress/flutter_image_compress.dart:90:12)
I/flutter (19871): <asynchronous suspension>

flutter doctor -v:
[✓] Flutter (Channel stable, v1.2.1, on Mac OS X 10.14.4 18E226, locale en-GB)
    • Flutter version 1.2.1 at /Users/tudor/dev/flutter
    • Framework revision 8661d8aecd (9 weeks ago), 2019-02-14 19:19:53 -0800
    • Engine revision 3757390fa4
    • Dart version 2.1.2 (build 2.1.2-dev.0.0 0a7dcf17eb)

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /Users/tudor/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = /Users/tudor/Library/Android/sdk
    • Java binary at: /Users/tudor/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/182.5314842/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)
    • All Android licenses accepted.

[✓] iOS toolchain - develop for iOS devices (Xcode 10.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 10.2, Build version 10E125
    • ios-deploy 1.9.4
    • CocoaPods version 1.6.1

[✓] Android Studio (version 3.3)
    • Android Studio at /Users/tudor/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/182.5314842/Android Studio.app/Contents
    • Flutter plugin version 34.0.1
    • Dart plugin version 182.5215
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)

[✓] Connected device (1 available)
    • SM G950F • 988a1644544d44384c • android-arm64 • Android 9 (API 28)

• No issues found!

As given in the information: Path must not be null.
You have to make sure that the incoming path is not empty in your code.
In image_picker, if you exit without selecting a image, the result is null, so if you use file?.path to get a path, the path is null. Then passing this value to the native will cause problems.
I will deal with this situation in the code, but you also need to make sure the params is not null.

close by #41

@CaiJingLong Can you please re-open this issue ? Sorry, but I disagree with this behaviour of quickly closing issues based on your personal assumptions.
I don't pass null to this plugin and I don't exit image_picker without selecting an image. I send valid data and it fails. I don't know what the exception is because I took a look at the code and there is an catch(Exception ) {result(null)} line which doesn't log the exception.

I'm not assuming to close the issue.
I modified it to handle the case where the input parameter is null from the dart layer.
If you still have problems using this library, please paste the log here.
The Android section can use ADB logcat to view native logs.

Thanks. The issue with pasting the logs is that I think the library crashes on native side, but the exception is swallowed here: https://github.com/OpenFlutter/flutter_image_compress/blob/f18bd08f3cfd923f101eb817d82f6d024ef17021/android/src/main/kotlin/com/example/flutterimagecompress/CompressFileHandler.kt#L50
I think if the exception would be logged we would be able to see it.

Then post the log here and I will try to help you solve this problem.

Yes, now I can get the logs. Looks like it crashes because of storage permission.
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err: java.io.FileNotFoundException: /storage/emulated/0/DCIM/Camera/20190414_210950.jpg (Permission denied)
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err:     at java.io.FileOutputStream.open0(Native Method)
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err:     at java.io.FileOutputStream.open(FileOutputStream.java:308)
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err:     at java.io.FileOutputStream.<init>(FileOutputStream.java:238)
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err:     at java.io.FileOutputStream.<init>(FileOutputStream.java:180)
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err:     at com.example.flutterimagecompress.CompressFileHandler$handleGetFile$1.run(CompressFileHandler.kt:46)
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
2019-04-17 16:53:26.500 11826-13419/{my_app}} W/System.err:     at java.lang.Thread.run(Thread.java:764)
2019-04-17 16:53:26.511 11826-11948/{my_app}} I/flutter: Another exception was thrown: NoSuchMethodError: The getter 'path' was called on null.

It's strange, why does it need storage permission ? For example flutter_native_image (another library for compression) works without storage permission.

Looks like this library tries to write to Camera folder while the other library writes to Context.getExternalCacheDir() which is a directory owned by the app.

No, the path and targetPath of flutter_image_compress is provided by the caller, who should ensure permission for the directory.
https://github.com/OpenFlutter/flutter_image_compress/blob/8d8a49dd38c70122f47e055731a5a06b815f0705/lib/flutter_image_compress.dart#L91-L98

Now I understand. Yes, I think it's a good idea for this to be configurable.
Thanks a lot for the help, you can close this issue.

In fact, I added the same question to Readme long ago.
https://github.com/OpenFlutter/flutter_image_compress#troubleshooting


Zero Networks v1.0.0

Required items, please complete
Change(s):

Created Azure Function based data connector
Added Zero Networks Logo
Created Custom Connector and 3 Playbooks
Created sample data
Created Solution:
3 Analytics Rules
Native Poller Data Collector
4 Hunting Queries
Parser
Playbook (linked deployments)
Workbook.

Reason for Change(s):

New integration with Zero Networks

Testing Completed:

Yes

Checked that the validations are passing and have addressed any issues that are present:

Yes

@aprakash13 Please review analytic rules. Thanks.

@anki-narravula Please review the Playbooks. thanks!!!

@dicolanl In sample data, the columns timestamp_d,auditType_d,enforcementSource_d,userRole_d have the numeric value but as you have added " " to values they are been saved with _s rather than _d , please check and update, thanks.

@dicolanl In checks list Data connector validations failing due to DataConnector_API_AzureFunctionApp_ZeroNetworks_AccessOrchestrator_Audit.json does not match with Azure Function Connector Template.
Path reference : https://github.com/Azure/Azure-Sentinel/blob/master/DataConnectors/Templates/Connector_REST_API_AzureFunctionApp_template/DataConnector_API_AzureFunctionApp_template.json

@aprakash13 Please review analytic rules. Thanks!!!

vmanojreddy

This error

@dicolanl In checks list Data connector validations failing due to DataConnector_API_AzureFunctionApp_ZeroNetworks_AccessOrchestrator_Audit.json does not match with Azure Function Connector Template. Path reference : https://github.com/Azure/Azure-Sentinel/blob/master/DataConnectors/Templates/Connector_REST_API_AzureFunctionApp_template/DataConnector_API_AzureFunctionApp_template.json

Fixed!

@anki-narravula Please review the Playbooks. thanks!!!

@aprakash13 Please review analytic rules. Thanks!!!

@dicolanl Please move the files into solution folder. thanks!!!

@vmanojreddy what files exactly? most are already in solution folder, except for a few things like logo, function app connector and playbooks.

@dicolanl  Yes, Can You please move  function app connector and playbooks into solution folder. thanks

@dicolanl Please move the files into solution folder. thanks!!!

@vmanojreddy what files exactly? most are already in solution folder, except for a few things like logo, function app connector and playbooks.

@dicolanl Yes, Can You please move function app connector and playbooks into solution folder. thanks

@vmanojreddy done

@aprakash13 Please review analytic rules. Thanks!!!

@dicolanl In Data connector zip folder should not be in correct format. Can you please check once. thanks!!!

@dicolanl In Data connector zip folder should not be in correct format. Can you please check once. thanks!!!

/azp run

Hey @dicolanl ,
Sorry for responding late.
I saw that the zip folder we are using here is not in a correct format. We need to change the structure for that. Inside zip folder there is another folder and then contents. It should be contents directly inside the zip folder. We don't use another folder inside zip. That is causing some deployment issues.
Thanks
Nikhil

Hi @dicolanl Also after deploying the data connector I am not able to get the function. I checked the .zip folder and I found out that there are no function added in zip folder. Can you please update the zip package with all the required content. Here is a zip package link for you reference. https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/Cloudflare/Data Connectors/CloudflareConn.zip
Thanks
Nikhil

@rushriva Please review the Playbooks and provide your sign. thanks!!!

Hi @dicolanl,
Functions are not loading. Can you please look into it?

This will not work until it’s merged becuase the url for the zip files doesn’t actually exist until it’s merged in.

Hi @dicolanl,
Functions are not loading. Can you please look into it?

This will not work until it’s merged becuase the url for the zip files doesn’t actually exist until it’s merged in.

Hi @dicolanl
After updating the link it is still not working. I used the PR zip link in configuration.

Hi @dicolanl,
Functions are not loading. Can you please look into it?

This will not work until it’s merged becuase the url for the zip files doesn’t actually exist until it’s merged in.

Hi @dicolanl After updating the link it is still not working. I used the PR zip link in configuration.

@NikTripathi  try now.  found an issue with the zip.  also fixed an issue with storage account naming.

@NikTripathi @vmanojreddy @preetikr
What else is needed to merge this??

Hi @dicolanl can you please confirm us why we are using two types connectors function app based connector and ccp connector?

@vmanojreddy because according to this CCP is in preview.  Some customers will be unwilling to use a preview feature so I created the azure function model as this is supported.


Found potential runtime exception

Hello,
First of all i want to thank you for sharing your code.
But when I was operating some of my files, there was a runtime exception of the type 'java.lang.IlegalArgumentException'. The runtime exception was from method:
com.redis.util.JdomUtils.getIntArrayAttriValue(org.jdom.Element,java.lang.String)
    public static int getIntAttriValue(Element e, String attrName) {
        String attributeValue = e.getAttributeValue(attrName);
        if (attributeValue == null) {
            return -1;
        }
        return Integer.valueOf(attributeValue);
    }

it might cause the program to crash. But the caller may not know these problems and may cause the program to crash suddenly.
so I suggest you add catch or throw to the source code An exception goes out, which is convenient for the caller to develop. Thank you!
The issue can be reproduced by passing the method with non-number string

thanks for your help


Controller background RSSI not collected for a controller-only network

Is your problem within Home Assistant (Core or Z-Wave JS Integration)?
NO, my problem is NOT within Home Assistant or the ZWave JS integration
Is your problem within Z-Wave JS UI (formerly ZwaveJS2MQTT)?
NO, my problem is NOT within Z-Wave JS UI
Checklist

[X] I have checked the troubleshooting section and my problem is not described there.

[X] I have read the changelog and my problem was not mentioned there.

Describe the bug
What causes the bug?
A single node network, with only the controller.
What do you observe?
Controller background RSSI is not collected when the network is only a controller. Once you add a node, it starts to work.
What did you expect to happen?
The background RSSI is collected even without other nodes.
Obviously, not a critical issue, but as discussed in Slack, this is expected to work.
Steps to reproduce the behavior:

Start a network with only the controller
Watch the driver logs or statistics events for Background RSSI, observe it is never reported
Add a node to the network
Observe background RSSI is now collected

Device information
No response
How are you using node-zwave-js?

[ ] zwave-js-ui (formerly zwavejs2mqtt) Docker image (latest)
[ ] zwave-js-ui (formerly zwavejs2mqtt) Docker image (dev)
[ ] zwave-js-ui (formerly zwavejs2mqtt) Docker manually built (please specify branches)
[ ] ioBroker.zwave2 adapter (please specify version)
[ ] HomeAssistant zwave_js integration  (please specify version)
[ ] pkg
[ ] node-red-contrib-zwave-js (please specify version, double click node to find out)
[ ] Manually built from GitHub (please specify branch)
[X] Other (please describe)

Which branches or versions?
version:
node-zwave-js branch: 10.14.0
zwave-js-server standalone.
Did you change anything?
no
If yes, what did you change?
No response
Did this work before?
Don't know, this is a new device
If yes, where did it work?
No response
Attach Driver Logfile
I left the server running for about a day before adding a node. You can see the driver logs are silent during that time. Immediately when a node is added, RSSI reports are shown.

Log Snippet of example time period
2023-04-05T22:59:00.149Z DRIVER   driver ready
Starting server on <IP_ADDRESS>:3000
2023-04-05T22:59:00.153Z CNTRLR   [Node 001] The node is alive.
2023-04-05T22:59:00.153Z CNTRLR   [Node 001] The node is ready to be used
2023-04-05T22:59:00.154Z CNTRLR   All nodes are ready to be used
ZwaveJS server listening on <IP_ADDRESS>:3000
New client
New client
2023-04-07T00:30:11.388Z CNTRLR   Starting inclusion process with strategy Default...
2023-04-07T00:30:11.397Z SERIAL » 0x0105004ac10170                                                     (7 bytes)
2023-04-07T00:30:11.398Z DRIVER » [REQ] [AddNodeToNetwork]
                                    node type:    Any
                                    high power:   true
                                    network wide: true
                                    callback id:  1
2023-04-07T00:30:11.403Z SERIAL « [ACK]                                                                   (0x06)
2023-04-07T00:30:11.405Z SERIAL « 0x0107004a01010000b2                                                 (9 bytes)
2023-04-07T00:30:11.406Z SERIAL » [ACK]                                                                   (0x06)
2023-04-07T00:30:11.406Z DRIVER « [REQ] [AddNodeToNetwork]
                                    status:      Ready
                                    callback id: 1
2023-04-07T00:30:11.410Z CNTRLR   The controller is now ready to add nodes
2023-04-07T00:30:13.404Z CNTRLR   stopping inclusion process...
2023-04-07T00:30:13.414Z SERIAL » 0x0105004ac50277                                                     (7 bytes)
2023-04-07T00:30:13.414Z DRIVER » [REQ] [AddNodeToNetwork]
                                    action:       Stop
                                    high power:   true
                                    network wide: true
                                    callback id:  2
2023-04-07T00:30:13.420Z SERIAL « [ACK]                                                                   (0x06)
2023-04-07T00:30:13.422Z SERIAL « 0x0107004a02060000b6                                                 (9 bytes)
2023-04-07T00:30:13.423Z SERIAL » [ACK]                                                                   (0x06)
2023-04-07T00:30:13.423Z DRIVER « [REQ] [AddNodeToNetwork]
                                    status:      Done
                                    callback id: 2
2023-04-07T00:30:13.427Z CNTRLR   The inclusion process was stopped
2023-04-07T00:30:18.433Z SERIAL » 0x0103003bc7                                                         (5 bytes)
2023-04-07T00:30:18.433Z DRIVER » [REQ] [GetBackgroundRSSI]
2023-04-07T00:30:18.438Z SERIAL « [ACK]                                                                   (0x06)
2023-04-07T00:30:18.439Z SERIAL « 0x0107013b9596969fc8                                                 (9 bytes)
2023-04-07T00:30:18.440Z SERIAL » [ACK]                                                                   (0x06)
2023-04-07T00:30:18.441Z DRIVER « [RES] [GetBackgroundRSSI]
                                    channel 0: -107 dBm
                                    channel 1: -106 dBm
                                    channel 2: -106 dBm

zwave_js.log

Interestingly this does not reproduce for me. This is after a factory reset and restart of Z-UI:
19:57:10.910 CNTRLR   [Node 001] The node is alive.
19:57:10.942 CNTRLR   [Node 001] The node is ready to be used
19:57:10.942 CNTRLR   All nodes are ready to be used
19:57:10.945 SERIAL » 0x0104000b13e3                                                       (6 bytes)
19:57:10.946 DRIVER » [REQ] [SerialAPISetup]
                        command: GetPowerlevel16Bit
                        payload: 0x13
19:57:10.978 SERIAL « [ACK]                                                                   (0x06)
19:57:10.979 SERIAL « 0x0108010b1300000021cf                                              (10 bytes)
19:57:10.980 SERIAL » [ACK]                                                                   (0x06)
19:57:10.982 DRIVER « [RES] [SerialAPISetup]
                        command:               GetPowerlevel16Bit
                        normal powerlevel:     0.0 dBm
                        output power at 0 dBm: 3.3 dBm
19:57:10.986 SERIAL » 0x0104000b20d0                                                       (6 bytes)
19:57:10.987 DRIVER » [REQ] [SerialAPISetup]
                        command: GetRFRegion
                        payload: 0x20
19:57:10.992 SERIAL « [ACK]                                                                   (0x06)
19:57:10.994 SERIAL « 0x0105010b20ff2f                                                     (7 bytes)
19:57:10.994 SERIAL » [ACK]                                                                   (0x06)
19:57:10.996 DRIVER « [RES] [SerialAPISetup]
                        command: GetRFRegion
                        region:  Default (EU)
19:57:11.049 DRIVER   Usage statistics sent - next transmission scheduled in 23 hours.
19:57:12.480 DRIVER   No configuration update available...
19:57:16.008 SERIAL » 0x0103003bc7                                                         (5 bytes)
19:57:16.011 DRIVER » [REQ] [GetBackgroundRSSI]
19:57:16.015 SERIAL « [ACK]                                                                   (0x06)
19:57:16.017 SERIAL « 0x0107013b9095957f2d                                                 (9 bytes)
19:57:16.018 SERIAL » [ACK]                                                                   (0x06)
19:57:16.023 DRIVER « [RES] [GetBackgroundRSSI]
                        channel 0: -112 dBm
                        channel 1: -107 dBm
                        channel 2: -107 dBm

Yeah that may be it. I'll try and reproduce later with the bare driver.


Implement easy access query parameters and request body validation based on query parameters

Description
This PR fixes https://github.com/sansyrox/robyn/issues/289 and may help with https://github.com/sansyrox/robyn/issues/351 (since msgspec encoding is apparently much faster than the standard json lib)
Query validation is optional, may find another way to indicate whether to verify (or not) instead of saving it in
the FunctionInfo, or force the change in general (when the "full" type-checking and schema validation is implemented)
The schema validation method itself supports nested classes and does not require the user to use external libraries.
The general idea of the fix is:
When a request is handled, use the Python GIL to run the steps below using the Python interpreter (possible bottleneck):

Get the signature of the model using the features in the inspect module. The current PR supports custom user-defined (potentially nested) objects in the handler function signature.
Generate a msgspec.Struct based model for schema validation based on either the member variables of the class (if it does not have a user-defined init constructor, or the constructor signature itself)
Compare the model against the received request (by decoding the JSON result into the model object) to see if the schema validation fails; if it does, an InternalServer error is raised.
If schema validation succeeds, we convert the object generated by the model validation into a dictionary and pass it to the handler function as kwargs and get the output of the handler function as usual

Further extensions:

Implement process/thread-safe caching of Encoder and Decoder objects for the msgspec encode and decode calls. Since a call to either method requires some setup, multiple calls can be made faster if we use an Encoder/Decoder object for the respective calls.

Also @kliwongan , apologies for these one liner changes. I will do a thorough review either tomorrow or on Saturday 😄

Hi @kliwongan, and thanks for the PR !
I took a look at this too as it aims to bring an important new feature to Robyn. I agree with you that the current architecture isn't the best as it implies that we use Robyn's Python code in Robyn's Rust code. This could be an issue for separating Robyn's web server and Robyn's web framework in the future AFAIK (see #268), and also decrease performances. I would suggest indeed to do the validation in Rust directly, which will probably and unfortunately be much harder :disappointed:  Maybe we could use or be inspired by projects like pydantic-core which aims at migrating pydantic core validation to Rust ?
I don't know if @sansyrox agrees with me, but should we rewrite the data validation in Rust, I could try to help you with it :slightly_smiling_face:

Hi @kliwongan, and thanks for the PR ! I took a look at this too as it aims to bring an important new feature to Robyn. I agree with you that the current architecture isn't the best as it implies that we use Robyn's Python code in Robyn's Rust code. This could be an issue for separating Robyn's web server and Robyn's web framework in the future AFAIK (see #268), and also decrease performances. I would suggest indeed to do the validation in Rust directly, which will probably and unfortunately be much harder 😞 Maybe we could use or be inspired by projects like pydantic-core which aims at migrating pydantic core validation to Rust ? I don't know if @sansyrox agrees with me, but should we rewrite the data validation in Rust, I could try to help you with it 🙂

I have a buggy branch (not sure if it's my machine or not) that somewhat addresses this problem https://github.com/kliwongan/robyn/tree/buggy-serialize; I essentially store the "validator" function in the FunctionInfo and then call it when necessary; for some reason I am getting a lot of connection errors when running tests (pytest and sending requests using Postman).
Anyways, if required, I would also love to rewrite the query validation in Rust! I've seen FastAPI's code and I have a good idea on how to apply other kinds of dependency injection as well (at least on the Python side). I agree on perhaps taking inspiration from pydantic-core, it might be a good idea. We can perhaps mix it in with the "validator" concept I have, where the user can also use whatever model validation function they want, which would return the **kwargs that the handler needs.

Hey @kliwongan , thank you for your great work 😄 You PR shows a lot of hard work. Especially needing you to dive deep in Pydantic and FastAPI's codebase.
But as @AntoineRR suggested, we would love to have the validation written in the Rust side. As it makes the performance slower and makes it harder to segregate the responsibilities in the codebase.
Why we are emphasizing this much on writing the validation in Rust side is because we believe that we do the hard things so that the developers using Robyn don't have to. This way we will be to provide the best experience.
We are more than happy to help you out with the implementation if need be. 😄
Having said that, great work so far! ✨

Speaking of which, @kliwongan are you on our discord channel?

Speaking of which, @kliwongan are you on our discord channel?

I don't believe I am.

Hey @kliwongan , thank you for your great work 😄 You PR shows a lot of hard work. Especially needing you to dive deep in Pydantic and FastAPI's codebase.
But as @AntoineRR(thanks for the review btw 😄 ) suggested, we would love to have the validation written in the Rust side. As it makes the performance slower and makes it harder to segregate the responsibilities in the codebase.
Why we are emphasizing this much on writing the validation in Rust side is because we believe that we do the hard things so that the developers using Robyn don't have to. This way we will be to provide the best experience.
We are more than happy to help you out with the implementation if need be. 😄
Having said that, great work so far! ✨

Would you want to call the Python functions individually in Rust as you previously suggested? (i.e. import libraries like inspect and use callmethod1 on them) or would you work completely in Rust? I am a little confused.

@kliwongan , apologies for the confusion. I meant this. To write as much as possible in rust.

or would you work completely in Rust

I don't believe I am.
https://discord.gg/rkERZ5eNU8 , you can join here. For faster feedback if needed 😄

@kliwongan @AntoineRR , I have been thinking about this PR for a long time, and I have a suggestion. Let me know what you folks think of it.
Many people are still having debates on Pydantic vs something like marshmallow or now emerging msgspec. And to be possibly faster than msgspec, we have to implement data validation ourselves, which is a project of its own. As big as Robyn itself.
I was thinking of a direction for adding optional dependencies like how we have implemented jinja2 template support and allow the developer to choose their library for data validation. We will create a nice abstraction that would work with all of them(or at least the ones we support out of the box).
This way, we won't be slowing down Robyn and will not be maintaining and implementing something as complicated as Robyn.
What do you think about this?

Something like a @validate decorator?

@kliwongan @AntoineRR , I have been thinking about this PR for a long time, and I have a suggestion. Let me know what you folks think of it.
Many people are still having debates on Pydantic vs something like marshmallow or now emerging msgspec. And to be possibly faster than msgspec, we have to implement data validation ourselves, which is a project of its own. As big as Robyn itself.
I was thinking of a direction for adding optional dependencies like how we have implemented jinja2 template support and allow the developer to choose their library for data validation. We will create a nice abstraction that would work with all of them(or at least the ones we support out of the box).
This way, we won't be slowing down Robyn and will not be maintaining and implementing something as complicated as Robyn.
What do you think about this?

I can try two approaches: first to implement the validate decorator (to some extent, I have some functionality for this implemented already since I pass in a validator function to validate the incoming request body), and two: spend more time trying to integrate pydantic-core with the existing codebase. I can wrap up the first one in a jiffy, and it can be merged into the master branch, while we can leave the second one for now (as I keep working on it).

I can try two approaches: first to implement the validate decorator (to some extent, I have some functionality for this implemented already since I pass in a validator function to validate the incoming request body), and two: spend more time trying to integrate pydantic-core with the existing codebase. I can wrap up the first one in a jiffy, and it can be merged into the master branch, while we can leave the second one for now (as I keep working on it). @sansyrox

@kliwongan , this sounds good. As long as we can create an abstraction that will allow the developer to choose or override, then I think we can proceed with it.

Hey @kliwongan 👋
I hope all's well. 😄  Is there something that we can do to help unblock you?

Hey @kliwongan 👋
I hope all's well. 😄 Is there something that we can do to help you out with the issue?

Hello, I've just been busy recently, and also after pulling recently I've had trouble getting the tests to run successfully; something regarding lifecycle handlers?

@kliwongan , feel free to push your changes on this PR and then maybe we can suggest a few things?

Hey @kliwongan ,
I was taking a deep look in msgspec today and realized that it has some limitations with the typing system.
It does not support mixins, has issues with things like protocols and abcs. It also doesn't support Generics and TypeVars from python types. So, I think we should be using pydantic for data validation.
However, since msgspec is absurdly fast. We should be using it in place of jsonify method.
What do you think?

I think it's a good idea; I can modify the code to use Pydantic similar to how FastAPI does it, although that will also take additional time. Since msgspec is still absurdly fast, we can use it for jsonify, yes. My only problem with that is the over-optimizing aspect; caching one or multiple decoders/serializers since using those objects is faster than calling the functions, and also making this work in a multithreaded context.

Hi, main author of msgspec here! Hope you don't mind me chiming in.

It does not support mixins

This has been resolved (but not yet released).

has issues with things like protocols and abcs

To be clear, the limitation is that you can't create a struct that also subclasses from an abc or Protocol type. However, both of these concepts support "duck-typing" - the defined type doesn't actually need to subclass from the abc/protocol for isinstance/mypy/pyright checking to work properly.
Due to a python limitation we can't support both Protocol and abc types as base classes for msgspec.Struct classes. We'd have to pick one. pydantic chose to support abc base classes, meaning that it can't subclass from protocols. I'd rather not pick favorites, so we support neither.
That said, I've yet to see a strong use case for adding either as a base class for a Struct type, and would be surprised if this limitation is a blocker for any user. Can you comment more on why this matters to you?

It also doesn't support Generics and TypeVars from python types.

We currently don't support generics for Struct types. I expect this limitation to be resolved in the next 2 months. Generics for builtin cpython types are already fully supported. As with the above, I suspect that most users won't ever have a use case for this feature.

So, I think we should be using pydantic for data validation.

Given the current 20-80x decoding speedup when using msgspec for instead of pydantic, I'd hope you'd at least support msgspec for handling JSON request bodies. You might do as starlite is considering, and use msgspec for decoding request bodies if the user declares the body as a msgspec.Struct instance (opting in to using msgspec, rather than by default). Using msgspec for validating other data sources (path/query parameters, headers, cookies, ...) will still currently be faster than pydantic, but the performance difference there is much less (2x speedup at most).
If you're considering using msgspec by default for validation, the restrictions I'd think would be most useful to consider would be our current restrictions on union types. We take a much different stance of unions than pydantic, opting to disallow anything that may be considered ambiguous. This eliminates a wide range of accidental bugs, but also means we can't support any arbitrary Python type union.

That all said, please don't feel pressured to use msgspec here. If you decide you want to support it in any way I'm happy to answer usage questions and resolve issues. But I'm not a robyn user or dev, so I'll leave the question of whether msgspec makes sense for your use case up to you :).

Thank you for the detailed explanation @jcrist 😄  This explanation has given me some great points to think upon. I really appreciate your explanation 😄

@kliwongan , I spent the last week using both pydantic and msgspec. I believe both of them are amazing libraries(with their own nuances) and we should support both of them but should allow the user to decide. How could we do that?
I have an idea that will give users more choice.
class CustType(BaseModel/Struct):
...

@app.get("/")
def route(request:CustType):
...

If the route has a custom type(be it msgspec struct or pydantic basemodel) only then we validate otherwise we don't do any kind of validation.
We will be able to provide enough options and also avoid what I like to call decorator proliferation.

@kliwongan , I spent the last week using both pydantic and msgspec. I believe both of them are amazing libraries(with their own nuances) and we should support both of them but should allow the user to decide. How could we do that?
I have an idea that will give users more choice.
class CustType(BaseModel/Struct):
...

@app.get("/")
def route(request:CustType):
...

If the route has a custom type(be it msgspec struct or pydantic basemodel) only then we validate otherwise we don't do any kind of validation.
We will be able to provide enough options and also avoid what I like to call decorator proliferation.

I believe I can make the necessary modifications. As it is right now, base classes are turned into msgspec structs for every request (which I think was inefficient unless we did some form of caching), but now what I can do is check which custom class a type is, and simply run the schema validation functions for that specific object. More modifications might need to be done to allow for true dependency injection and such, but I'll get started on a rudimentary approach first.

More modifications might need to be done to allow for true dependency injection and such

Yep. I agree. But we can add them later 😄

, but I'll get started on a rudimentary approach first.

Sounds good. Let me know if you need anything from me 😄

Hi @kliwongan, do you still have time to work on this?

Hi @kliwongan, do you still have time to work on this?

Yes, I'm starting to get a bit more free time now, I will restart working on it soon.

Hey @lecronium#4930 :wave:
So, I have the comments ready for the PR. I had this idea that we can allow multiple libraries for data validation. And if the there are any custom types(not the python default types), we can set validate=True by default unless someone overrides it to False.
e.g.
@app.post("/query_validation_complex", validate=True)
async def test_validation_complex(a: int, b: str, c: MsgSpecClass, d:PydanticClass):
    return jsonify({"a": a, "b": b, "c": c})

Also, we should make the data validation imports optional(maybe). Like we do with templating - To keep keep the dependencies as minimal as possible.
Let me know what you think of it 😄


Change uid from 999 to 333333.

This is for security reasons - 999 typically maps to a system user on
the host, which gives attackers privileges in the case of a container
breakout. 333333 is unlikely to map onto anything, but respects the
/etc/user.defs convention of being in the range 10000-60000.

Closes #107.

Merging this. Release to follow later today.


npm run push is not working

NPM run push getting error.

which version of node are you using?
These work for me:
node -v
v10.16.3

npm -v
6.9.0

run ng update @angular/core@8 @angular/cli@8 will fix it.


Auto healing of module helm release

New helm run deduplication was introduced in #67. And now there is no simple way to upgrade module helm release. Addon-operator should check release resources and add ModuleRun task if resources are deleted externally. Also, helm run deduplication should ignore values checksum if resources are deleted by beforeHelm hooks.

Done in #78


Show error message when WootricRemoteRequestTask fails

When executing the WootricRemoteRequestTask, if the request fails, the log only shows response: null

It would be great if the actual error message was show
A simple solution would be to add a Log.d on the catch block
@Override
protected String doInBackground(Void... params) {

    try {
         /*...*/

    } catch (IOException e) {
         Log.d("WOOTRIC_SDK", "request failed because of: " + e.getMessage());
         onError(e);
    }
        return null;
    }

Added this in the latest release https://github.com/Wootric/WootricSDK-Android/releases/tag/2.17.0
Closing this issue. Thanks!


Comments on Blog Posts

I think it would be interesting to have comments on blog posts, as there are times when I wish I could respond to a blog entry, but cannot.

This is a good idea, will try and implement

This isn't something we're going to implement - I'm marking this as out of scope.
Observatory is not a blogging platform - if we want more features from blogs (comments, drafts, published state, RSS) we should start to investigate turn-key platforms that facilitate those use cases - closing.


Amgeo/update luis authoring

Update the LUIS CLI tool to pick up the latest swagger changes

Pull Request Test Coverage Report for Build 2369

53 of 106 (50.0%)  changed or added relevant lines in 18 files are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-1.2%) to 66.823%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

packages/LUIS/src/operations/apps.ts
3
5
60.0%

packages/LUIS/src/operations/settings.ts
8
10
80.0%

packages/LUIS/src/operations/azureAccounts.ts
10
14
71.43%

packages/LUIS/bin/luis.js
3
48
6.25%

Files with Coverage Reduction
New Missed Lines
%

packages/LUIS/src/operations/apps.ts
1
62.0%

packages/LUIS/bin/luis.js
1
33.83%

Totals

Change from base Build 2357:
-1.2%

Covered Lines:
4001

Relevant Lines:
5666

💛  - Coveralls


Handle error in hint UINT256_MUL_DIV_MOD when divides by zero

Handle error in hint UINT256_MUL_DIV_MOD when divides by zero
Description

Handle error in hint UINT256_MUL_DIV_MOD when divides by zero
Also, checks possible zeros before div_mod_floor with BigUints

Checklist

[x] Linked to Github Issue
[ ] Unit tests added
[x] Integration tests added.
[ ] This change requires new documentation.

[ ] Documentation has been added/updated.
[ ] CHANGELOG has been updated.

LGTM, but we need tests for the failing cases as reported by Codecov.

Nice catch @Oppen!!
I remove both of that checks because they were unreacheble
0cfcf7b has a previous check
2766a75 Gets the values from a constant defined in the https://github.com/starkware-libs/cairo-lang/blob/0614f265421735bbb567d9b5897791a51b3705de/src/starkware/cairo/common/math.cairo#L156 library


Indexing Error with codeqai on Conda Environment: Continuous Indexing Without Completion

While using the codeqai tool within a conda environment, I encountered an issue during the indexing process where it continuously attempts to index without completion. This problem occurred when I tried to utilize codeqai's search functionality in my project directory. Specifically, the error IndexError: list index out of range was thrown, indicating an issue with handling the document vector indexing. Below are the detailed steps to reproduce, along with the specific environment setup.
Steps to Reproduce:

Installed codeqai using pip within a conda environment.
Ran codeqai configure and configured the tool with the following settings:

Selected "y" for using local embedding models.
Chose "Instructor-Large" for the local embedding model.
Selected "N" for using local chat models and chose "OpenAI" with "gpt-4" as the remote LLM.

Attempted to start the codeqai search by navigating to my project directory (2-006) that includes .m, .mat, .txt. files. Running codeqai search in the terminal.
Received a message indicating no vector store was found for 2-006 and that initial indexing may take a few minutes. Shortly after, the indexing process started but then failed with an IndexError: list index out of range.

Expected Behavior:
The indexing process should be completed, allowing for subsequent searches within the codebase using codeqai.
Actual Behavior:
The application failed to complete the indexing process due to an IndexError in the vector indexing step, specifically indicating a problem with handling the document vectors.
Environment:

codeqai version: 0.0.14
langchain-community version: 0.0.17
sentence-transformers version: 2.3.1
Python version: 3.11
Conda version: 4.12.0
Operating System: Windows (with Conda environment)

Full Terminal Output and Error
{GenericDirectory>}conda activate condaqai-env

(condaqai-env) {GenericDirectory>}codeqai search
Not a git repository. Exiting.

(condaqai-env) {GenericDirectory>}ls
'ls' is not recognized as an internal or external command,
operable program or batch file.

(condaqai-env) {GenericDirectory>}cd 2-006

(condaqai-env) {GenericDirectory}\2-006>codeqai search
No vector store found for 2-006. Initial indexing may take a few minutes.
⠋ 💾 Indexing vector store...Traceback (most recent call last):
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\Scripts\codeqai.exe\__main__.py", line 7, in <module>
    sys.exit(main())
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\lib\site-packages\codeqai\__main__.py", line 5, in main
    app.run()
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\lib\site-packages\codeqai\app.py", line 146, in run
    vector_store.index_documents(documents)
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\lib\site-packages\codeqai\vector_store.py", line 34, in index_documents
    self.db = FAISS.from_documents(documents, self.embeddings)
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\lib\site-packages\langchain_core\vectorstores.py", line 508, in from_documents
    return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\lib\site-packages\langchain_community\vectorstores\faiss.py", line 960, in from_texts
    return cls.__from(
  File "C:\Users\Edge\anaconda3\envs\condaqai-env\lib\site-packages\langchain_community\vectorstores\faiss.py", line 919, in __from
    index = faiss.IndexFlatL2(len(embeddings[0]))
IndexError: list index out of range
⠴ 💾 Indexing vector store...

Additional Context:
This issue seems to stem from the vector indexing process within the langchain-community package, possibly due to an empty or malformed document set being processed for vectorization. Given the configuration steps and the use of a conda environment, there might be specific dependencies or configurations that contribute to this problem.

Thanks for that detailed report! I think the cause is probably an empty split set for a document, as you also mentioned already.


Component status extension

Type of change:

[x] New feature (non-breaking change which adds functionality)
[x] This change requires a documentation update

Checklist:

[x] Pull request title reflects its content
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Description:

Add Component status types to the Type Manager in context_impl.cpp ("Ok", "Warning", and "Error")
Define ComponentErrorState
Declare and define initComponentErrorStateStatus, setComponentErrorStateStatus, and setComponentErrorStateStatusWithMessage in component_impl.h
Add getStatusMessage in component_status_container.h, and addStatusWithMessage, setStatusWithMessage in component_status_container_private.h , implement all three in component_status_container_impl.h, and make other appropriate changes wherever necessary
Use initComponentErrorStateStatus and setComponentErrorStateStatusWithMessage in all reference Function Blocks
Add appropriate tests and supplement existing ones
Modify Doxygen and Antora docs
Fix typos, and other minor issues, etc. (unrelated)

@NikolaiShipilov Can you check if we need to add anything to the streaming connection statuses here?

I think the additional messages are unnecessary for connection statuses since the status values have a fixed meaning and don’t require further clarification. Therefore, connection statuses can remain as they are for now.


[FEATURE] TestSet Generator

Is your feature request related to a problem? Please describe.
I am encountering difficulties in creating a 'Correct Answer' set. GPT-4 is capable of generating answers that are close to being correct. Therefore, I would like to use GPT-4 to generate a set of correct answers.
Describe the solution you'd like
During an evaluation run, answers are generated.
Users will have the option to mark each answer as correct (O) or incorrect (X), thereby allowing them to curate a set of desired answers.
Although there is an existing feature titled 'Create a test set with API', it does not allow users to directly select correct or incorrect answers. Enhancing this feature to enable users to make such selections would be beneficial, as it allows them to determine which answers should be included in the answer set.

@mmabrouk Thanks for quick response 👍
Recently i handle a case that there are a lot of 'input parameter value' 😇  at least 1000 ~ 1200 probably
if using 'playground' for generating testset, specifically determining true/false for many fields in json, click 'add test set' and 'save' 1000 or more times... that's so painful

@brockWith That's good point. How about this idea:
You create a single evaluation (see video). And then you mark each output as correct or wrong. Then you need to export the data, filter it in excel or in a notebook and upload it again as a test set.
We can make this workflow much faster by modifying the number input for the grade by a slider.
https://github.com/Agenta-AI/agenta/assets/4510758/ed45e823-3882-4a77-8d4c-4bb31a91e6e7
What do you think about this idea?

It's so nice! One more thing i just want to add is just check True/False, not score!

@mmabrouk I tried the solution, And it works nice! Thx!!


Error reading a Svaba vcf

Im trying visualise SVs identified by Svaba using Ggenome. However, I get an error when trying to read the file. The file permissions are 777 and there are lines in the file.
Anyone else had this problem?
> svaba = jJ(system.file('extdata', "I-H-135463-T1-2-D1-1.svaba.somatic.sv.vcf", package = "gGnome"))
Error in read.table(file = file, header = header, sep = sep, quote = quote,  :
no lines available in input
In addition: Warning message:
In file(file, "rt") :
file("") only supports open = "w+" and open = "w+b": using the former

I realized, I was looking at the file in the installation directory. Something like the line below would let you read a file.
svaba=jJ("/Users/yellapa/local/downloads/R-3.5.0/I-H-135463-T1-2-D1-1.svaba.somatic.sv.vcf")


[SPARK-18674][SQL][Follow-Up] improve the error message of using join

What changes were proposed in this pull request?
Added a test case for using joins with nested fields.
How was this patch tested?
N/A

cc @cloud-fan @hvanhovell

Test build #69513 has started for PR 16110 at commit e754ef5.

Test build #69513 has finished for PR 16110 at commit e754ef5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69513/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #69522 has started for PR 16110 at commit e754ef5.

shall we put the test in ResolveNaturalJoinSuite and follow the code style there?

OK if you are improving this, can you fix the grammar? "USING column col1.field1 can not be resolved with the left join side, the left output is: [col1]" technically is grammatically incorrect.

Sure, will do it. Thanks!

Test build #69522 has finished for PR 16110 at commit e754ef5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69522/
Test PASSed.

Just tried MySQL. The message is not good to me:
mysql> select * from t1 inner join t2 using (col2);
ERROR 1054 (42S22): Unknown column 'col2' in 'from clause'

Will try to improve it by:
Using column `col1.field1` cannot be resolved on the left side of the join. The left-side columns: [col1]

That SGTM

Test build #69548 has started for PR 16110 at commit 4d43bca.

LGTM pending tests.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69548/
Test FAILed.

retest this please

Test build #69554 has started for PR 16110 at commit 4d43bca.

Test build #69554 has finished for PR 16110 at commit 4d43bca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69554/
Test PASSed.

thanks, merging to master/2.1!


Add metrics for threads

Add metrics for threads or guide me where I can find them?

I think codahale provides a metric set for threads - com.codahale.metrics.jvm.ThreadStatesGaugeSet.
The dev process afterwards should be similar to #628
Also, not sure how light the APIs are for thread metrics. Might be a good idea to make this one configurable (ie enable/disable)


Keycloak interface visuals

Slightly alter the appearance of the current webinterface for accessing keycloak functions by changing some style parameters inline

LVGTM! :)


Account for removing Scanners that exist but are no longer Scanners

If a scanner no longer acts as a scanner, it still shows in the config flow diagnostics, with requisite skullybones.
Waiting for confirmation if this happens with both esphome and shelly.
Raised via convo at https://community.home-assistant.io/t/bermuda-bluetooth-ble-room-presence-and-tracking-custom-integration/625780/390?u=agittins
Also (only slightly) related to #214

(comments also indicate that reinstalling the integration in this case works, so there must be a visible difference for Bermuda to key from).

ESPHome device that provided digital input/output devices to Home Assistant. Added BLE proxy config, reconsidered and removed the BLE proxy config. Device still performs input/output functions but now appears as dead in Bermuda and can't be removed.

I have pushed changes that should address this in PR#360 and created a release candidate with the fix in v0.7.1rc1 - please let me know if this solves the dead-proxy-hauntings :-)


Bulk Action for Download PDF Reports

Hi everyone,
I implement a bulk action on my Grid.ts with checkboxes like that:

When I click the Scarica button, I'd like to download the reports PDF that I checked, so I do that in my OrderBulkAction.ts:

The problem is when I checked many record: it'll be download only the last and not all of them.
I don't understand why of that.
Could anyone help me, please ?
Thanks all for your time :)
Chiara

Closing due to inactivity.


Update demographics.py and other files

This PR:

updates demographics.py to use the UN population data portal data for South Africa. This only uses the population, fertility, and mortality data through 2021, which solves Issue #4 . But we could use the forecast data as well, as recommended in Issue #8. However, when I experimented with the forecast data, it looked very different from the past data.
adds extrapolation functions to demographics.py for the fertility rate data. The new functions are ?
makes minor updates to the .gitignore
updates the run_og_zaf.py script to include the new South Africa data in demographics.py
updates the calibrate.py file to include the new South Africa data in demographics.py
adds four UN data portal .csv files so the function does not have to download them
updates the macro_params.py file to comment out things we don't use and change things we don't have data for. This file still has a bunch of India calibration data. We will do a lot of work here in the next month to update the South Africa calibration.

Another note is that when I run the run_og_zaf.py script, the model stops after the first (baseline) steady-state computation because the resource constraint (the equation left out of the solution by Walra's Law) is only solved to 0.0006 precision. That is pretty precise, but it should really be arbitrarily small, like 1e-8. I need to go through the theory and why the resource constraint is not solving precisely. My hypothesis is that there is some inconsistency with how the demographics are handled between the fertility rates, mortality rates, and immigration rates versus the steady-state population distribution. I'll do more work on this. I will leave this PR as a draft (i.e., cannot be merged) until I fix this problem.
...
GE loop errors =  [-8.03523914072457e-15, -8.042178034628478e-15, 6.772360450213455e-14, 0.0, 1.3066214776813467e-12, -4.246603069191224e-15, 4.709427292581836e-13, -4.597017211338539e-15, -2.6229018956769323e-15, 4.626715677247262e-13, -3.3757718842508666e-15, -6.262351748276274e-16, 1.175934349895158e-13, -7.314981953499e-14]
Iteration: 01  Distance:  1.5534757137146956e-11
SS debt =  0.6699739181340526 0.006316971073325546
IO:  (1, 1) , C:  (1,)
Foreign debt holdings =  0.1741932187148537
Foreign capital holdings =  0.1631655427856682
resource constraint:  [0.00062941]
Resource Constraint Difference: [0.00062941]
Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 131, in <module>
    main()
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 58, in main
    runner(p, time_path=True, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/execute.py", line 45, in runner
    ss_outputs = SS.run_SS(p, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/SS.py", line 1159, in run_SS
    output = SS_solver(
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/SS.py", line 902, in SS_solver
    raise RuntimeError(err)
RuntimeError: Steady state aggregate resource constraint not satisfied

closes #4
cc: @SeaCelo

@rickecon To help understand the South African population profiles, here are the detailed profiles prepared by UN Population: https://population.un.org/wpp/Graphs/DemographicProfiles/Line/710

@rickecon To help understand the South African population profiles, here are the detailed profiles prepared by UN Population: https://population.un.org/wpp/Graphs/DemographicProfiles/Line/710

@rickecon I looked at the population trends for South Africa and the data is correct. South Africa experienced a big change in fertility and mortality due to a few causes. My quick search of the literature give some clues (preferences, risk aversion by women, HIV epidemic and mortality, etc). When we analyze the results, we will need to keep these factors in mind as the population structure has been affected by the large changes in the 2000s.

@SeaCelo. When I run the example run script run_og_zaf.py, it solves the baseline steady-state equilibrium. However, the transition path equilibrium is converging, but finishes with a resource constraint error. I suspect this is another issue of not getting one of the demographics.py time path objects exactly right.
Maximum debt ratio:  1.000000025085511
w diff:  5.956415498609147e-07 -6.274392516658622e-08
r diff:  7.951739967870797e-09 -8.710239011999743e-08
r_p diff:  7.951739967870797e-09 -8.710239011999743e-08
p_m diff:  0.0 0.0
BQ diff:  1.6103533946251747e-07 -6.463835558120912e-08
TR diff:  1.5414218641762734e-07 -2.5376482863659344e-07
Iteration: 22
	Distance: 7.272261087114081e-06
Max absolute value resource constraint error: 0.007107331731424233
Checking time path for violations of constraints.
Max Euler error, savings:  7.559286530067766e-12
Max Euler error labor supply:  2.0685675394815917e-12
Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 131, in <module>
    main()
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 58, in main
    runner(p, time_path=True, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/execute.py", line 68, in runner
    tpi_output = TPI.run_TPI(p, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/TPI.py", line 1293, in run_TPI
    raise RuntimeError(
RuntimeError: Transition path equlibrium not found (RC_error)

I will look into this more. For the time being, I am moving this PR back to draft status.

@SeaCelo. I put your two comments (first and second) in Issue #8 so we can work on those in the future. That is something that we want to treat outside of this PR.

@SeaCelo. I made the changes that remove the min_yr and max_yr arguments from the functions in the demographics.py module. These are now hard coded to be min_yr=1 and max_yr=100 to correspond with the annual age data in the UN population data. I left the totpers arguments. This allows the model to adjust from annual model years totpers=100 to model years that are fewer totpers<100. This is valuable in testing situations and other situations where you want to run a smaller and quicker version of the model.
The steady-state still solves, but I am getting the same resource constraint error in the baseline time path after it converges.
GE loop errors =  [-1.1796119636642288e-16, -1.249000902703301e-16, 1.5543122344752192e-15, 0.0, 1.341149413747189e-13, 3.8510861166685117e-16, -1.0408340855860843e-15, -1.2385925618474403e-15, -6.782768791069316e-16, -7.73686670285656e-16, -1.0130785099704553e-15, -4.3194614551822497e-16, 1.2073675392798577e-14, -8.590350653037149e-15]
Iteration: 01  Distance:  1.2025259571510712e-13
SS debt =  0.6693507845437849 0.006379343173246392
IO:  (1, 1) , C:  (1,)
Foreign debt holdings =  0.17403120398138408
Foreign capital holdings =  0.17525777572767656
resource constraint:  [-2.39946951e-14]
Checking constraints on capital, labor, and consumption.
	There were no violations of the constraints on labor  supply.
	There were no violations of the constraints on  consumption.
Maximum error in labor FOC =  1.6564527527407336e-13
Maximum error in savings FOC =  7.949196856316121e-14
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-ZAF/examples/OG-ZAF-Example/OUTPUT_BASELINE/SS/SS_vars.pkl

The transition path error output after running the baseline is the following:
Maximum debt ratio:  1.000000023879215
w diff:  5.778020204516565e-07 -5.8068007069422833e-08
r diff:  7.361659268290133e-09 -8.447208207224e-08
r_p diff:  7.361659268290133e-09 -8.447208207224e-08
p_m diff:  0.0 0.0
BQ diff:  1.5559656061162053e-07 -4.9671494802699945e-08
TR diff:  1.5398244257447402e-07 -2.456320713772886e-07
Iteration: 22
	Distance: 7.07245418554592e-06
Max absolute value resource constraint error: 0.007064895621105965
Checking time path for violations of constraints.
Max Euler error, savings:  7.44737604918555e-12
Max Euler error labor supply:  1.9104717807749694e-12
Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 131, in <module>
    main()
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 58, in main
    runner(p, time_path=True, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/execute.py", line 68, in runner
    tpi_output = TPI.run_TPI(p, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/TPI.py", line 1293, in run_TPI
    raise RuntimeError(
RuntimeError: Transition path equlibrium not found (RC_error)

I'll take this PR out of draft status after I fix this error.

@SeaCelo. I fixed one error in demographics.py but the resource constraint in the transition path equilibrium is still not solving. The current error I fixed, which I thought would solve the whole problem, was a typo in the get_imm_resid() function in demographics.py. I had to fix the denominator of the lines (lines 727-732) that compute the immigration rate as a residual. It used to be pop_2020_EpS[1:], but it needed to be pop_2020_EpS[:-1]. Dumb typo.
imm_rates_s_2020 = (
    pop_2021_EpS[1:] - (1 - mort_rates[:-1]) * pop_2020_EpS[:-1]
) / pop_2020_EpS[:-1]
imm_rates_s_2019 = (
    pop_2020_EpS[1:] - (1 - mort_rates[:-1]) * pop_2019_EpS[:-1]
) / pop_2019_EpS[:-1]

I did have to change the starting interest rate to initial_guess_r_SS = 0.055 in ogzaf_default_parameters.json. As with previous versions of this PR, the steady-state solves with all equations, including the resource constraint, solving to arbitrary accuracy.
GE loop errors =  [-4.506811590587745e-14, -4.508893258758917e-14, 3.2440716779547074e-13, 0.0, -6.094347249074872e-12, 1.477290512141849e-14, 4.3770542745846797e-14, -9.944371714976086e-13, 5.0964441000722616e-14, 6.760217385881617e-14, 9.533346334578141e-14, 2.1935578353726726e-14, -5.484918075282508e-13, -1.179903397208193e-12]
Iteration: 01  Distance:  2.529198625454508e-11
SS debt =  0.6711779592951316 0.006393461432212574
IO:  (1, 1) , C:  (1,)
Foreign debt holdings =  0.17450626941673422
Foreign capital holdings =  0.20186988283166193
resource constraint:  [-6.64399091e-14]
Checking constraints on capital, labor, and consumption.
	There were no violations of the constraints on labor  supply.
	There were no violations of the constraints on  consumption.
Maximum error in labor FOC =  1.5276668818842154e-13
Maximum error in savings FOC =  7.238654120556021e-14
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-ZAF/examples/OG-ZAF-Example/OUTPUT_BASELINE/SS/SS_vars.pkl

But the transition path equilibrium still does not solve the resource constraint to arbitrary precision. The solution method converges, but the resource constraint error in the converged problem is 0.005, which is too big.
Maximum debt ratio:  1.000000021248
w diff:  5.317299547336063e-07 -6.817061692920845e-08
r diff:  8.724744218757685e-09 -7.817221110012484e-08
r_p diff:  8.724744218757685e-09 -7.817221110012484e-08
p_m diff:  0.0 0.0
BQ diff:  1.409597274459884e-07 -5.723262022291897e-08
TR diff:  1.733481109628121e-07 -2.2378731083516623e-07
Iteration: 22
	Distance: 6.44583182750035e-06
Max absolute value resource constraint error: 0.0054670522852872366
Checking time path for violations of constraints.
Max Euler error, savings:  7.541522961673763e-12
Max Euler error labor supply:  1.9180212973424204e-12
Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 131, in <module>
    main()
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 58, in main
    runner(p, time_path=True, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/execute.py", line 68, in runner
    tpi_output = TPI.run_TPI(p, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/TPI.py", line 1293, in run_TPI
    raise RuntimeError(
RuntimeError: Transition path equlibrium not found (RC_error)

I will keep working on this.

@SeaCelo. I fixed an error in the demographics.py module in the get_pop_objs() function in which the steady-state population growth rate g_n_ss was based on the original population law of motion with the original immigration rates. This change made g_n_ss based correctly on the adjusted immigration rates. The steady-state solves.
GE loop errors =  [1.3584827707191494e-12, 1.3584897096130533e-12, -1.3342438265340206e-11, 0.0, -6.299083477045997e-11, -4.4683146072088675e-12, 3.1430170965851545e-12, 3.014186122918261e-12, 2.113132585579436e-12, -4.1530529015787465e-12, -1.0508271336417963e-11, 1.2577005409353248e-12, -5.66917346400686e-12, 6.457390178127298e-12]
Iteration: 01  Distance:  2.8559618457782164e-10
SS debt =  0.6711779592859342 0.006393461432124962
IO:  (1, 1) , C:  (1,)
Foreign debt holdings =  0.17450626941434288
Foreign capital holdings =  0.20186988275228981
resource constraint:  [-4.47763354e-13]
Checking constraints on capital, labor, and consumption.
	There were no violations of the constraints on labor  supply.
	There were no violations of the constraints on  consumption.
Maximum error in labor FOC =  1.5631940186722204e-13
Maximum error in savings FOC =  6.572520305780927e-14
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-ZAF/examples/OG-ZAF-Example/OUTPUT_BASELINE/SS/SS_vars.pkl

But the transition path still breaks after converging because the resource constraint is not solved to arbitrary precision.
Maximum debt ratio:  1.0000000212422069
w diff:  5.317299542895171e-07 -6.817061737329766e-08
r diff:  8.724744232635473e-09 -7.817221105155259e-08
r_p diff:  8.724744239574367e-09 -7.817221105155259e-08
p_m diff:  0.0 0.0
BQ diff:  1.4095972736272167e-07 -5.723262049006639e-08
TR diff:  1.7334811101832326e-07 -2.2378731083516623e-07
Iteration: 22
	Distance: 6.445831828830127e-06
Max absolute value resource constraint error: 0.005467052285404431
Checking time path for violations of constraints.
Max Euler error, savings:  7.496225862269057e-12
Max Euler error labor supply:  1.9104717807749694e-12
Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 131, in <module>
    main()
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 58, in main
    runner(p, time_path=True, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/execute.py", line 68, in runner
    tpi_output = TPI.run_TPI(p, client=client)
  File "/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/TPI.py", line 1293, in run_TPI
    raise RuntimeError(
RuntimeError: Transition path equlibrium not found (RC_error)

@SeaCelo. I fixed the issue with the transition path equilibrium not satisfying the resource constraint in the first period. For the transition path, we have to compute the population distribution in the period before the initial period. Things like the total capital stock and total bequests in the first period depend on the population distribution in the period before that. So demographics.py produces an omega object that is the population distribution from period t=1 to period t=T+S. The module also produces omega_S_preTP, which is the population distribution in period t=0. My error was that I was taking that data from the actual data. However, the data are not consistent with the population demographics law of motion we specified in the theory, at least not with constant fertility rates and immigration rates. For this reason, I had to make omega_S_preTP consistent with the omega matrix (in terms of fertility rates, mortality rates, and immigration rates).
The baseline steady-state solution output is the following:
GE loop errors =  [-8.227446501862801e-14, -4.869021852371702e-14, -1.3922196728799463e-13, 0.0, 2.1715962361668062e-13, -1.293652684974944e-13, -5.678790770957676e-14, -5.6534638082084143e-14, -6.17214612752548e-15, -3.7567171595753734e-14, -5.0067588963642606e-14, -1.5612511283791264e-14, 1.9546864127306662e-14, 3.039235529911366e-15]
Iteration: 01  Distance:  4.784789093966305e-12
SS debt =  0.6711715191110103 0.0063955512984892274
IO:  (1, 1) , C:  (1,)
Foreign debt holdings =  0.17450459496886267
Foreign capital holdings =  0.20239138201602544
resource constraint:  [-4.2802567e-14]
Checking constraints on capital, labor, and consumption.
	There were no violations of the constraints on labor  supply.
	There were no violations of the constraints on  consumption.
Maximum error in labor FOC =  1.6076029396572267e-13
Maximum error in savings FOC =  7.638334409421077e-14
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-ZAF/examples/OG-ZAF-Example/OUTPUT_BASELINE/SS/SS_vars.pkl

The baseline transition path solution output is the following;
Maximum debt ratio:  1.0000000212110733
w diff:  5.364696991083662e-07 -7.536044477340909e-08
r diff:  9.654272145609433e-09 -7.871194662723013e-08
r_p diff:  9.654272145609433e-09 -7.871194662723013e-08
p_m diff:  0.0 0.0
BQ diff:  1.4075238878932805e-07 -5.199504206701566e-08
TR diff:  1.678670771831925e-07 -2.235329153860155e-07
Iteration: 22
	Distance: 6.4398646151337245e-06
Max absolute value resource constraint error: 1.0536127898710945e-05
Checking time path for violations of constraints.
Max Euler error, savings:  7.473133223356854e-12
Max Euler error labor supply:  1.9113599591946695e-12
Time path iteration complete.
It took 1815.7290947437286 seconds to get that part done.
run time =  1815.729147195816

The reform steady-state solution output is the following:
GE loop errors =  [7.81388842518993e-14, 8.174017018802715e-14, -4.771738559838923e-13, 0.0, 3.5582647939236267e-13, -2.2105581254372453e-14, -1.0152295670806666e-13, -3.642919299551295e-16, -2.207435623180487e-14, 4.0554365421385796e-14, -2.8102520310824275e-16, 2.6020852139652106e-18, 3.202299536653186e-14]
Iteration: 01  Distance:  3.1749021791539607e-12
SS debt =  1.2004356041121698 0.015262156036278306
IO:  (1, 1) , C:  (1,)
Foreign debt holdings =  0.48017424164486794
Foreign capital holdings =  0.08048294759663217
resource constraint:  [1.7069679e-14]
Steady state government spending is negative to satisfy budget
Checking constraints on capital, labor, and consumption.
	There were no violations of the constraints on labor  supply.
	There were no violations of the constraints on  consumption.
Maximum error in labor FOC =  1.3011813848606835e-13
Maximum error in savings FOC =  7.149836278586008e-14
/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/SS.py:1319: UserWarning: Warning: The combination of the tax policy you specified and your target debt-to-GDP ratio results in an infeasible amount of government spending in order to close the budget (i.e., G < 0)
  warnings.warn(
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-ZAF/examples/OG-ZAF-Example/OUTPUT_REFORM/SS/SS_vars.pkl

And the reform transition path solution output is the following:
Maximum debt ratio:  2.000104723711181
w diff:  2.5915477188664937e-06 -1.1417283154457891e-06
r diff:  1.3815679607981357e-07 -4.063585497746103e-07
r_p diff:  8.413844220650502e-08 -2.596077518443596e-07
p_m diff:  0.0 0.0
BQ diff:  1.1468624764796043e-07 -5.7201863237418826e-08
TR diff:  3.424993502282314e-07 -2.4845994033712016e-07
Iteration: 21
	Distance: 9.290020424167063e-06
Max absolute value resource constraint error: 6.777817169005551e-07
Checking time path for violations of constraints.
Max Euler error, savings:  9.631240249774464e-12
Max Euler error labor supply:  1.979305608301729e-12
Time path iteration complete.
It took 1644.864609003067 seconds to get that part done.
run time =  1644.8646562099457
Percentage changes in aggregates: Year                    Variable  2021  2022  ...  2030  2021-2030     SS
0                    GDP ($Y_t$)  3.71  2.09  ... -2.05      -0.58 -10.57
1            Consumption ($C_t$) 16.87  7.72  ... 12.41      12.78   6.61
2          Capital Stock ($K_t$)  1.97 -0.80  ... -5.58      -4.45 -26.04
3                  Labor ($L_t$)  0.57 -0.38  ... -4.01      -2.39  -4.80
4     Real interest rate ($r_t$)  9.35 12.07  ... 14.06      14.70  52.48
5                      Wage rate  1.05  0.41  ... -0.00      -0.17  -7.95

[6 rows x 13 columns]

@rickecon I can't checkout this  branch.
git checkout -b rickecon-demographics
git pull https://github.com/rickecon/OG-ZAF.git demog
Gives me an error:
remote: Enumerating objects: 29, done.
remote: Counting objects: 100% (14/14), done.
remote: Total 29 (delta 14), reused 14 (delta 14), pack-reused 15
Unpacking objects: 100% (29/29), 14.20 KiB | 632.00 KiB/s, done.
From https://github.com/rickecon/OG-ZAF
 * branch            demog      -> FETCH_HEAD
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

I was able to successful run the model.
FYI, these values for r_guess have a SS solution: 0.02, 0.035, 0.04, 0.06, 0.065, 0.07, 0.075, 0.08, 0.085
The default value is currently 0.045. I suggest using instead 0.04.

@sarvonz Add your review when you have a chance please

@SeaCelo and @sarvonz. This PR should have been ready to merge, but it is not. My reform steady state did not solve. This is a very rare occurrence, but I should not be surprised given the test that we ran. The steady state solution can be pretty sensitive to starting values. As such, I need to take the robustness algorithm that I built into OG-Core in PR #836 and add it to the reform steady state. My failed results from this run are below.
Baseline steady state equilibrium computation output
GE loop errors =  [-2.508410146262463e-14, -2.5070223674816816e-14, 4.933831121434196e-13, 0.0, 3.461231301571388e-12, -1.1719791803699309e-14, 4.936606678995759e-13, -1.3822276656583199e-14, -7.806255641895632e-15, 2.426600587135397e-13, 5.061367991388011e-13, -5.950101522600448e-16, 1.3844481117075702e-13, -1.092459456231154e-13]
Iteration: 01  Distance:  6.245774321886237e-12
SS debt =  1.061290349047439 0.002900917870000058
IO:  (1, 1) , C:  (1,)
Foreign debt holdings =  0.2494251742773182
Foreign capital holdings =  0.04224643260693746
resource constraint:  [2.01106493e-14]
Checking constraints on capital, labor, and consumption.
	There were no violations of the constraints on labor  supply.
	There were no violations of the constraints on  consumption.
Maximum error in labor FOC =  7.904787935331115e-14
Maximum error in savings FOC =  4.929390229335695e-14
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-ZAF/examples/OG-ZAF-Example/OUTPUT_BASELINE/SS/SS_vars.pkl

Baseline transition path equilibrium computation output (20 min, 12 sec)
Maximum debt ratio:  4.903478755422615
w diff:  1.5803858472995813e-06 -5.354729988304996e-07
r diff:  3.21790575019687e-08 -8.971252557499199e-08
r_p diff:  3.21790575019687e-08 -8.971252557499199e-08
p_m diff:  0.0 0.0
BQ diff:  4.3528094994388367e-07 -3.640637166735239e-08
TR diff:  1.0657384290346794e-07 -2.65468051423301e-07
Iteration: 23
	Distance: 8.970078192451272e-06
Max absolute value resource constraint error: 2.1151583345013422e-05
Checking time path for violations of constraints.
Max Euler error, savings:  8.604783552357276e-12
Max Euler error labor supply:  2.9523050670832163e-12
Time path iteration complete.
It took 1211.9648201465607 seconds to get that part done.
run time =  1211.965884923935

Reform steady state equilibrium computation output
GE loop errors =  [0.036488251251174025, 0.049594113119542794, -0.5320786226318877, 0.01524214322893791, -0.17955166455931948, 0.005327985899518087, 0.004273354336397325, -0.0005952612832190347, 0.00723037616235019, 0.024715917993280444, -0.0006519048822944025, 0.012633697617219582, -0.016159649810338753]
Traceback (most recent call last):
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 122, in <module>
    main()
  File "/Users/richardevans/Docs/Economics/OSE/OG-ZAF/./examples/run_og_zaf.py", line 82, in main
    runner(p2, time_path=True, client=client)
  File "/Users/richardevans/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/execute.py", line 45, in runner
    ss_outputs = SS.run_SS(p, client=client)
  File "/Users/richardevans/opt/anaconda3/envs/ogzaf-dev/lib/python3.9/site-packages/ogcore/SS.py", line 1358, in run_SS
    raise RuntimeError("Steady state equilibrium not found")
RuntimeError: Steady state equilibrium not found

OK. @jdebacker's suggestion fixed it. I updated run_og_zaf.py. The reform is now running very well. This PR is ready to merge. Below are my reform steady-state and transition path output that correspond to the baseline run in my previous comment.
Reform steady state equilibrium computation output
GE loop errors =  [6.938893903907228e-18, 6.938893903907228e-18, 0.0, 0.0, -2.220446049250313e-16, 3.469446951953614e-17, -1.4224732503009818e-15, 4.163336342344337e-17, 3.469446951953614e-17, -1.429412144204889e-15, 2.0816681711721685e-17, 3.660266534311063e-16, -6.938893903907228e-18]
Iteration: 01  Distance:  2.274998632632352e-14
SS debt =  1.269064660476182 0.0034688455944833566
IO:  (1, 1) , C:  (1,)
Foreign debt holdings =  0.298256433211293
Foreign capital holdings =  0.1677431527583257
resource constraint:  [-3.33066907e-15]
Checking constraints on capital, labor, and consumption.
	There were no violations of the constraints on labor  supply.
	There were no violations of the constraints on  consumption.
Maximum error in labor FOC =  7.949196856316121e-14
Maximum error in savings FOC =  4.796163466380676e-14
JUST SAVED SS output to  /Users/richardevans/Docs/Economics/OSE/OG-ZAF/examples/OG-ZAF-Example/OUTPUT_REFORM/SS/SS_vars.pkl

Reform transition path equilibrium computation output (21 min, 25 sec)
Maximum debt ratio:  4.87467226523405
w diff:  1.5875761869565963e-06 -3.954484590895646e-07
r diff:  2.3654958110352897e-08 -9.037962634544927e-08
r_p diff:  2.3654958110352897e-08 -9.037962634544927e-08
p_m diff:  0.0 0.0
BQ diff:  4.332964183076382e-07 -3.657225344094739e-08
TR diff:  1.0994371051759044e-07 -2.6346526710296914e-07
Iteration: 23
	Distance: 8.94115225742335e-06
Max absolute value resource constraint error: 2.1143231016623742e-05
Checking time path for violations of constraints.
Max Euler error, savings:  8.690714814463263e-12
Max Euler error labor supply:  2.97939450888407e-12
Time path iteration complete.
It took 1199.5632781982422 seconds to get that part done.
run time =  1199.5642490386963
Percentage changes in aggregates: Year                    Variable  2021  2022  2023  2024  ...  2028  2029  2030  2021-2030    SS
0                    GDP ($Y_t$) -0.20 -0.20 -0.20 -0.20  ... -0.20 -0.20 -0.19      -0.20 -0.35
1            Consumption ($C_t$) -0.08 -0.13 -0.13 -0.11  ... -0.11 -0.11 -0.11      -0.11 -0.17
2          Capital Stock ($K_t$) -0.42 -0.42 -0.43 -0.43  ... -0.42 -0.42 -0.42      -0.42 -0.76
3                  Labor ($L_t$) -0.02 -0.02 -0.02 -0.02  ... -0.02 -0.02 -0.02      -0.02 -0.03
4     Real interest rate ($r_t$) -0.06 -0.06 -0.06 -0.06  ... -0.07 -0.07 -0.07      -0.06  0.34
5                      Wage rate -0.18 -0.18 -0.18 -0.18  ... -0.18 -0.18 -0.18      -0.18 -0.32

[6 rows x 13 columns]

cc: @SeaCelo @sarvonz

@SeaCelo. Make sure you updated your ogzaf-dev conda environment. Your old environment might have the old version of OG-Core installed from pip.

conda remove --name ogzaf-dev --all
While in the OG-ZAF directory, re-create the conda environment: conda env create -f environment.yml
Activate the ogzaf-dev environment: conda activate ogzaf-dev
Install the ogzaf package in this environment: pip install -e .

Now run run_og_zaf.py from your version of my demog branch. Make sure that your version of my demog branch was pulled yesterday afternoon after my last commit.
If this doesn't work, let's set up a video call for sometime today for me to look at what you have. You and I have the same hardware setup. This should be working.

Re-installing OG-CORE solved the issue and I have identical results. Merging this PR


cross account access with accesss and secret key

Hi,
I am trying to check if this exporter can be used for cross account with access and secret key. it will be great if we are having example for same.

Hey,
i think it should work with roleArn setting. See README.md where to set it for static / discovery jobs.
But not 100% sure how your setup looks like. Maybe you can explain it a little bit more.
Greetings, Thomas

@polganesh ping :)

i am trying to use  this as a helm chart but not sure how to attach role to k8s cluster.

Closing due to very old issue. Please tag me if this is still a problem. Then i reopen it.


feat: add border style support to layout components

Description

Adds border-style values to tokens
Adds support for border style to layout components

Tasks
KNO-6154

#111  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @kylemcd and the rest of your teammates on  Graphite

Merge activity

Jun 6, 4:46 PM CDT: @kylemcd started a stack merge that includes this pull request via Graphite.


Wizard should not reuse internal account

As described by @konradkonrad in #123 the user only has the chance to control the internal account and fund it at the first run of the wizard.
After that the user is on its own to figure out how to use raiden again and get his funds back. As @Dominik1999 mentioned the wizard is currently designed to only set up raiden initially and is not designed to be reused.
This leads to a potential loss of funds.
Furthermore the internal account should not be used outside of Raiden. If we want to exchange for new token, as well as, top up the UDC which should be somehow possible, then the current wizard cannot be reused as the internal account does all the work.
Handle internal account funds by external Metamask account
One potential solution would be, that the whole process of interacting with the blockchain to set up the internal account is controled by an metamask account. this process follows like funding the internal account with ETH. Instead of the internal account exchanging for RDN it could also be done by the metamask account and then directly send to the UDC on behalf of the internal account.
This functionality could also be placed in the WebUI of Raiden itself.
The downside is more clicks for the user in metamask to authorize transactions.

Thank you Fred for your input here. So from my understanding now

Wizard can be reused a second time to start Raiden
Wizard cannot be reused a second time to do any BC transaction, e.g. refunding the account

Can you elaborate on how this will lead to a loss of funds?

Your understanding is correct. We have two things here. Can we reuse for funding? Then we need to do it by an metamask account. Can we reuse to start raiden? Certainly. This is how the wizard works now already.
What we cannot do is, redeem any funds nor does the user currently know where the account and the passphrase is stored. This leads actually to the fact that the user can close all channels by the WebUI but the user actually cannot send it to its metamask account or somewhere else. Thus, if the user does not know where the keystore or the passphrase is stored, can lead to the loss of funds.
Another point, which is coming to my mind. If the funds are too low, the user indeed will be redirected to top up the funds by doing BC transactions outside of raiden. If this confuses raiden in any way then it would temendously increase the difficulty of getting the funds back. (Probably only by closing channels manually via etherscan). I hope this helped

Wouldn’t it be easier (at least for now) to go with the second proposal from #123 and make the wizard not immediately launch Raiden on subsequent starts?

What do you mean by second proposal? Do you mean the second bullet point? I think #123 are observations how the wizard currently works. The user should at least be able to find its keystore IMO.
He could then transfer his tokens to other accounts

ok, good that we are at a point where we can discuss those things and act iteratively upon. Here are still several problems together from what I see.
Let's try to break the open issues down in as many small changes as possible

[ ] the user should be able to restore / transfer his wallet in a safe way https://github.com/raiden-network/raiden-wizard/issues/132
[ ] the user should be able to click the Wizard a second time and Raiden should start with the same account as in the first time - works atm
[ ] the user should be able to top up his funding and deposits (that should imho happen in the Raiden WebUI)

Based on your last point: We also had the discussion if the user is supposed to run raiden always with the wizard (there is a difference between he can run the wizard a second time (nothing breaks) and he should ).
One point I just realized. If everything should happen in the WebUI as you mentioned, we are gonna run into the nonce problem again. We can only  top up, deposit, etc... while raiden is not running. If we want to do everything by the WebUI then there must be a way that either raiden is doing the transactions or the WebUI is running while raiden isnt.

The WebUI only works via API calls to Raiden itself. So the actual swap and call would happen in Raiden indeed

this can be close due to the note by @ulope that the internal account can be used as long as raiden is not running (which should be implicit if you want to start raiden through the use of the wizard)


lab09b.png

Image was not translated

Hello contributor,
Thank you for your contribution! We are processing your suggestions and we will update you shortly.
Kind regards,
Microsoft Worldwide Learning Team

Hello contributor,
Thank you for your contribution! We are processing your suggestions and we will update you shortly.
Kind regards,
Microsoft Worldwide Learning Team

Hello contributor,
Thank you for your contribution! We are processing your suggestions and we will update you shortly.
Kind regards,
Microsoft Worldwide Learning Team

Hello contributor,
Thank you for your contribution! We are processing your suggestions and we will update you shortly.
Kind regards,
Microsoft Worldwide Learning Team

Re-set locfile in Prehandoff status, waiting for checkedIn. ICMS QUERY

Was set to PHO in C&E - now in CAICritical  - One still inloc

This image will not be translated.


fix: bb unnecessary env var

Closes https://github.com/AztecProtocol/barretenberg/issues/822

Benchmark results
No metrics with a significant change found.

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit 537630ff and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
128 txs

l1_rollup_calldata_size_in_bytes
45,444
179,588
716,132

l1_rollup_calldata_gas
223,008
868,196
3,449,924

l1_rollup_execution_gas
867,117
3,746,249
23,306,689

l2_block_processing_time_in_ms
1,323 (-2%)
5,108 (-1%)
21,098 (-3%)

note_successful_decrypting_time_in_ms
355 (-6%)
1,138 (-2%)
4,116 (-6%)

note_trial_decrypting_time_in_ms
65.2 (-29%)
51.5 (+35%)
136 (-6%)

l2_block_building_time_in_ms
14,913 (-1%)
59,372 (-1%)
238,255 (-1%)

l2_block_rollup_simulation_time_in_ms
10,603 (-1%)
42,265 (-1%)
170,088 (-1%)

l2_block_public_tx_process_time_in_ms
4,276 (-2%)
17,031 (-1%)
67,901 (-1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
15,074 (-2%)
28,841 (-1%)

note_history_successful_decrypting_time_in_ms
2,628 (-7%)
5,199 (-6%)

note_history_trial_decrypting_time_in_ms
140 (-19%)
172 (+17%)

node_database_size_in_bytes
3,622,108
3,725,839

pxe_database_size_in_bytes
29,923
59,478

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
210
44,709
25,561

private-kernel-ordering
136 (-1%)
30,953
14,809

base-rollup
1,074
128,374
881

root-rollup
82.1 (-1%)
4,088
889

private-kernel-inner
286 (-1%)
71,236
25,561

public-kernel-private-input
202
31,891
25,561

public-kernel-non-first-iteration
199 (-1%)
31,933
25,561

merge-rollup
9.65 (-2%)
2,608
881

Miscellaneous
Transaction sizes based on how many contracts are deployed in the tx.

Metric
0 deployed contracts
1 deployed contracts

tx_size_in_bytes
15,487
31,879


timeline plugin for archive search

just wondering if the timeline plugin could be used for full-archive search (i.e. through search end-point instead of timeline end-point)? If so, I'd imagine the plugin would enable us to fetch the timelines from a specific period of time for a list of users stored in txt file (similar to how the timeline plugin works for now). Thanks a lot!!!

I've just uploaded twarc-timelines v0.0.4 which adds a --use-search option. It will use the search api to fetch as many of the users's tweets as possible from the search/all API endpoint, and so isn't restricted to the last 3,200 from the timeline API. I didn't add options for specifying the time range, but please reopen or add a new issue if that's a problem.


Dashboard improvements

This PR aims at improving the CloudStack dashboard and introduces the following:

Admin dashboard: six cards that are responsive to screen sizes and show zone specific compute, storage and network allocation, as well as instance/hosts stats, alerts and events
Account/project dashboard: six cards that are responsive to screen sizes and show account or project specific resource lists/counts, and limits shown in three cards as (a) compute (with running stopped instances), (b) storage and (c) network allocation, an admin-defined links/docs card (via config.json) and events cards
A global create button on the top global header/user-menu to allow for quick actions such as to deploy a VM, CKS cluster and create a volume (more actions can be added as desired via code changes)

Screenshots:
Quick create button:

Admin dashboard:

Project view:

Responsive (smaller screen) layout:

Domain admin dashboard:

Domain admin project dashboard:

Translation screenshot:

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Testing UI build
@blueorangutan ui

@blueorangutan ui
@blueorangutan package

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

UI build failed: :heavy_multiplication_x:
(SL-JID-173)

Packaging result [SF]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 7012

@blueorangutan ui
@blueorangutan package

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-175)

Packaging result [SF]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 7023

@blueorangutan ui

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-176)

@blueorangutan ui
@blueorangutan package

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-177)

@blueorangutan package

@blueorangutan ui

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-178)

@blueorangutan ui

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-179)

@blueorangutan ui
@blueorangutan package

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-180)

Packaging result [SF]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 7051

@blueorangutan ui
@blueorangutan package

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@blueorangutan package

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 7053

@blueorangutan package
@blueorangutan ui

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with@blueorangutan ui SystemVM template(s). I'll keep you posted as I make progress.

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-185)

@blueorangutan package
@blueorangutan ui

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-186)

@blueorangutan package
@blueorangutan ui

@blueorangutan package
@blueorangutan ui

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with@blueorangutan ui SystemVM template(s). I'll keep you posted as I make progress.

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

Packaging result [SF]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 7074

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-188)

Packaging result [SF]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 7080

@blueorangutan ui

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@blueorangutan package
@blueorangutan ui

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-190)

@shwstppr that's correct, the create button is show only on desktop as the screen isn't that wide on phone and tablets to show all of project and create button.

@blueorangutan package
@blueorangutan ui

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with@blueorangutan ui SystemVM template(s). I'll keep you posted as I make progress.

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-191)

@blueorangutan package
@blueorangutan ui

@rohityadavcloud a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with@blueorangutan ui SystemVM template(s). I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7956 (QA-JID-192)

Packaging result [SF]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 7094

@rajujith a [SF] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 7240