Hash maps

Closed lemmas about findD and expand_go for HashSets. One axiom was incorrect and was corrected.

Very nice, thank you!
Which one was the faulty axiom? The one which is now theorem HashMap.findD_insert (hm : HashMap A B) (a a' : A) (h: ¬ hm.contains a = true): ∀ b, (hm.insert a b).findD a' b = hm.findD a' b?

Very nice, thank you! Which one was the faulty axiom? The one which is now theorem HashMap.findD_insert (hm : HashMap A B) (a a' : A) (h: ¬ hm.contains a = true): ∀ b, (hm.insert a b).findD a' b = hm.findD a' b?

Yes. hm = { 1:3}.
(hm.insert 1 4).findD 1 4 = 4
hm.findD 1 4 = 3


Get outdated logic fails on any invalid ids, for example from OpenPype

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior:
The check whether there are any outdated containers in the scene fails if there happens to be any invalid ids found, like e.g. ids coming from OpenPype instead of AYON.
Expected Behavior:
The invalid IDs should be ignored, similar to how manager may display it as "NOT FOUND" and alike.
Version
1.0.0
What platform you are running on?
Windows
Steps To Reproduce:

Set up an invalid id on a container that is not allowed for the graphql query
The get invalid query should ignore the invalid entry instead of crashing the query.

Are there any labels you wish to add?

[X] I have added the relevant labels to the bug report.

Relevant log output:
# Warning: EventCallback : Failed to execute event callback < EventCallback - on_open > E:\dev\ayon-core\client\ayon_core\hosts\maya\api\pipeline.py
# Traceback (most recent call last):
#   File "E:\dev\ayon-core\client\ayon_core\lib\events.py", line 339, in process_event
#     callback()
#   File "E:\dev\ayon-core\client\ayon_core\hosts\maya\api\pipeline.py", line 653, in on_open
#     if any_outdated_containers():
#   File "E:\dev\ayon-core\client\ayon_core\pipeline\load\utils.py", line 790, in any_outdated_containers
#     if get_outdated_containers(host, project_name):
#   File "E:\dev\ayon-core\client\ayon_core\pipeline\load\utils.py", line 817, in get_outdated_containers
#     return filter_containers(containers, project_name).outdated
#   File "E:\dev\ayon-core\client\ayon_core\pipeline\load\utils.py", line 870, in filter_containers
#     for repre_entity in repre_entities:
#   File "C:\Program Files\Ynput\AYON 1.0.2\dependencies\ayon_api\server_api.py", line 5058, in get_representations
#     for parsed_data in query.continuous_query(self):
#   File "C:\Program Files\Ynput\AYON 1.0.2\dependencies\ayon_api\graphql.py", line 380, in continuous_query
#     raise GraphQlQueryFailed(
# ayon_api.exceptions.GraphQlQueryFailed: GraphQl query Failed: Invalid entity ID<PHONE_NUMBER>bcba09068baf0a on item 'project/representations' (Line 3 Column 5)

Additional context:
No response

@iLLiCiTiT any good ideas how we could best approach this?

Do we know all places where it's used that way? Because I'm afraid those are arbitrary and best would be to filter then in host integration instead of places where are used...

Well, the tricky thing is. They should be returned by get_containers so they can be represented in the manager by the "not found" entries in the UI.
As for, where this applies. Hard to tell, but here's another one:
# Traceback (most recent call last):
#   File "E:\dev\ayon-core\client\ayon_core\tools\sceneinventory\view.py", line 473, in _process_custom_action
#     result = action.process(containers)
#   File "E:\dev\ayon-core\client\ayon_core\hosts\maya\plugins\inventory\rig_recreate_animation_instance.py", line 32, in process
#     contexts_by_repre_id = get_representation_contexts_by_ids(
#   File "E:\dev\ayon-core\client\ayon_core\pipeline\load\utils.py", line 220, in get_representation_contexts_by_ids
#     repre_entities_by_id = {
#   File "E:\dev\ayon-core\client\ayon_core\pipeline\load\utils.py", line 220, in <dictcomp>
#     repre_entities_by_id = {
#   File "C:\Program Files\Ynput\AYON 1.0.2\dependencies\ayon_api\server_api.py", line 5058, in get_representations
#     for parsed_data in query.continuous_query(self):
#   File "C:\Program Files\Ynput\AYON 1.0.2\dependencies\ayon_api\graphql.py", line 380, in continuous_query
#     raise GraphQlQueryFailed(
# ayon_api.exceptions.GraphQlQueryFailed: GraphQl query Failed: Invalid entity ID<PHONE_NUMBER>bcba09068baf0a on item 'project/representations' (Line 3 Column 5)

For the recreate animation instance action in the scene manager.
I've hit this issue in a few other areas as well, which is I passed by pre-filtering the API call by only querying for valid uuid strings.

I'm also dealing with some issues where the publisher UI just won't refresh, doesn't log any errors and I can't publish. Usually starting from a "update products" button that shows, and after that just nothing works anymore. I need to restart Maya for the publisher to become functional again. This also occurs in a scene with these invalid IDs and I wonder whether it may be related.

This has been solved with the invalid id check here for some time. I haven't faced issues with this recently so will close it.


ParseKeyFromURL Is missing some of the helper functions for fields.

ParseKeyFromURL Is missing some of the helper functions for fields, refs below:

https://github.com/google/google-authenticator/wiki/Key-Uri-Format#counter
https://github.com/google/google-authenticator/wiki/Key-Uri-Format#algorithm
https://github.com/google/google-authenticator/wiki/Key-Uri-Format#digits

Would be nice to have them all and use Key object as a building block for TOTPConfig/HOTPConfig

Hey, wanna make a PR ?

Yea, a bit later hopefully today

@cristaloleg Go merge ?)


Unclear if rules specify when device is allowed, or device is blocked

A simple change to the text to clarify would help the UX.

Can you cut a new release with this in it, please?


geometryservice intersect

ArcRest or ArcRestHelper
ArcRest
Version or date of download
3.5.4
Bug or Enhancement
document examples for geometryservice functions
Repo Steps or Enhancement details
I have two features with the following geometries:
{u'y': 6520782.7078824835, u'x': -15918260.703705259, 'spatialReference': {u'wkid': 102100, u'latestWkid': 3857}},
{u'y': 6520782.7078824835, u'x': -15918260.70370526, 'spatialReference': {u'wkid': 102100, u'latestWkid': 3857}}

I'm trying to prove that they intersect  given some amount of tolerance. I think intersect is the function I want to use, but I'm not certain. Perhaps I should be using generalize. I don't know the correct parameter syntax for this function either.
Thanks,

I found a semi-workaround using the arcpy geometries.
>>> feature1.geometry.distanceTo(feature2.geometry)
4.4777
>>> # Using a geometryservice object
>>> geoservice.distance(sr=102100,geometry1=arcrest.geometry.Point(feature1.geometry,wkid=102100),arcrest.geometry.Point(feature2.geometry,wkid=102100))
{u'distance': 4.47213595499958}

I could say if the distance is below some number, that the points are the same. Unfortunately, this function doesn't seem to work when one feature has a wkid but the other feature has only wkt.
The distance function has parameter name "distanceUnit" which is supposed to take these Possible Values: meters | feet | kilometers | miles | nautical-miles | yards. When I try any of these, I get the same error.
>>> geoservice.distance(102100,arcrest.geometry.Point(feature1.geometry,102100),arcrest.geometry.Point(feature2.geometry,102100),distanceUnit='feet')
{u'error': {u'message': u'Unable to complete Distance operation.', u'code': 400, u'details': [u'The specified WKID (factory code) or WKT (definition string) is not a  linear or angular unit.']}}

I can't tell if the distanceUnit error is a REST or ArcREST issue.

@DShokes - is this an ArcGIS Online or ArcGIS Server geometry service?

The distance units must be these:
http://resources.arcgis.com/en/help/arcobjects-cpp/componenthelp/index.html#/esriSRUnitType_Constants/000w00000042000000/
or
http://resources.arcgis.com/en/help/arcobjects-cpp/componenthelp/index.html#/esriSRUnitType_Constants/000w00000041000000/

Usage Example:
pt1 = arcrest.common.geometry.Point(coord=[4,4], wkid=4326)
pt2 = arcrest.common.geometry.Point(coord=[5,5], wkid=4326)
print (gs.distance(sr=4236, geometry1=pt1, geometry2=pt2, distanceUnit=9036, geodesic=True))

Thanks @achapkowski , I didn't think to use  an integer for distanceUnit or that the options were located in the ArcObjects documentation.


use extract-css-core instead of API call

Resolve WIP when extract-css-core hits v2.0.0

closes #76


javascript error

If you do not set urls in AppConfig, you get errors, it should not check for https like this.
this.base = window.location.href.match(/http[s]?://api./)[0];
We get js error: Cannot read property '0' of null
ERROR TypeError: Cannot read property '0' of null
at new Formio (Formio.js:125)
at FormBuilder.setForm (Form.js:74)
at FormBuilder.render (Form.js:115)
at FormBuilderComponent.push../node_modules/angular-formio/components/formbuilder/formbuilder.component.js.FormBuilderComponent.buildForm (formbuilder.component.js:53)
at FormBuilderComponent.push../node_modules/angular-formio/components/formbuilder/formbuilder.component.js.FormBuilderComponent.ngOnChanges (formbuilder.component.js:81)
at checkAndUpdateDirectiveInline (core.js:9239)
at checkAndUpdateNodeInline (core.js:10507)
at checkAndUpdateNode (core.js:10469)
at debugCheckAndUpdateNode (core.js:11102)
at debugCheckDirectivesFn (core.js:11062)

@travist What is the status on this? I'm use angular-formio v.4.4.4 and I get the same error as dolphinsd. I fixed it for now by setting de appUrl and apiUrl in the AppConfig but it's nog the way it should work.

This just needs a check around that line of code. If someone would submit a pull request, I will happily accept it.

Closed due to no activity. If the issue is still relevant, please reopen


Check for duplicate selections in generate command

Aims to partially address #158
Queries w/ duplicate selections (top-level & nested)

Output produced when running houdini generate

Getting a strange type error in yarn tests here, though my text editor doesn't throw these type errors. Any idea why this is happening?

As mentioned in #158 I'm not sure I understand the reasoning behind imposing this limitation on the user.  I'm going to hold off on reviewing this closely until we agree its the right move :)

Going to close this since there was a small miscommunication about the intents. @ledesmablt - feel free to open up another PR that handles duplicate list operations!


Fix login failed message

Fixes #2748
@timabbott, please review.

Does it make sense to create a common validator here?
def validate_login_email(email, check_ldap=True):
    if ldap_auth_enabled():
        # Our LDAP backend is our only backend that
        # can authenticate against a non-email.
        return

    try:
        validate_email(email)
    except ValidationError as err:
        riase JsonableError(str(err.message))

Yep.

@timabbott, @showell, I have updated the code. Please review.

Merged, thanks @umairwaheed!

@kunall17 @Sam1301 FYI -- this is now deployed on chat.zulip.org.  I can't remember if we had an Android project issue for this, but it's worth checking if this now does the right thing on zulip-mobile at least.


cell output location is not preserved when exporting

Describe the bug
I use "bottom" option for "Cell output area". When I try to export (download) as HTML, outputs appear on top.
Environment
{
  "marimo": "0.8.3",
  "OS": "Windows",
  "OS Version": "11",
  "Processor": "AMD64 Family 23 Model 24 Stepping 1, AuthenticAMD",
  "Python Version": "3.11.9",
  "Binaries": {
    "Browser": "--",
    "Node": "--"
  },
  "Requirements": {
    "click": "8.1.7",
    "importlib-resources": "missing",
    "jedi": "0.19.1",
    "markdown": "3.7",
    "pymdown-extensions": "10.9",
    "pygments": "2.17.2",
    "tomlkit": "0.12.3",
    "uvicorn": "0.30.6",
    "starlette": "0.38.2",
    "websockets": "11.0.3",
    "typing-extensions": "4.9.0",
    "ruff": "0.5.6"
  }
}

Code to reproduce
No response

hey @Zaf4, i have a fix out for this https://github.com/marimo-team/marimo/pull/2129


Fix prefix maximum length

18 is a too short to enter an IPv6 prefix (ie. 2001:db8:ffff:ff00:/56 is longer than 18 chars)

18 is a too short to enter an IPv6 prefix (ie. 2001:db8:ffff:ff00:/56 is longer than 18 chars)

Thanks for reporting, nice catch and yes you're right, but 43 would be the absolute maximum for a valid IPv6 address with prefix length. IMO in this context we would only use IPv6 prefixes and the longest valid prefix would be something like 2001:db8:ffff:ffff::/64 which would require 24 chars. However, I also saw longer IPv6 prefixes in the data which IMHO are wrong but still used - so your patch will cope with that, too.
Thanks again.


Markdown isn't showing/rendering after cleaning the page

Description
My app contains multiple logical pages (I am cleaning the page and adding the needed objects for the needed page).
I am using Markdown to render some texts within one of these pages (help page).
I saw that when I run the page containing the markdown object, it renders correctly, but if I go to another page within the same app, and go back to the page containing the Markdown, it shows nothing.
Code example to reproduce the issue:
    def help_page(e):
        page.clean()
        add_basic_buttons(page) ### Func to add the basic buttons above
        help_msg = """
## About...

- aa
- bb
- cc
        """
        page. Add(
            Markdown(
                value=help_msg,
                selectable=True,
                extension_set="commonMark",
                on_tap_link=lambda e: page.launch_url(e.data),
                expand=True,
                )
        )

Describe the results you received:
As said, first click on 'Help page' it shows the below rendered:

If I go to another page, and then come back to 'Help page', it doesn't show the Markdown
Describe the results you expected:
I expect every time to render the Markdown and show the result.
Flet version (pip show flet):
Name: flet
Version: 0.1.52
Summary: Flet for Python - easily build interactive multi-platform apps in Python
Home-page:
Author:
Author-email: Appveyor Systems Inc<EMAIL_ADDRESS>License: MIT
Requires: beartype, repath, websocket-client
Required-by:
Operating system:
macOS
Additional environment details:
NA

Thanks for starting an issue with providing the details!

Could you please upgrade to the latest Flet 0.1.60?
if I go to another page within the same app, and go back to the page containing the Markdown - provided code is not enough to see how do you navigate between pages. Could you please provide more complete repro?

hmm but now tested again for your example mentioned here:
https://flet.dev/docs/controls/markdown
And seems the scrolling isn't working anymore

omg, so scrolling has been resolved on flet==0.1.60 by removing page.scroll from the code


[$250] Android - Send invoice - In conversation page, transaction preview shown briefly and disappeared

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.71
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail: N/A
Issue reported by: Applause - Internal Team
Action Performed:

Launch app
Tap fab --- send invoice
Enter an amount and tap next
Select a contact
Tap send invoice

Expected Result:
In send invoice conversation page, transaction preview must not be shown briefly and disappeared
Actual Result:
In send invoice conversation page, transaction preview shown briefly and disappeared
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/f060fa77-cfb8-4ab6-a9eb-5aa3726a8326
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~014e2c68306bbf442b
Upwork Job ID:<PHONE_NUMBER>335278080
Last Price Increase: 2024-05-08

@lschurr  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors

Root cause
In the server's response to SendInvoice, the disappearing ReportAction has an empty message.
Solution
Server
reportAction.message[0].html and reportAction.message[0].text should be non-empty strings.
Front-end
This can be worked-around by adding the message to the successData on the front-end.

We knew about this issue and we're working internally on it

Any update @cristipaval?

^fix PR up

The fix PR hit production. No payments due for this issue.


Removed API method from InstanceOperations that exposed Thrift object

Removed API method that exposed Thrift object from InstanceOperations API. Modified method that called it to use the method internal directly and fixed up the test
Closes #2983

So the issue here with the test breaking is that the point of the new method that was removed was to only return the system properties but now the assertion is done on getSystemConfiguration() which is going to return the entire config.
Why not just keep the public getSystemProperties() method but have it return a map instead?


Async actions before the server starts

There are a number of async actions you might want to do before the server starts. While this works fine if you create the listener itself, it precludes using things like micro-dev or the micro binary with this package. It'd be nice if we could include some async "setup" steps somehow. Maybe something like this?
const db = new DB()
module.exports = async (req, res) => await db.getSomeAsyncData()
module.exports.setup = async () => { await db.connect() }
module.exports.teardown = async () => { await db.disconnect() }

EDIT: added teardown

Maybe this should also include an action for shutting down. micro-dev would need to shut down and restart the database service in the above example.

Probably you know, you can use micro programmatically something like that:
const micro = require('micro')
const sleep = require('async-sleep')

// here your request handler
const handler = async (req, res) => {
  return 'Hi'
}

const createServer = () => {
  const server = micro(handler)
  server.listen(process.env.PORT || 3000, () => {
    console.log('Running on http://localhost:3000')
  })
}

const setup = async () => {
  // setup db your connections here
  await sleep(3000)
}

const main = async () => {
  await setup();
  console.log('Setup completed')
  createServer()
}

main()

For graceful shutdown you can use process events such exit, SIGINT and other.

For graceful shutdown you can use process events such exit, SIGINT and other.

This doesn’t work with micro-dev because it doesn’t use a separate process.

Looks like not only process signals are not working in this case.
If you will use micro programmatically, micro-dev and micro runners will not work.

Another question is do you really need setup connection before your server is running and keep it in runtime? micro is good for granular microservices, so probably you no need keep db connection all time while your microservice is running. Just setup new one as soon as it's really needed.
Correct me if I'm not right.

The database was just an example. I think shutting down the connection is actually more important for making micro-dev work. If you use something like next.js, then it will start a new webpack instance each time the server is restarted. You could do the initialization in the handler itself, but there is no way to shut down next.js instance with micro-dev once the reference is lost (after it drops the require cache). I also don't think leaving a bunch of connections to the database open during development is a great solution either.

I guess you can do something like this to ensure prework is done before request:
const dbPromise = new Promise((resolve, reject) => {
    resolve(new Db()); // Or something async
});

module.exports = async (req, res) => {
    const db = await dbPromise();
    // Do some work with db
}

Teardown is a little trickier – if https://github.com/zeit/micro-dev/issues/21 is resolved (for example with sending SIGUSR2 to own process), then we could just listen to it:
process.once('SIGUSR2', function () {
    dbPromise.close();
});

@floatdrop SIGNUSR2 signal not working. I support @cjhowe7 suggestion.

@ansarizafar check, that you have fresh version of micro-dev, SIGNUSR2 was released not so long time ago – https://github.com/zeit/micro-dev/commit/b0281bff4d0e5a570be116e32c61599ad0e4c85f

Also chiming in to voice my support for the original suggestion. I'm currently using the implementation that @floatdrop has suggested, but I feel that @cjhowe7 proposal would be cleaner and more self-descriptive. It also removes the need for checking prior to every request that the initialisation has already been done.


[NO-JIRA] Various fixes and updates

Overview

add binary name to gitignore
add makefile
change dockerfile org and make it use the makefile
support newlines and spaces in ignored jobs list
support more than 30 statuses (default github API limit)

@dambrisco opened the following:

DH specific stuff - #3
https://github.com/upsidr/merge-gatekeeper/pull/48
https://github.com/upsidr/merge-gatekeeper/pull/49

Gonna close this PR


Add support for Kia Australia

Hyundai / Kia Connect version: com.kia.kiaconnect.aus / 1.0.1
Python version: Version included with HA
Operating System: HA OS

Description
I would like to add support for the Kia Connect Australia app. It looks like the base of the configuration is complete as the app looks very similar to the Hyundai Bluelink app which is already supported in Australia.
What I Did
I get the error "Kia is not supported in Australia yet" when adding the integration to Home Assistant. This looks to be the response from KiaUvoApiAU.py.
Looking through the code, it looks like some simple extension is required.
URL:
BASE_URL: au-apigw.ccs.kia.com.au:8082
I'm unsure how to gather the remaining three pieces of information for the PR. Can you please advise what you need me to do to gather these bits of information?
Source code:
if BRANDS[brand] == BRAND_KIA:
raise APIError("Kia is not supported in Australia yet")
elif BRANDS[brand] == BRAND_HYUNDAI:
self.BASE_URL: str = "au-apigw.ccs.hyundai.com.au:8080"
self.CCSP_SERVICE_ID: str = "855c72df-dfd7-4230-ab03-67cbf902bb1c"
self.APP_ID: str = "f9ccfdac-a48d-4c57-bd32-9116963c24ed"  # Android app ID
self.BASIC_AUTHORIZATION: str = "Basic ODU1YzcyZGYtZGZkNy00MjMwLWFiMDMtNjdjYmY5MDJiYjFjOmU2ZmJ3SE0zMllOYmhRbDBwdmlhUHAzcmY0dDNTNms5MWVjZUEzTUpMZGJkVGhDTw=="

You need to install nox with the mobile app and sniff the https traffic to collect most of those.   I think the xor value also needs to be generated.

@bitnimble  is our resident australia expert.

Any chance this would be possible/ @bitnimble is this something you are involved in?

I've tried playing with au-apigw.ccs.kia.com.au:8082 and posting but cannot get past authentication

I tried to use "Burp" to intercept the calls and watch the traffic but the Kia Connect app gives me a network error. So i'm stumped how to intercept.

Probably not anytime soon sorry - even Hyundai is out of date and hasn't worked for months and I haven't found the time to fix that one 😅

No prob. I just spent some time reviewing the Hyundai AU code and tried to copy it for Kia but keep getting 4010 Authentication errors. I then set up an account with Hyundai and followed the steps to get cookies and then try to login, but got 4003 Invalid Values - i guess this is what you mean by Hyundai not working currently either.
Happy to help @bitnimble  - any suggestions how best to sniff the traffic given there is only an Android app, no website?

Would decompiling the APK be an option?

Try install nox player. Load up android 6 with it.  No rooting.    Use Charles proxy to setup ssl decrypt.  If you can see the traffic you can get the ids and I can point you in the right direction from that.

Thanks. I did try Memu emulator and got the apk to run, but it kept giving network errors on login. Oddly it would correctly give an error for wrong password, but a network error for correct pwd.. Ie they somehow detect its on an emulator? This was even without running an interceptor / proxy

From: cdnninja @.>
Sent: Saturday, February 10, 2024 11:16:40 AM
To: Hyundai-Kia-Connect/hyundai_kia_connect_api @.>
Cc: zappoo @.>; Comment @.>
Subject: Re: [Hyundai-Kia-Connect/hyundai_kia_connect_api] Add support for Kia Australia (Issue #420)
Try install nox player. Load up android 6 with it. No rooting. Use Charles proxy to setup ssl decrypt. If you can see the traffic you can get the ids and I can point you in the right direction from that.
—
Reply to this email directly, view it on GitHubhttps://github.com/Hyundai-Kia-Connect/hyundai_kia_connect_api/issues/420#issuecomment-1936763226, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADU3BF2VBTRUJAJBZ3HIBEDYS236RAVCNFSM6AAAAAA5UEE6NSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZWG43DGMRSGY.
You are receiving this because you commented.Message ID: @.***>

Try nox to rule it out.  It worked for Australia last time and also for Europe.

Spent hours trying to get NOX working but keep getting stuck on the "Failed to start Virtual Machine" screen. I'll try again later :(

OK got BlueStacks working. Installed Kia Connect. Logged in - SUCCESS! Now i will try to do some sniffing - i assume using something like Fiddler??

Got a bit further.. set up fiddler, modified some files in bluestack emulator to set up fiddler as a proxy.. anything i now do in Chrome on the emulator proxies into Fiddler.. but Kia Connect says "Network error" when i start the app now.

Got a bit further.. set up fiddler, modified some files in bluestack emulator to set up fiddler as a proxy.. anything i now do in Chrome on the emulator proxies into Fiddler.. but Kia Connect says "Network error" when i start the app now.

Have you tried installing the SSL certificate (as system CA certificate) from fiddler? If you're running the app on android 6, and using fiddler correctly, you should be able to view all the traffic. If you're using a later android version, you'll need to root it to move the certificate from the user store to the system store.
Once that's sorted, the network error should be solved.

Am trying to figure that bit out. Bluestack doesn't give me an option to install certificates anywhere

Status:

Kia connect works on the emulator
Emulator is proxying via fiddler, so i can see traffic from chrome etc
I think i've managed to install the Fiddler CA cert correctly
If i try to set the fiddler proxy as a global system proxy, i need root. If i enable root, then Kia Connect fails to start as it detects root

That's where I am currently stuck.

yeah, the australian version have quite strong root prevention. you'll need to either use something like magisk to do root hiding, or decompile + strip the protection + recompile.

Yep - that's a bit beyond me :(

The other thing i've tried is to enable root, modify 'settings_global.xml' on the device, add in all proxy settings, then UNSET root. Again, chrome traffic gets picked up - but Kia Connect says "Network error". Perhaps this idea works, but i haven't installed the fiddler cert properly??

ah that's probably the SSL pinning at work - if you load your own cert in order to sniff the traffic, then the app detects that the certificates have changed and rejects any requests 😅 so you need to modify the apk to disable the SSL pinning detection, which even then doesn't work yet because Hyundai (and presumably Kia as well) in Australia have one more layer of protection which detects that the apk signature has been changed. So you need to also remove that protection / inject the SSL certificate dynamically at runtime instead.
If you join the Discord, I think there might be some old messages from me that go through the flow. Otherwise, I'll try to get the Hyundai one fixed in the next few weeks and document the whole procedure (for myself, too... I can hardly remember what I did last time lol)

Thanks! Yep - very frustrating :(

the australian one is quite a bit more annoying than other countries unfortunately 🥲

It looks like the PR from @cdnninja looks to have fixed it. The latest version discovers my car (Niro EV) as expected.


[WebUI] Fix collapse of WebUI layout

When we decrease the width of browsers, the header of WebUI wraps and collapses like as following image.

Test build #25397 has started for   PR 3995 at commit 5d83e18.

This patch merges cleanly.

Test build #25397 has finished for   PR 3995 at commit 5d83e18.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25397/
Test PASSed.

@sarutak can you give a screenshot of what it looks like before your patch (or after?). I can't tell whether the one here is before or after.

@pwendell That's the before patch applied. I've updated the description.

Hey @sarutak I tried this out locally and it appears that the "Jobs, Stages, Storage ..." and the application name are shifted upwards too close to the top of the page. Is there something we can do about it?

@andrewor14 I fixed the layout as follows. How do you think?

Test build #25673 has started for   PR 3995 at commit 3e60b5b.

This patch merges cleanly.

Test build #25673 has finished for   PR 3995 at commit 3e60b5b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SparkListenerJobEnd(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25673/
Test FAILed.

Ok, looks good. This just changes CSS so the tests are irrelevant here. I'm merging this into master thanks.


Frontend issues in 0.17

Home Assistant release (hass --version):
0.17.2
Python release (python3 --version):
3.4
Component/platform:
Front end
Description of problem:
Really difficult to see the state I am watching.
If you got something that change state often, lines get thin.
When I hover the mouse over the time line it's hard to tell if it is "home" or "away" the pop-up shows. Before 0.17 the state where in the popular together with the time.
And I really would like an option to get "a few minutes ago" back in the front end.
Now it's only empty space there.

The last time the state was changed has been removed during our UI revision. To see that information you can open the more info dialog.
Our history is not optimized for navigation on a phone. It's on the to do list but very low priority.

The printsceen is on a phone but I got the same issue in Chrome, Explorer and Firefox.
Look att these pics, this is in Chromomium.

Very hard to tell if I'm looking at home or away state.

Use the history tab to explore them. There is currently no good solution
for this.
On Sun, Apr 10, 2016, 23:59 xydix<EMAIL_ADDRESS>wrote:

The printsceen is on a phone but I got the same issue in Chrome, Explorer
and Firefox.
Look att these pics, this is in Chromomium.
[image: img_20160411_085343]
https://cloud.githubusercontent.com/assets/18333779/14419083/9885c0a8-ffc3-11e5-9615-3756a457ddda.jpg
[image: img_20160411_085355]
https://cloud.githubusercontent.com/assets/18333779/14419085/9d12dafc-ffc3-11e5-9543-22bed55f609b.jpg
Very hard to tell if I'm looking at home or away state.
—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub
https://github.com/balloob/home-assistant/issues/1787#issuecomment-208189347

I see all the old info at the top of the more info dialog, in the mobile browser screenshot 'Peter Home, 25 minutes ago' and in the desktop browser screenshot 'Peter Home, 42 minutes ago'.
The red/blue bar under that has always been somewhat hard to use when there are frequent, or short interval, updates. So that isn't really a 0.17 regression, maybe more of a feature request that should go to the forum.

Hi guys.
i know i can see 'Peter Home, 42 minutes ago' if I click it first.
Just missing it in the frontend. If devs want it gone I accept that and will probably get used to it but more info is better if you ask me.
But the issue i try to explain is this:

The state in the red circle. This show the actual state with the time. This is GREAT.
It's OK in the history section but on the statepage you can't get this info. It was there before 0.17.
This is not an improvement if you ask me.
Anyway, give it a thought. Thanks for awesome software.
Now i will just try to learn how to config everything. :-)


Issue color in siconos_vexport

When using siconos_vexport with or without global-filter with object with multiple contactor, the cell data are not connected with the shape. Then, it is not possible to colorize objects with cell data in paraview.
The following example show the problem.
[
test_relative_orientation.tar.gz
](url)

With or without --global-filter, the contact shape instance (a number) is associated in CellConnector (in vview.py). So you can choose it in the coloring box in paraview as this:
.
When --global-filter is not specified, I have the coloring of each block (each contact shape) directly:
.
In this case, you can also choose instance or vtkCompositeIndex

Hi @bremond,
If you have a look when --global-filter is specified in your first picture, there is two cylinder that stay in blue, whatever the value of the coloring variable you chose. There is a problem with compound objects (objects with multiple contactor shapes). For instance, they should share the same kinetic energy color, but it is not the case.
I attach the same example with only compound objects.

test_relative_orientation.tar.gz


Study

Study Papers

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

https://github.com/Sohl-Dickstein/Diffusion-Probabilistic-Models

Denoising Diffusion Probabilistic Models

https://github.com/hojonathanho/diffusion
https://github.com/lucidrains/denoising-diffusion-pytorch

Improved Denoising Diffusion Probabilistic Models

https://github.com/openai/improved-diffusion

Diffusion Models Beat GANs on Image Synthesis

https://github.com/openai/guided-diffusion

High-Resolution Image Synthesis with Latent Diffusion Models

https://github.com/CompVis/stable-diffusion

Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models

https://github.com/compvis/latent-diffusion

U-Net: Convolutional Networks for Biomedical Image Segmentation

https://github.com/milesial/Pytorch-UNet


fix: adding explicit $ sign to hex32. format in mp_md5() macro

Credit to the eagle eyes of FreelanceReinhard!

:tada: This PR is included in version 4.15.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


NuGetCommand restore fails when using Fiddler as per troubleshooting instructions but does not exit with error code

Required Information
Question, Bug, or Feature?
Type: Bug
Enter Task Name: NuGetCommand@2
Environment
Azure Pipelines
Private Agent
OS: Windows Server 2016
Agent version:2.179.0
Issue Description
I followed the instructions for troubleshooting Azure Pipelines https://docs.microsoft.com/en-us/azure/devops/pipelines/troubleshooting/review-logs?view=azure-devops under the paragraph "Use full HTTP tracing - Windows".
After setting the agent to use Fiddler as proxy the NuGet task would exit with the message:
##[warning]unable to verify the first certificate
(node:5700) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: unable to verify the first certificate

The task was reported as successful but the following MSBuild task would fail because the NuGet packages had not been restored.
I could get around the problem by setting the system environment variable NODE_TLS_REJECT_UNAUTHORIZED to 0, but I think the NuGet task should be reported as failed and not successful.
Error logs
Starting: NuGet restore contoso.sln
==============================================================================
Task         : NuGet
Description  : Restore, pack, or push NuGet packages, or run a NuGet command. Supports NuGet.org and authenticated feeds like Azure Artifacts and MyGet. Uses NuGet.exe and works with .NET Framework apps. For .NET Core and .NET Standard apps, use the .NET Core task.
Version      : 2.179.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/package/nuget
==============================================================================
"C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/MSBuild/Current/Bin/msbuild.exe" /version /nologo
"C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/MSBuild/Current/Bin/msbuild.exe" /version /nologo
Found tool in cache: NuGet 5.4.0 x64
Found tool in cache: NuGet 5.4.0 x64
Resolved from tool cache: 5.4.0
Using version: 5.4.0
Found tool in cache: NuGet 5.4.0 x64
SYSTEMVSSCONNECTION exists true
##[warning]unable to verify the first certificate
##[warning]unable to verify the first certificate
(node:5700) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: unable to verify the first certificate
Finishing: NuGet restore contoso.sln

Would be nice with some response from a human.


getShortPinyin报错

像  "a,b" 这种会报错，中间是英文逗号，我试了中文逗号，句号，英文句号等都可以，唯独英文逗号不行，这么什么问题？
The following RangeError was thrown while handling a gesture:
RangeError (index): Invalid value: Valid value range is empty: 0
When the exception was thrown, this was the stack:
#0      _StringBase.[] (dart:core-patch/string_patch.dart:252:55)
#1      PinyinHelper.getShortPinyin. (package:lpinyin/src/pinyin_helper.dart:51:21)

你这都不是汉字怎么能拿来获取首拼呢
:octocat: From gitme Android

fix


Add support for force loading some image classes

For example lazy loading mega menu images in wordpress works only scrolling a bit
that is where force loading may be come handy, also help web devs who want to load some images without lazy loading

LazyLoad's job is to check whether images are inside the viewport, then load them automatically.
For what I understand, you need something to load some images when your menu is shown. So in the code that manages your menu, you want to do something like new LazyLoad('.megaMenuImage', {immediateLoad: true}), right?

Yes :)

Good,
then you don't need LazyLoad for the menu.
LazyLoad'a main job is to check whether or not images are in the viewport, or close to it, then load it in the positive case.
What you need to do is just to copy the data-src attribute value to the src attribute and eventually the data-srcset attribute value to the srcset one.
This can be easily done with a couple of lines of code that you can put in your menu component / manager / handler.

@Niresh12495 can we close this issue?

Hi, sorry for very late reply so do you want me to add a code to the code that manages megamenu? that is not possible in most cases, is there anything that i can do with lazyload js options itself

Look, I don't know megamenu but you need to write some code to do what I wrote some comments before. Not necessarily editing megamenu's code, but Listening to the events it fires maybe.


added endpoints for removing public IPv4 addresses

Coverage decreased (-0.03%) to 78.132% when pulling 4d53f2cdb3de521bb0b53d680cc278faad4036cf on mparke:remove-public-ipv4 into ddbe8233ea0409da5b5aa820e290c7f4fce1f21c on linode:develop.


Improve/documentation

just some minor corrections (Numeric instead of Number, "name" instead of "nam")

I have no idea the last two commits got into this branch... sigh


Saved requests in collection are not appearing in the UI

I have checked the following:

[X] I use the newest version of bruno.
[X] I've searched existing issues and found nothing related to my issue.

Describe the bug
As of version 1.33.0, creating a new request in a collection will save it to the directory and display it in the UI.
When reopening Bruno, the request is no longer there. I can see the .bru file in the directory. Closing and reopening the collection doesn't work.
However, creating a new request with the same name as the previous one, doesn't work due to it saying the name has already been taken.
Any one else seen this issue?
I'm using a Macbook Pro 2019, with an intel processor.
.bru file to reproduce the bug
Cannot share.
Screenshots/Live demo link
Will share soon

I’m experiencing the same issue, and additionally, it also occurs with the saved environments. Could you provide further guidance or support on this?
.bru file to reproduce the bug
Download Reports.bru
Screenshots/Live demo link

In this example, the requests for 'Search Items' and 'Download Reports' have been omitted.

yeah i have the same issue. requests are disappearing, sometimes also collections. when the requests are disappeared it sometimes help to try to re-open it, close bruno after the error message that is it allready opened and then start bruno again. very unstable and nearly unusable, had lots of hassle with it. if you say that was better befor e 1.33.0 i'll try to downgrade.

Same issue here. The collection is still in the list after re-opening bruno (1.33), but all requests have disspeared. If I try to open the collection manually I get an error and the console log shows:
ValidationError
    at Object.createError (_app-5b55a57df7ed022e.js:52:106730)
    at Object.test (_app-5b55a57df7ed022e.js:55:18238)
    at validate (_app-5b55a57df7ed022e.js:52:106987)
    at runTests (_app-5b55a57df7ed022e.js:52:104378)
    at _app-5b55a57df7ed022e.js:55:758
    at _app-5b55a57df7ed022e.js:52:104111
    at _app-5b55a57df7ed022e.js:52:104631
    at _app-5b55a57df7ed022e.js:52:107051
407-ca3247c8a0362824.js:1 BrunoError: The Collection file is corrupted
    at 407-ca3247c8a0362824.js:1:6101

OK, it seems there were two problems:

the collection had a ":" in the name  field set in bruno.json, that's why trying to open it resulted in the validation error above. I had no issues creating the collection with that name before, and I'm not really sure why such a common character is not allowed in the name.
now that I manually renamed it, I can get around the bug by closing the collection and re-opening it. If I restart bruno, there will again be zero requests in the collection until I close and re-open it again.

when my bru file(s) are corrupt why should them load sometiimes but most of the time not, but when i close the collection and reopen? i can't send my collections, they contain private data. maybe someone else could.

same issue here! I think this happend due to syncing issues after using a different code editor like vscode or cursor. I could send you the specific get request file, however I cannot send the whole collection.

Error invoking remote method 'renderer:bru-to-json': Error: Line 11, col 3:
10 |   auth: inherit

11 | } when { !bru.getVar('skip_lead_creation') }
^
12 |
Expected end of input, "docs", "tests", "script:post-response", "script:pre-request", "assert", "vars:post-response", "vars:pre-request", "body", "body:multipart-form", "body:form-urlencoded", "body:graphql:vars", "body:graphql", "body:sparql", "body:xml", "body:text", "body:json", "auth:apikey", "auth:wsse", "auth:oauth2", "auth:digest", "auth:bearer", "auth:basic", "auth:awsv4", "headers", "params:query", "params:path", "query", "trace", "connect", "head", "options", "patch", "delete", "put", "post", "get", or "meta"

Found this error in Bruno when I copy pasted my code into Bruno. Basically shouldn't be using a "when" condition within the bruno file structure.

Hey everyone,
I completely get how frustrating it can be when requests vanish without an obvious explanation.
We're working swiftly to set up an error console at the collection level to show parsing errors, helping you understand why a request might not load.
Thanks for your patience.

ok here my newest observations. i tried all diffent methods of bruno install. since there is no difference in the vanishing effect, i sticked to the flatpak/flathub installation now. when i start bruno over its automatically generated flatpak desktop file (using ubuntu 24.04), the problem appears more often. when i launch it over the command line, less often. sometimes some collections are missing, sometimes some requests, sometimes everything is there. here is the ouput i get when i start it over command line:
[3:1112/204121.071269:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
Preferences file is located at: /home/ulf/.var/app/com.usebruno.Bruno/config/bruno/preferences.json
Gtk-Message: 20:41:22.268: Failed to load module "xapp-gtk3-module"
Gtk-Message: 20:41:22.329: Failed to load module "canberra-gtk-module"
Gtk-Message: 20:41:22.331: Failed to load module "canberra-gtk-module"
[3:1112/204122.353557:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[3:1112/204122.353636:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
watcher add: (several lines of this for all my collections)

a friend of mine uses endeavour linux with the arch package of bruno and doesn't have any problem with the disappearing requests/collections issue. i am using the actual ubuntu lts with gnome and the flathub package of bruno and have these problems.

I am running Debian Trixie and Bruno v1.36.1. I have a collection with 4 folders. If I open Bruno before VSCode all folders in that collection are available. If I open VSCode first and then Bruno, only the first 2 folders are available.

Hello
I've got a problem like it: since i have installed the 1.36.1 version, all the requests in the root of all my collections are unavailables.
I solve it by closing and reopening all the collections i have, one by one.


Caitlin M. Cusack has two first names.

In some cases, Caitlin has two given (first) names -- "Caitlin" and "M." -- and in others a single given name "Caitlin M.". E.g.
https://github.com/preston/vha-kbs-knarts/blob/612e893c65226b306a71c1b9869c39f53554a3af/content/womens_health_cervical_cancer_screening/o22/CDSK_KRprt_SADT_O22CervCan.xml#L220
The labels correspond to cases with multiple first names.

Should now be consistent across all artifacts. All current artifacts have been revalidated.


inputs.UnknownEventType: We don't know what kind of event a Key is.

code:
print(inputs.DeviceManager().gamepads[0].read())

I also have this issue. To detect gamepads which have been plugged in after the program is started, you need to create a new DeviceManager() instance. However, whenever I try to use a gamepad gotten from a new DeviceManager instance, I get an inputs.UnknownEventType error.


feat: Bump Alpine to 3.14 in Dockerfiles

Signed-off-by: Gábor Lipták<EMAIL_ADDRESS>

@seankelly001 please release workflows

@seankelly001 I'm not sure what the actual error here is https://github.com/fidelity/kconnect/pull/359/checks?check_run_id=3522791099#step:3:10
Error: Resource not accessible by integration


Issue final Credential to DI

User story

As a HR-employee
I want / need that after the owner of a connection has answered to a proof request (thus sent self-attested attributes via Lissi Wallet) a final credential which possesses all attributes is created for that connection
So that the connection has a final credential which resembles the full Mitarbeiterausweis for example.

Acceptance criteria

process should be:

HR-employee creates "basic" credential definition which is used for the proof template
HR-employee creates the proof template and defines all self-attested attributes
HR-employee creates "final" credential definition which includes the attributes from the basic credential definition plus the self-attested attributes

With the help of this "final" credential definition, a credential which holds all information will be issued

Proposal: In the process of the proof template creation, provide the HR employee with a list where he can select the final credential definition (must already exist in Lissi at that moment) which should be used for the final credential creation

Expected Functionality for now:

after a proof request is answered from holder of the DI/connection via the Lissi Wallet, a credential based on the final credential definition (which has been linked to the basic credential definition or the proof template before) is created and visible in Lissi wallet.
attributes from proof template and basic credential definition are matched with attributes of the final credential definition based on their name. Thus, attribute names must match (suggestion from POs)
currently, the process is triggered via automatic checks on a fixed timely basis (e.g. every 30 mins): if new proof requests have been answered, trigger process

Definition of done (DoD)

code peer-reviewed (by at least one team member)
acceptance criteria met
integrated into a clean build
documentation updated
code is well-formatted (according to our norms) and structure
Code supports all functions/cases and all responses defined in the API
no severe bugs during main functionalities (program should not stop executing due to these bugs, e.g. runtime exception in Docker)
Bill of Materials updated if necessary
70% of the tests pass
important functions shall be tested
Code has been deployed via Docker to the server

DoD general criteria

Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release

@TamaraHoock @veitmo From a technical point of view, I would require the final credential when creating a proof template. This way, the linking makes more sense (as you define the self attested attributes within the proof template)

@Idontker @TamaraHoock you are right. I flipped the two steps in the process description. I hope that it makes more sense now


Please add custom calendar class property

Many solution has .datepicker class so much conflicts in common. Please change classname or make it dynamic. Thank you!

Will be fixed with #391

Fixed with #391


Function of dtoggle_led in tools.py ???

What should the function "dtoggle_led" in tools.py do?
Two times NOT results in YES, which does not change the state of the LED!

This is exactly, what we want. So the state won't changed but the led flickers. Pls see the README for details.

I think that is wrong.
The two lines are without delay directly behind each other.
The time interval is only nano/microseconds.
There is absolutely nothing to see on the LED!
I have tested this in a loop.

Sorry, Harry - the code is running and I see the led flickering. I agree, that I wasn't sure, that it will be function without delay, but it do!

I think the flickering you see comes from the toggle_led in _send_answer.

If you think so - without INIT on CPplus the port doesn't send one byte, but the led is flickering. I don't understand, what the problem is because it is only a debugging feature.


Airflow improperly shows task status as 'up for retry' for a task that failed on re-run

Dear Airflow Maintainers,
Environment
Before I tell you about my issue, let me describe my Airflow environment:

Airflow version: 1.7.0
Airflow components: webserver, mysql, scheduler with celery executor
Python Version: 2.7.6
Operating System: Linux Ubuntu 3.19.0-26-generic

Description of Issue
Now that you know a little about me, let me tell you about the issue I am having:
What I expect:
If I do a re-run and it fails - The task should be either re-tried again (resetting retry count) and marked accordingly in GUI OR not retried - and marked in GUI as 'failed'

What happened instead?
The task in the GUI was presented as 'up_for_retry' however it was not retried, even after retry_delay has passed

Reproducing the Issue
DAG does not have some strange settings:
concurrency= 3,
max_active_runs = 2,
start_date = datetime(2016,04,03,01),
default_args={
'depends_on_past': False,
'retries': 2,
'retry_delay': timedelta(minutes=3) }

@kretes Can you create a Jira issue for this and provide a reproducible case with it plus screenshots? Thanks

migrated to https://issues.apache.org/jira/browse/AIRFLOW-138


Validation of datetimes

We currently have a BlogPost class that looks like this:
import typing
from datetime import datetime

from gilbert.content import Templated, Content

class BlogPost(Templated, Content):
	title: str
	authors: list
	posted: datetime
	modified: typing.Union[None, datetime]
	category: str
	tags: list
	template: str = "blog/post.html"
	summary: str
	draft: bool

We want to make sure a posted item (the post date) contains a proper datetime.
The following contains no posted however it seems to pass through the validation with no issues:
content_type: BlogPost
title: My Third Post!
authors:
- Alysha
category: software
summary: A short description
---

Hello!

Is there something we are getting wrong in our BlogPost definition here?

Currently the schema only ensures values you set are correct.
It doesn't ensure values without defaults are provided. It wouldn't take much to convince me that it ought enforce that.

In this particular instance we want to make sure that every one of these items is present with the exception of the modified field that may not be there.
I guess our need is some way to make sure that all fields we wanted to be present are actually there, and we have the notion of an optional field, like the modification date field that may be missing but has to be a DateTime if it's there. If someone supplies a None for the modified time here I suspect this passes validation?
If there was a change that introduced that lets you enforce that the values have to be there it would be nice to have some way of marking that the values could be optional. If there's no enforcement then it would be nice to have some optional way of making sure that a field is present and valid.

So I'm experimenting with adding this, and my tests [oh wow, I have tests, don't I? :) ] show my original intention was that the error would raise if you accessed a no-default attribute without a value.
This is conflicting with the new approach.
From memory, I wanted the schema to act stand-alone, so you could instantiate a schema derivative, and then set its values.  I'm not sure of how much utility this still is.


imageWithBlocks fix. Rotate the original image to the right or it sho…

…ws upside down.

Maybe this is my own problem. When I use the library, I have to rotate the image left before pass to it.
let image2: UIImage = UIImage(CGImage: imageRef, scale: image.scale, orientation: UIImageOrientation.Left)

Otherwise, it cannot recognize properly.
I'm taking a photo with iphone vertically, so maybe that is the problem.
I only realized this after I created the pull request.

@sunshineo,  as I remember, the framework can rotate images if they contain the rotation info.
Do you do any image preprocessing, which removes rotation info from the image?

@ws233 No pre-processing. Straight out of the camera and into Tesseract. It did not work at first at all. Later I discovered that if I took the picture horizontally and pass to Tesseract it works. So since my app  will always took the picture vertically (My app does not have a horizontal mode), I programmatically rotated it left as shown above in the Swift code. And then I ran into this problem when I try to use imageWithBlocks. All the blocks are in the right place, but the image is upside down, unless I do that fix to rotate it right.

I just discovered that if the thresholded is set to true, this rotation is not needed. I'm going to close this one and create a new pull request.

The issue is that nevermind of the orientation of the device, the pixels in the image are always located in the same order. Only the imageOriantation tells how to draw the image on the screen.
imageWithBlocks doesn't consider the imageOriantation property of the image.
So you'd better fix the drawing code there.
In the case of the thresholding, the image is preprocessed for you automatically. And if I remember correctly, the preprocessing considers the imageRotation. Therefore you have a correct result after the preprocessing.

Hi @ws233 , could you help me understand your comments? The imageOriantation is a property on UIImage?
By "So you'd better fix the drawing code there.", where do you mean exactly? The imageWithBlocks function?
Thank you very much!

Yes it's. Twice :)

Thank you! I'm copying your comments to the newer pull request I created.

You may work in this PR if you wish


Fix iceberg-spark-runtime transitive dependencies.

This updates the iceberg-spark-runtime Jar to fix dependency leaks.
Now, compileOnly dependencies (and transitive dependencies from compileOnly) are shaded and relocated. Compile dependencies remain declared compile dependencies in the output artifact. The only compile dependencies are slf4j and guava. All other dependencies are shaded.

@rdsr, please review since you were interested in deps for --packages. This fixes transitive dependencies for iceberg-spark-runtime. We may want to add more declared dependencies.
@aokolnychyi, I think this should fix your Jackson dependency problem because it is no longer a transitive dependency of the runtime Jar.

Thanks @rdblue! I'll look at it today.

This does solve the problem with Jackson. Thanks, @rdblue.

I tested this out by publishing the jar locally and adding as a dependency in Gradle. It works fine and has no transitive dependencies
I'm just curious that we are overloading Gradle's compileOnly configuration to achieve this.  I think this will work fine with a Maven repo, but might cause issues with Ivy repositories [ We internally at LinkedIn have an Ivy based repository]. We will test this out internally and create tickets if required.

Thanks for the reviews, @aokolnychyi and @rdsr!


config error

Hello, when I try to play a song, it says:
An error  encountered: Error: play(music) encountered:
Could not find player config
Config is set so I don't know why it writes.

This is ytdl-core error. Maybe it's outdated.
Run npm i ytdl-core@latest to update it

This is ytdl-core error. Maybe it's outdated.
Run npm i ytdl-core@latest to update it

Omg thank you very mutch, my bad sorry. 👍

Omg thank you very mutch, my bad sorry. 👍


How to calculate translation from end-effector positions with Robomimic datasets

Hi！Thank you so much for sharing the source code. I am confused of calculating translation of the action.
For example, as shown in the code below, the last row [ 0.40681412 -0.14598518  0.21143397] minus the fourth row  [ 0.4095132  -0.13039334  0.22040763] of end-effector x-y-z positions does not match the fourth row [ 0.00740245 -0.39340851 -0.15288217] of actions considering the sign ([-, -, -] v.s. [+, -, -] ). In addition, the value of action is also not equal to the difference between two rows of end-effector positions multiplying some constant. So how should I calculate translation?
dp= './datasets/can_real/ph/image.hdf5'
f = h5py.File(dp, 'r')
xyz = f["data/demo_1/obs/ee_pose"][10:15,:3]
next_xyz = np.concatenate((xyz[1:,:],xyz[-1:,:]))
difference = next_xyz-xyz
actions = f["data/demo_1/actions"][10:15,:3]

'''
print(xyz)
[[ 0.42206772 -0.07756579  0.25226842]
 [ 0.41977403 -0.08709799  0.24656297]
 [ 0.41412277 -0.11014967  0.23220704]
 [ 0.4095132  -0.13039334  0.22040763]
 [ 0.40681412 -0.14598518  0.21143397]]
print(difference)
[[-0.00229368 -0.0095322  -0.00570545]
 [-0.00565126 -0.02305168 -0.01435593]
 [-0.00460957 -0.02024367 -0.01179941]
 [-0.00269908 -0.01559184 -0.00897366]
 [ 0.          0.          0.        ]]
print(actions)
[[-0.03366617 -0.44941242 -0.20676356]
 [-0.03480152 -0.48495732 -0.22616874]
 [-0.01148498 -0.42967414 -0.18070786]
 [ 0.00740245 -0.39340851 -0.15288217]
 [ 0.0180418  -0.36251448 -0.13617327]]
print(actions[0]/difference[0])
[14.6777652  47.14678873 36.23968019]
'''

The actions are delta pose actions - the way this works is identical to robosuite. Basically, the first 3 dimensions of the action correspond to a desired delta end effector displacement from the current end effector position. If you add this displacement to the current position, you get a desired target position -- this position is given to the OSC controller, which tries to reach it for the next control timestep. It may not necessarily reach that goal location though, so the end effector pose in the next timestep will in general be different from the commanded target position. You can read more about the logic for how delta pose actions get converted to control targets here, and more about the OSC controller here.


if identifier_field is set (not default to title), slug has to be also set

An entry can't be saved, if identifier_field is set, but not slug.
identifier_field should also change the default of slug
https://decapcms.org/docs/configuration-options/#slug:

a different field can be used via identifier_field

Also it can't anymore be saved, if no title-field exists when no identifier_field or slug is set.
I've had a collection set up this way, and it just saved the entries with a random id (chars and numbers).

also this has had to be functional at some time, as this problem appears for a collection I've used like this successfully for some time

In the previous versions of Sveltia CMS, if a slug could not be determined, it automatically used a random slug. I removed the fallback at some point because it might be confusing. I’ll fix the identifier_field issue and restore the fallback code.

The fix is in! Thanks for the report 💯

I wouldn't have a problem with there being no random fallback.
But either way it would be nice, if the error-message wouldn't just say "there was some error. try again later.", as this gives no info what caused it (hat to search for a while) and "trying later" does nothing ...

I’ll improve the error message 😅


Error message on pull related to race in content status

Sometimes an error is seen during pull related to requesting ingest data which has not yet been started. The content store should more gracefully handle this rather than returning an unknown error.
ERRO[0000] active check failed                           error="could not get status for filter \"\": stat /var/lib/containerd/io.containerd.content.v1.content/ingest/b3c88f7
59992d03f577169b9fd9b2ad1a5926c6be1f33d3e9cd261ade11ae14e/data: no such file or directory: unknown"

See https://asciinema.org/a/IJ1RtZmTfNLA4JxZ7K53idIOU

When the status is no longer available, it should check if the target content is now available.


Add docs to explain the internals

Is your feature request related to a problem? Please describe.
We should add some docs to explain the internals of Mars.

Contents

Structure

Code structure
How tasks are executed

Oscar actor system

Actor creation
Actor execution
Communication and routing
Batch methods

Service topics

Cluster
Lifecycle
Meta
Scheduling
Session
Storage
Subtask
Task
Web


feat(ngModel): pass the previous and current model to view change listeners

Technically, when ngChange is called, the model should always have been changed. What's your use case for this?

We are using this, to compare the old and new values of the form elements.
When autosaving we build a model from the valid values to be sent to the server, but want to keep the possibly invalid new value, for the user to still be editable, even if they leave the invalid field.


Plugin to get host utilization

Hi,
How can we incorporate a plugin to get the host memory utilization? It's not for auditing but we just need to monitor all hosts which are, say, more than 75% utilized.
Thx,
LP

See the Plugin structure wiki page.
Then just add your logic, something like $vmh | Where {($_.MemoryUsageMB/$_.MemoryTotalMB) -gt 0.75} | Select Name, @{"Name"="Memory usage %";Expression={($_.MemoryUsageMB/$_.MemoryTotalMB)*100}}

Thanks, but I am not sure how it fits within the plugin. Is something like below looks fine?
Start of Settings
End of Settings
Setup plugin-specific language table
$pLang = DATA {
ConvertFrom-StringData @'
pluginActivity = Checking overcommit state for hosts
'@
}
Override the defaults (en) if language file exists in lang driectory
Import-LocalizedData -BaseDirectory ($ScriptPath + "\lang") -BindingVariable pLang -ErrorAction SilentlyContinue
$ResUtil = @()
$VMHCount = $VMH | Measure
Foreach ($VMHost in $VMH)
{
$ResUtil = $VMH | Where {($.MemoryUsageMB/$.MemoryTotalMB) -gt 0.75} | Select Name, @{"Name"="Memory usage %";Expression={($_.MemoryUsageMB/
$_.MemoryTotalMB)*100}}
}
generate your report content here. Simple placeholder hashtable for the sake of example
@{"Plugin"="Hosts Resource utilization"}
$Title = "Resource utilization for the hosts"
$Header =  "Hosts above the 75% threshold : $(@($ResUtil).count)"
$Comments = "Over utilized hosts cause performance issues in the environment which may result in deteriorating performance for the underlying virtual machines. Please analyse and report accordingly"
$Display = "Table"
$Author = "LP"
$PluginVersion = 1.0
$PluginCategory = "vSphere"

Close. You have a heap of redundant code though.
  # Start of Settings
  # End of Settings

  $ResUtil = $VMH | Where {($_.MemoryUsageMB/$_.MemoryTotalMB) -gt 0.75} | Select Name, @{"Name"="Memory usage %";Expression={($_.MemoryUsageMB/$_.MemoryTotalMB)*100}}
  $ResUtil

  $Title = "Resource utilization for the hosts"
  $Header =  "Hosts above the 75% threshold : $(@($ResUtil).count)"
  $Comments = "Over utilized hosts cause performance issues in the environment which may result in deteriorating performance for the underlying virtual machines. Please analyse and report accordingly"
  $Display = "Table"
  $Author = "LP"
  $PluginVersion = 1.0
  $PluginCategory = "vSphere"

Thanks, works as expected, I will build the same for CPU. Just a couple of questions, since naive on this:

The value for the percentage is coming as 88.07092511357235465251273786. Can we round this to two digits after decimal?
Where is it picking $VMH value from? I do not see the variable in the script neither it's there in Global variables.

Got the reply for point 2, found it in vcenter connection plugin..thx!

Sure, just use the standard round() function.
 $ResUtil = $VMH | Where {($_.MemoryUsageMB/$_.MemoryTotalMB) -gt 0.75} | Select Name, @{"Name"="Memory usage %";Expression={[Math]::Round(($_.MemoryUsageMB/$_.MemoryTotalMB)*100,2)}}

yep, that's what was looking for..another doubt, i wanted to exclude a few datastores which I did during the -config wizard (comma separated) but they still appear in the list..can you let me know where can i manually check the entry and modify it in the script.

There isn't any global filtering of datastores, so you probably just added them to the specific plugin.
However, if you do want to remove certain datastores completely, you can filter them out in pluings\00 Initialize/00 Connection Plugin for vCenter.ps1

Presumably, I wasn't clear the last time. What I meant was there was a section where we can explicitly filter out multiple datastores for the below two sections. This does not seem to be present in Initialize plugin. Also when specifying multiple datastores over the command line via vcheck -config the datastores are still not excluded
Datastore Information
Set the warning threshold for Datastore % Free Space [25]: 20
Do not report on any Datastores that are defined here (Datastore Free Space Plugin) [local]:
Datastore OverAllocation
Datastore OverAllocation % [50]:
Exclude these datastores from report [datastore1,datastore2,datastore3,datastore4]:

Yeah sorry, I wasn't sure exactly what you were asking :)
The Datastore Information filter works for me, as we have all our local datastores with -local appended to the server name which do not show in our report.
Bear in mind these filters generally use the -notmatch operator, so you want to use regular expressions. So for example datastore[1-4] to exclude datastores named datastore1,datastore2,datastore3,datastore4

For some reason it still doesn't work for me, anyplace where I can hard code it?

Hi, any updates on this? I actually tried doing it for several other sections including vmtools issues, no vm tools, datastores over certain% etc.. but it works fine till there is only 1 value. When I have multiple entries separated by comma it simply doesn't exclude them.

Please ignore, got the solution!


Constructing an AnchorPair outside of Anchorage will be impossible

To extend Anchorage in an application-specific way, one might create convenience methods to return an AnchorPair, make new AnchorGroupProviding conformers, or extend AnchorGroupProviding to specify new groups. These approaches start with being able to construct an AnchorPair outside of the Anchorage module.
I'm not sure this qualifies as the most "practical" example, but here is my real-world use case. I've implemented a vertical columns-based layout system in order to match the way my designer works, to support handling devices with different widths by scaling the UI rather than implementing (and designing) a responsive design or doing nothing.
So for example, I'd like to be express a layout like
titleLabel.horizontalAnchors == AnchorPair(first: columnOne.leadingAnchor, second: columnTwo.trailingAnchor)

That's pretty raw, and you might imagine a convenience method to make this
titleLabel.horizontalAnchors == anchorsSpanningColumns(.one, .two)

The AnchorPair initializer is marked internal, so I can't use it directly. I've hobbled by with a hack of replicating the internal initializer with slightly different parameter labels, intending to bring this very question in front of this group. With Swift 4.1 and SE-0189, this is now explicitly deprecated.
What is the reasoning behind disallowing AnchorPair construction outside of Anchorage? Is there a downside or complication to making its initializer public that I'm not seeing?
Obviously, I have alternatives. I can always use Anchorage as the implementation for layout convenience methods, but integration with the DSL is so much nicer than mixing it with method calls.
I'd love to hear what you think. In the event that there aren't objections – maybe this initializer was just very conservatively marked internal – I'll post a PR. In that case, I'd have some organizational questions around the Internal.swift file.

My guess is that it wasn't marked as public because we didn't see a use case, but yours is definitely intriguing. @jvisenti should make the call, but this seems like a reasonable request to me.

@rlwimi @ZevEisenberg I believe AnchorPair is essentially an internal type because exposing the initializer publicly could allow for unsafe usage of the API. Consider this function:
func == <T, U>(lhs: AnchorPair<T, U>, rhs: AnchorPair<T, U>) -> ConstraintPair

If you were to construct two AnchorPair<NSLayoutAnchor, NSLayoutAnchor>, you could mix NSLayoutXAxisAnchor and NSLayoutXAxisAnchor however you wanted, which would result in a crash if you attempted to use this operator on mismatched types, i.e. trying to constraint an XAxisAnchor to a YAxisAnchor. The current API forbids this by construction.
That said, the existence of a valid use case may outweigh the pretty minor risk that comes with it. Happy to take a look at a PR if you have something put together already, and see how it could fit in.

I was just reading through some code that I wrote in a project, and found what I believe is a valid use case:
var safeCenterAnchors: AnchorPair<NSLayoutXAxisAnchor, NSLayoutYAxisAnchor> {
    if #available(iOS 11.0, *) {
        return AnchorPair(first: safeAreaLayoutGuide.centerXAnchor, second: safeAreaLayoutGuide.centerYAnchor)
    }
    else {
        return centerAnchors
    }
}

Thanks, everyone! Much appreciated.


[move-stdlib][rfc] Standalone module for capabilities.

This adds a proposal for a generic capability module to the nursery.  This is similar designed like the Vault, but decouples the notion of capability from what it does protected (in the Vault case storage). If we agree on this design, we can refactor the Vault to be based on the new Capability module.
To read the proposal, it might be best to check out the generated documentation.
Motivation
Access control.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
NA
Related PRs
NA

Nice! I think this will simplify the Vault module which has the redundant code to manage different capabilities such as "read", "modify", delegate" and "transfer". Independent from Vault, the Capability module could be used to replace some storable capabilities in the Diem Framework. This would help the access control reasoning because it make it explicit who "has permission" (who are the cap owner and the cap delegates).

Really like this approach!
I'm wondering whether eliminating copy here would make this library a bit more powerful:

If a sensitive operation wants "exactly once enforcement" (i.e., a linear capability), it can ask for a Cap<T>
If a sensitive operation can be safely invoked an arbitrary number of times, it can ask for a &Cap<T> instead
Another implementation strategy could be adding is_linear: bool to CapState and defining a separate LinearCap struct.

Linear capabilities are in fact an interesting feature. However, to make this work the authorized function cannot take a &Cap but would need to take a Cap to actually consume it. So I think copyable caps are not subsumed by linear caps. These are two different concepts. My guts feeling is that copyable caps are the more common use case. We can add LinearCap if we come along a use case for it.

An related example: An organization gives its members the right to vote (once each!) on proposals that come up from time to time. Alice wants to delegate her right to vote on proposal X (and only proposal X) to Bob.
Perhaps this could be implemented via the is_linear scheme above + removing an address from the delegate list after it claims a LinearCap?

We can add LinearCap if we come along a use case for it.

These are the two use-cases I had in mind. I think the second one is especially important to support, since this sort of thing is very common in on-chain governance.

A related example: An organization gives its members the right to vote (once each!) on proposals that come up from time to time. Alice wants to delegate her right to vote on proposal X (and only proposal X) to Bob.

We can add LinearCap if we come along a use case for it.

These are the two use-cases I had in mind. I think the second one is especially important to support, since this sort of thing is very common in on-chain governance.

A related example: An organization gives its members the right to vote (once each!) on proposals that come up from time to time. Alice wants to delegate her right to vote on proposal X (and only proposal X) to Bob.

Got it. I think we can iterate on this one.

Got it. I think we can iterate on this one.

Looking at https://github.com/diem/diem/pull/9214 and thinking about how it could use Capability + leverage it for more advanced features like vote delegation could be interesting.

I forgot the decision made but this is still in nursery, shall we move it to sources in this PR or in a follow-up?

Lets submit and move then.
/land


Remove gdrdrv-dkms dependency enforcement from gdrcopy main package

Problem: See issue #188.
This PR:

removes gdrdrv-dkms dependency enforcement from gdrcopy.deb.
removes gdrcopy-kmod dependency enforcement from gdrcopy.rpm.
adds libsubunit0 dependency enforcement to gdrcopy.deb. sanity requires it but the dependency is not declared in the deb package. For rpm, this dependency is auto-discovered.

With this PR, now the package metadata looks like
$ dpkg-deb -I gdrcopy_2.2-1_amd64.deb
 new Debian package, version 2.0.
 size 54016 bytes: control archive=900 bytes.
     388 bytes,    12 lines      control
     494 bytes,     8 lines      md5sums
      20 bytes,     1 lines      shlibs
      74 bytes,     2 lines      triggers
 Package: gdrcopy
 Version: 2.2-1
 Architecture: amd64
 Maintainer: Davide Rossetti<EMAIL_ADDRESS> Installed-Size: 176
 Depends: libsubunit0, libc6 (>= 2.14)
 Section: libs
 Priority: optional
 Multi-Arch: same
 Homepage: https://github.com/NVIDIA/gdrcopy
 Description: A low-latency GPU memory copy library
  A low-latency GPU memory copy library based on NVIDIA GPUDirect RDMA technology.

$ rpm -qp --requires gdrcopy-2.2-1.x86_64.rpm
check
libc.so.6()(64bit)
libc.so.6(GLIBC_2.11)(64bit)
libc.so.6(GLIBC_2.14)(64bit)
libc.so.6(GLIBC_2.2.5)(64bit)
libc.so.6(GLIBC_2.3)(64bit)
libc.so.6(GLIBC_2.3.4)(64bit)
libc.so.6(GLIBC_2.4)(64bit)
libgdrapi.so.2()(64bit)
libm.so.6()(64bit)
libm.so.6(GLIBC_2.2.5)(64bit)
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)
libpthread.so.0(GLIBC_2.3.3)(64bit)
librt.so.1()(64bit)
librt.so.1(GLIBC_2.2.5)(64bit)
librt.so.1(GLIBC_2.3.3)(64bit)
libstdc++.so.6()(64bit)
libstdc++.so.6(GLIBCXX_3.4)(64bit)
libstdc++.so.6(GLIBCXX_3.4.11)(64bit)
libstdc++.so.6(GLIBCXX_3.4.9)(64bit)
libsubunit.so.0()(64bit)
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rtld(GNU_HASH)

Is the idea for adding libsubunit dependency just that it's presently a silent requirement, and your change for https://github.com/NVIDIA/gdrcopy/issues/187 will split it back out again?
I realized in https://github.com/NVIDIA/gdrcopy/issues/187 that it might be better for backward consistency to have the gdrcopy package just become a metapackage and not drop the gdrdrv-dkms dependency from the metapackage; any thoughts on that?  If you think it's best to drop it and are sure this won't pose problems for those installing one and assuming it will install the other for them because it always did, then ok, LGTM.  But I do think you should consider that extra layer of indirection.

Yes, libsubunit is a silent requirement. So, I add it here to make thing correct.
In the current code (not this PR), we don't enforce libcuda as a dependency of gdrcopy package even though the tests need it. This is so that users can install CUDA via run file and gdrcopy from package. We can relax the dependency checking with gdrdrv as well. Users will get notify if they try to use libgdrapi without gdrdrv anyway.
PR for #187 will require more test. We have to ensure that users can upgrade from real package in the current version to metapackage in the next version. Or maybe we will use a different name for the metapackage to avoid compatibility issues.

In the current code (not this PR), we don't enforce libcuda as a dependency of gdrcopy package even though the tests need it.

This is fine with me and all; I was just thinking of upgrade compatibility for existing users.


IOTSSL-1879 buffer overflow in server key exchange parsing

Description
This PR prevents two buffer overflow errors from happening in the ssl_parse_server_key_exchange function.
Status
READY
Requires Backporting
YES
This fix should most probably be backported to all supported branches as the vulnerabilities aren't version speciffic.
Migrations
NO
Additional comments
None
Todos

[ ] Tests maybe?

Steps to test or reproduce
None prepared

I have just noticed the verification build failure. I will investigate it.

CI passed except for the FreeBSD timing self-test which is a known issue.

Can you please add a ChangeLog entry, and make backports to 2.1 and 2.7?

retest

retest


publish docker images to  ecr public gallery

Since we've added multi arch support, we started getting timeout errors while publishing container images to Amazon ECR Public Gallery.
goreleaser_publish_to_ecr
To unblock the release workflow, we temporarily stopped publishing to ECR, but we must bring this functionality back. One idea is to create a dedicated workflow that's scheduled after the main release workflow that will pull images from DockerHub, tag them, and publish them to ECR. We should also have an option to dispatch this workflow manually with Git revision as param.

relate to #250


Add Internal Benchmark tooling

To facilitate tracking of benchmarks internal to the job runs the MAAP team is recommending Scalene for python code.

[x] Wrap existing algorithm in Scalene

I implemented this on a new branch: https://github.com/MAAP-Project/get-dem/tree/scalene
I deployed a new version of the algorithm as GET-DEM:scalene and ran it with the following inputs:

bbox: -156.0 18.8 -154.7 20.3
compute: true
scalene_args: (left empty to use defaults)

I copied the outputs to /projects/shared-buckets/dschuck/get-dem/
I also tested that I can run it successfully from the ADE.

Closed too soon. Need PR approved first.

@nemo794, I made adjustments to produce a JSON profile by default, rather than HTML. (I'm continuing to make adjustments to the scalene branch, which we can pick apart for submitting incremental PRs once #2 lands.)
I also modified the notebook in a manner that should aid our eventual effort to collate/aggregate profiling metrics once we're ready to kick off a slew of jobs. The primary change was to associate names with the sample bboxes to allow us to use the names as part of the tags for the jobs we submit, because a job's tag is used to create a directory along the path structure to the job's output directory.
More specifically, in the notebook, a job's tag (the identifier argument to submitJob, but labeled "Tag" in the Jobs UI) is structured as {bbox_name}__compute__{queue} when compute is True, or {bbox_name}__no-compute__{queue} when compute is False.
As an example, I ran "Italy", and I copied the output to /projects/shared-buckets/dschuck/get-dem/scalene/Italy__compute__maap-dps-worker-32vcpu-64gb/2024/03/08/07/57/58/466407/ so you can see the results.
Note that the path from get-dem onward is the full structure I copied from /projects/my-private-bucket/dps_output. Also note that the directory Italy__compute__maap-dps-worker-32vcpu-64gb is the job tag.
The information we want to extract from profile.json is within the files/<python filename>/functions arrays (only 1 python file in our case).
Unfortunately, Scalene does not produce an HTML file corresponding to the JSON file, so it's not immediately and completely clear which stats from the JSON file are the ones that are nicely rendered in the HTML file, so my next step is to run another job with the same inputs, but passing arguments to scalene to produce an HTML file. Then I'll compare what I see in the browser for the HTML profile against the stats in the JSON file to match things up. (Obviously, the numbers will be different, but should be close enough for me to decipher things.)

@nemo794, it looks like these are the 2 primary values we want to pull out of the JSON profile:

elapsed_time_sec (time)
max_footprint_mb (space)

What other information might be needed for the large spreadsheet you presented in the recent meeting about gathering get-dem performance figures from both NASA and ESA jobs?

@nemo794, for comparison between the 2 Italy runs (one producing profile.json and the other profile.html), I've copied the second run to /projects/shared-buckets/dschuck/get-dem/scalene/Italy__compute__maap-dps-worker-32vcpu-64gb/2024/03/08/10/03/45/609726/

Hi @chuckwondo , This is a great progress! Thank you! We discussed this offline, but for the record, I really like how you modularized each algorithm step into its own function, in order to wrap each step individually in the scalene wrapper.
The DPS outputs are really helpful! A few thoughts:

Is Scalene able to generate both html and JSON outputs at the same time? JSON has higher priority because of the need to parse algorithmically, but as a human the html is much friendlier for spot checking.
max_footprint_mb (space)

Good call -- this overview metric is helpful. If the number of cores remains constant, I don't see any reason this value should change from run to run... but it will help to select correct instance types, see impacts from changing the number of cores, and also as a sanity check that the algorithm isn't doing anything fishy. For now, I don't think we need a finer level of granularity.

elapsed_time_sec (time)

This appears to be the total time of the algorithm. This is definitely a good metric to have; the MAAP platforms' DPS metrics should actually capture the same metric, so now we'll be able to compare that the platforms are correctly capturing that info. Nice!
For benchmarking, we'll need a finer level of granularity. It'd be great to have timings for each step. Looking at the JSON, I'm only seeing the percentages of time and total time, but maybe I'm missing something? Here are a few screenshots to help clarify:

After this, hopefully there's a way to get even finer granularity by digging into the sardem step via calling it directly from Python, but one step at a time. :)

Hi @nemo794, thanks for the great annotated screenshots. They are very helpful.
This perhaps got buried in one of my earlier comments:

Unfortunately, Scalene does not produce an HTML file corresponding to the JSON file

That is, Scalene won't produce both formats for a given run. It's one or the other, sadly.
Regarding, "finer granularity by digging into the sardem step via calling it directly from Python," I forgot to mention that I've already done that. The following shows the replacement of the system call:

That's within the get_dem function, so now we are getting the mem+cpu+io information that we were not seeing when we were using the system call.
Regarding the absolute time values vs. the percentages, it seems strange that the profile does not contain the individual elapsed time values. Unfortunately, I think that leaves us with having to do the extra calculations ourselves, but at least we have the necessary numbers to do so.

Fixed by #6.

@nemo794 and @arthurduf, I did a bit more digging into the profiling metrics captured by Scalene.
Unfortunately, I think Scalene's metrics are not providing the information we're looking for, or at least I haven't deciphered it yet, if it's there. I'm going to dig a bit deeper.
The problem is this: neither the percentages shown in the previous screenshot, nor the percentages shown in the screenshot below are the percentages that allow us to compute the percentage of elapsed time taken by each function.
To illustrate, the timings printed to _stderr.txt are as follows:
2024-03-08 07:33:55 [get_dem] [INFO] get_dem: 628.8 seconds
2024-03-08 07:33:57 [get_dem] [INFO] read_dem_as_array: 2.5 seconds
2024-03-08 07:57:57 [get_dem] [INFO] do_computations: 1440.1 seconds
2024-03-08 07:57:57 [get_dem] [INFO] main: 2071.4 seconds

The last number, 2071.4, closely aligns with the value of ~2072 for elapsed_time_sec in profile.json. So far, so good. An elapsed time of 628.8 seconds for get_dem is ~30.3% of 2072, and an elapsed time of 1440.1 seconds for do_computations is ~69.5% of 2072, giving us 99.8% of the total, with read_dem_as_array accounting for the remainder.
Unfortunately, neither the percentages circled in the preceding diagram, nor those highlighted below align with these percentages.
I believe the reasons are as follows:

The percentages circled in the previous diagram are memory consumption percentages, so it makes sense that they do not correspond to runtime percentages
The percentages below are CPU time percentages, which is related to elapsed time, but not equal to elapsed time. The CPU percentages do not account for non-CPU (i.e., I/O) time.

Therefore, I'm going to do a bit more digging into the other parts of the profiling metrics to see if the I/O time is accounted for somewhere.
If the I/O time cannot be accounted for within the profile metrics, where does that leave us?
I'm thinking that we could use Scalene's profile simply to obtain max memory usage, but then rely on the output from our own code for capturing the runtime metrics. If that's the case, I recommend we tweak our output to write the runtime values to a separate JSON file because we don't want to attempt to parse the values out of the _stderr.txt file.

I'm thinking that we could use Scalene's profile simply to obtain max memory usage
--> Agreed
we don't want to attempt to parse the values out of the _stderr.txt file.
And also that file does not exists on ESA DPS.

Per discussions w/ @nemo794, I have created a branch named gather-profile-metrics for working on gathering metrics from a batch of jobs.
Here's what I've done so far on that branch:

Modified get_dem.py to write function elapsed times to a file named elapsed.json (in addition to logging), since Scalene, unfortunately, does not appear to capture this specific metric for each function.
Added simplify_profile.py to extract max_footprint_mb and elapsed_time_sec from profile.json and combine them with the elapsed function times from elapsed.json into a file named simple_profile.json.
Added aggregate_profiles.py to aggregate (specifically compute the mean) values from a list of paths to simple_profile.json files. It currently prints the mean values to stdout.
Added esa/README.md and nasa/README.md with instructions for running the algorithm locally, simulating the ESA env and the NASA env, respectively.

If you want to test things out, you can run things locally by following the instructions in esa/README.md and nasa/README.md. After successfully following the instructions in both of those files, you should end up with the following files in your output directory:
$ tree --filesfirst output/
output/
├── dem.tif
├── elapsed.json
├── profile.json
├── simple_profile.json
└── esa
    ├── dem.tif
    ├── elapsed.json
    ├── profile.json
    └── simple_profile.json

At this point, you can compute the means by running the following:
./aggregate_profiles.py output/simple_profile.json output/esa/simple_profile.json

This will output the mean values to stdout, which should give you something like the following:
$ ./aggregate_profiles.py output/simple_profile.json output/esa/simple_profile.json
{
  "max_footprint_mb":<PHONE_NUMBER>981628,
  "total_elapsed_sec": 31.518759727478027,
  "download_and_stitch_sec": 19.908729314804077,
  "compute_sec": 4.638479709625244
}

@nemo794, okay to consider this closed now?


Iceberg partition use, configuration, and limits

This morning I tried changing some table partitions from "by year" (we have at most 30 years of data) to "by respondent_id" (there are over 500 distinct respondent_id values), resulting in this error message:
TrinoUserError: TrinoUserError(type=USER_ERROR, name=ICEBERG_TOO_MANY_OPEN_PARTITIONS, message="Exceeded limit of 100 open writers for partitions", query_id=20220329_122153_01603_bm8rx)
Technically this should be just a configuration parameter that can be tweaked, but it's likely more involved than that: https://github.com/prestodb/presto/issues/9870
Namely, the 100-writer limit is likely the result of mathing a number of configuration parameters which, if all changed coherently together, can have the effect of properly supporting partitions with more than a few dozen different values.
But this raises a different problem: should partitioning be such a fragile operation that a whole set of configs need to be changed just so a user can use an index column of their chossing?
Just as Datasette facets can be tuned but within reasonable limits well explained by the Datasette documentation, we should document for users how to tell whether or when their partition choice should be changed to something else.

Poking around the data directory, I noticed that I selected a partition value that had more than 100 distinct values, but never for each batch I wrote via the SQL query.  So one potential solution is to keep my batch_size below the max number of open writers.

As discussed with Michael, will use the sample https://github.com/os-climate/DERA-ingest-pipeline/blob/iceberg/notebooks/DERA-ingest.ipynb to work on partitioning and optimization of ingest.

At present that file is using only a single quarter's worth of data, to keep its overall runtime reasonable.  As we get performance up, we can slurp all available data by reducing the restrictions on what it slurps:
One quarter: objects=source_bucket.objects.filter(Prefix='SEC-DERA/2017q4')
One year: objects=source_bucket.objects.filter(Prefix='SEC-DERA/2018')
Years 2020 and later: objects=source_bucket.objects.filter(Prefix='SEC-DERA/202')
All years 2000 and later (really 2017-present): objects=source_bucket.objects.filter(Prefix='SEC-DERA/20')


Unclear error message for target test suite: KeyError: 'test_array_data'

In working on the target-duckdb port, ran into this error message output which was not clear.
Several tests fail with the same error:
KeyError: 'test_array_data'

@aaronsteers did you install the pytest plugin in conftest.py?
https://sdk.meltano.com/en/latest/testing.html#example-usage


Django logging not working when djdt is enabled

I recently noticed that django default logging is not working when I enable djdt.
django-debug-toolbar==4.1.0
Django==4.2.3
Is it possible to have SQL pannel and logging for sql queries work together?
Without djdt

With djdt

Hi @forrestkouakou, can you create a reproducible environment for this?

I've just dug deep into django's guts to figure out why our console logging isn't working either. It looks like it's caused by the DjDTCursorWrapper (debug_toolbar/panels/sql/tracking:wrap_cursor()) overriding the default CursorDebugWrapper that django installs based on settings.DEBUG or an option set somehow to force it. When wrap_cursor() is called by djdt, it's returning an instance of NormalCursorWrapper which eventually extends CursorWrapper (not CursorDebugWrapper where the logging functions are).
My guess is that the resolution to this is that DjDTCursorWrapper needs to be mindful of what cursor django has installed, and if it's installed CursorDebugWrapper instead of CursorWrapper then it should inherit that instead. Something can probably modify DjDTCursorWrapper.__bases__ or use a function in the class inheritance instead in order to dynamically inherit the correct cursor wrapper class.

@tim-schilling based on this would you support a revert of e7575e87dc9e2d2560b87d6fd5a123b9398cbd34?

@dannosaur thank you for that investigation! That's super helpful. Do you have a reproducible use case?
@living180 I think I'm on board for a revert, though I'd prefer to roll forward a fix.

@tim-schilling yes, I can reproduce the issue in my project.

@dannosaur Sorry, I meant can you create a way for me to reproduce the error so that I can investigate and attempt a fix?

@tim-schilling lol, sorry! I'll try and put something together quickly tomorrow that reproduces the issue. I don't think it's a complex setup though - django 4.2, any version of psycopg (2 or 3), and djdt 4.1. Leave it with me, I'll post back tomorrow with a git repo.

No worries. I can set up postgres on my end. Having the ability to pull a repo, set up a database and reproduce the problem makes this much, much easier to investigate. Otherwise I'm attempting random things to try to reproduce the behavior. I appreciate your work on this so far, thank you!

@tim-schilling here you go: https://github.com/dannosaur/djdttest
The django admin is installed with a user. Login dan/password. Observe the terminal running runserver - there should be some queries printed directed at auth_user table. In settings.py I put a DISABLE_PANELS section with a commented out item. If you uncomment that line (158) and reload the admin user list page, you'll see the SQL query in the console.
LMK if you need anything else, happy to help!

Thank you @dannosaur, I was able to reproduce this.

Can folks test their project against https://github.com/jazzband/django-debug-toolbar/pull/1820 to confirm that works?

@tim-schilling confirmed working now in my project.


[Feat]: Playlist, Category, dislike, remove from watch later

This PR includes the major features of the app, these are

Playlist,
Category,
Dislike,
Watch Later
Watch History
Remove from watch later

Thanks everyone, Merging now.


Scoped-slot not acknowledged in nested template tag

Version
2.5.16
Reproduction link
https://jsfiddle.net/hirokiosame/38nzxq6c/2/
Steps to reproduce
Just execute the JSfiddle. You'll notice the scoped-slot isn't being rendered.
What is expected?
For the scoped-slot to be rendered.
What is actually happening?
The scoped-slot is not in this.$scopedSlots and not being rendered.

The minimal reproduction is a little unpractical because who would unnecessarily nest content in an attribute-less template tag. This JSfiddle is more reflective of the real use-case we're dealing with: https://jsfiddle.net/hirokiosame/38nzxq6c/

Marked as both bug and improvement because I'm not sure it was intended to work so it could also be an improvement

Not sure if this is the same issue, but adding two elements with slot-scope in the default slot also causes the first one to be ignored: https://codepen.io/binary-koan/pen/BPyyKW
It wouldn't surprise me to learn that this is intentional, but it was odd to have <parent><child /><child /></parent> work, but stop working when changed to <parent><child slot-scope /><child slot-scope /></parent> with no warning message.

Closing as wontfix. Slot content must be at top level inside the component tag.


Refactor side-nav to have disabled save button

Allows the ability to set the Save button within the side-nav as disabled
TP: https://rockabox.tpondemand.com/entity/9824

waiting on changelog