Integrate Cancer Biomarker Database

@ckongEbi commented on Thu Dec 14 2017
The idea is to integrate data from https://cancergenomeinterpreter.org/biomarkers into the target profile page.
Tasks (to be expanded) :

[x] Start Specifications document
[x] Formulate Why and How the data should be integrated into the OT platform
[x] Fill in info on data accessibility: license, update frequency, access
[x] Replicate entries that have more than one drug (~80 entries), cancer type (~80 entries) or reference (> 100 entries) listed.
[x] Construct links for the ~400 database entries that cite a conference abstract or FDA/NCCN guidelines.
[x] Map gene names
[ ] Map drug names (have compound names, 'chemotherapy' appears a few times)
[x] Map diseases/cancer types (143 entries list 'Any cancer type’ as disease, 90 different diseases)
[x] Look at the data_pipeline, gene merging step --gen , the plugin system
[ ] Enrich/Map the initial dataset with Ensembl ID (biomarker) , Chembl ID (drug), Disease ID (Tumor type)
[ ] Check over data in Open Targets frontend
[x] Format individual_mutation field from FGFR3:V555M to FGFR3 (V555M)

New ticket will be raised for future updates of cancer biomarker data.


system-tests: create sets of standard address space tests in marathon test suite

https://github.com/EnMasseProject/enmasse/pull/1108


Add CRUD commands for all APIs' resources and the their throughput

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

[x] I adhere to the Command Guidelines.

This is a very large change. A command review is required.

Hi, how should I request a command review?
Thanks

@bizarreWizard please email @yugangw-msft (yugangw AT microsoft.com) to get a command review scheduled.

@bizarreWizard could you rebase this pr / resolve conflicts?

@bizarreWizard You should modify history.rst to update release notes for next release.


GCC Debugger doesn't work after update

VSCode Version:
Version: 1.48.2
Commit: a0479759d6e9ea56afa657e454193f72aef85bd0
Date: 2020-08-25T10:56:10.074Z
Electron: 7.3.2
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS Version: Linux x64 4.15.0-112-generic

Steps to Reproduce:

Run gcc debugger in attach mode.

Does this issue occur when all extensions are disabled?: Yes

Team, it needed a system restart to solve the issue.


a1 no longer reporting time since last block

it breaks the way we do it so people probably shouldnt mine it for now.,
edit: It sounds like he is just having technical issues in the forums.. he is talking about having problems with the reporting of lifetime shares which is different but if he is having problems there I wouldnt be surprised if this is just a technical issue. Any ways dont mine there until it is fixed.

He hasn't found a block yet, so I agree with you. Once a block is found, the estimated share count will be more stable.

please close

For reference the pool is down and we deleted it.


Modify: Fixed some minor syntax errors.

Thank you for making a good library!
I found some syntax errors in README.md and fixed them.

Thank you!


Unable to install platform on Symfony 2.8 Standard Edition

While trying to install the platform version ~1.10 in a project based on Symfony 2.8 Standard Edition an error occurs. The only version tagged on packagist for SensioDistributionBundle is 5.0, which is required in the composer.json of Symfony.

Your requirements could not be resolved to an installable set of packages.
Problem 1
- oro/platform 1.10.4 requires sensio/distribution-bundle 4.0.* -> satisfiable by sensio/distribution-bundle[4.0.x-dev, v4.0.0, v4.0.1, v4.0.10, v4.0.2, v4.0.3, v4.0.4, v4.0.5, v4.0.6, v4.0.7, v4.0.8, v4.0.9] but these conflict with your requirements or minimum-stability.
- oro/platform 1.10.3 requires sensio/distribution-bundle 4.0.* -> satisfiable by sensio/distribution-bundle[4.0.x-dev, v4.0.0, v4.0.1, v4.0.10, v4.0.2, v4.0.3, v4.0.4, v4.0.5, v4.0.6, v4.0.7, v4.0.8, v4.0.9] but these conflict with your requirements or minimum-stability.
- oro/platform 1.10.2 requires sensio/distribution-bundle 4.0.* -> satisfiable by sensio/distribution-bundle[4.0.x-dev, v4.0.0, v4.0.1, v4.0.10, v4.0.2, v4.0.3, v4.0.4, v4.0.5, v4.0.6, v4.0.7, v4.0.8, v4.0.9] but these conflict with your requirements or minimum-stability.
- oro/platform 1.10.1 requires sensio/distribution-bundle 4.0.* -> satisfiable by sensio/distribution-bundle[4.0.x-dev, v4.0.0, v4.0.1, v4.0.10, v4.0.2, v4.0.3, v4.0.4, v4.0.5, v4.0.6, v4.0.7, v4.0.8, v4.0.9] but these conflict with your requirements or minimum-stability.
- oro/platform 1.10.0 requires sensio/distribution-bundle 4.0.* -> satisfiable by sensio/distribution-bundle[4.0.x-dev, v4.0.0, v4.0.1, v4.0.10, v4.0.2, v4.0.3, v4.0.4, v4.0.5, v4.0.6, v4.0.7, v4.0.8, v4.0.9] but these conflict with your requirements or minimum-stability.
- Installation request for oro/platform ~1.10 -> satisfiable by oro/platform[1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4].

@ste93cry what is the minimum-stability defined in your application's composer.json?

There is no minimum-stability specified. You can find the composer.json file here

Could we relax a bit the composer dependencies? Almost all constraints declared in the composer.json of the platform are specific versions of a library, however this means that users trying to use the platform standalone are obliged to use the versions you defined, nothig older nothing newer, even if they are compatible. I tried running tests with a few dependencies updated on Windows, and the results for me are great: there are a few skipped tests 19 failed tests. Almost all of the failed tests comes from the Layout component and bundle), but nothing that seems related to the updated versions of the library because tests were failing even before (at least on my PC). Also, most of the failing tests is just because of directory separators (/ vs \)

@ste93cry we can update dependencies by request, but our experience of using .* style versions wasn't good as even patch version releases of some components were braking application. Feel free to submit a PR with updated package dependencies if you have issues with versions defined in composer.json.

I would be happy to submit a PR, but this would solve the issue only temporarily. I perfectly understand that given the size of the platform, having relaxed constraints can cause breaks but libraries should follow the semver conventions, so BC breaks should not be possible if the major version is not changed. A lot of other projects use such related constraints, and I don't think they have much problems. Also, even if I update the dependencies to their latest version and fix again that exact version as a dependency, maybe tomorrow a new version of that library that fixes some bugs is released and we cannot use it without updating again the composer.json. Also, if you continue to use fixed constraints then it can take a long time to update them again, meaning that some dependencies are a lot of versions old (take for example KnpMenu, which is at 2.0.0-alpha2 while there is 2.2. Now, this is a case where upgrading breaks the platform, but you're using a dev version (which is even old)

"twig/twig": "~1.26" is too wide, so in patch version update of platform might be minor vendor update and that is semver violation, see https://github.com/orocrm/crm/issues/210 for more details
cc @sbani

I'm not really sure I fully understand the issue about the semver violation. I'm a newbie on the topic, so please be patient with me, but according to semver  the version number is defined as:

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards-compatible manner, and
PATCH version when you make backwards-compatible bug fixes.

So, even if ~1.26 could install a new minor vendor update, this shouldn't break the platform is that library is following semver too because the only thing that it could so is add new functionality in a backwards-compatible manner. Anyway, maybe we can restrict the constraints to something like ~1.26.x? The only thing I'm trying to discourage if having fixed constraints that makes impossible to end-users to use their own vendor constraints, even if they know they are compatible with yours

It might. Lets check case with hwi/oauth-bundle update

~0.3 is required in composer.json

0.4.2 locked in composer.lock

$ composer show | grep hwi
hwi/oauth-bundle                         0.4.2           Support for authenticating users using both OAuth1.0a and OAuth2 in Symfony2.

0.5 released and matches ~0.3 constraints

Same time 0.4 release required changes in code

Added: configuration parameter firewall_name (will be removed in next major version) renamed to firewall_names to support multiple firewalls

And without platform support it becomes broken
Thats why with hwi/oauth-bundle:~0.3 constraint platform's PATCH release might require manual work from your side and that is not backwards-compatible bug fixes

You're correct in saying that updating that library broke your application, but it should not had. Infact, looking at the code they reverted the change to be BC-compatible in 0.4.0. Unfortunately, in 0.5 they removed that "BC-compatible layer" and imho, again, they should not had to because according to the definition of semver posted before only major version changes should introduce BC breaks, and 0.5 is not a major version (according to the specs). At least, this is what I understand from that website. If I'm wrong, sorry.

Just my 2cents:
Minor versions are not safe. Only patch versions should be safe.
You see a perfect bad example here in oro/platform (UPGRADE-1.10.1.md - More details here orocrm/crm#210)
Because you cannot trust vendors like the orocrm team, who have problems with semver, you should always fixate versions for production and that's why the composer.lock was introduced.

we can update dependencies by request, but our experience of using .* style versions wasn't good as even patch version releases of some components were braking application.
He's totally correct. But it's a matter of trust. I trust the symfony team and therefore always use variable patch versions for their libs to benefit from bug fixes etc.

I think you should always try use variable patch versions, because you benefit from security fixes with a minimum risk as far as the vendor is competent.
If there are problems with a patch version, you can update your own composer.json as fast as possible with a fixed version. But you have to be fast.
@sergeyz:
Why do you seem to understand semver and composer but still have "hwi/oauth-bundle": "~0.3" in the composer.json in master? You seem to know that it will break the application **but still have it in your vendors list?
Just to make sure: The composer.lock in oro/platform-application is no excuse, because every user who needs a new vendor has to rely on the composer.json files from his vendors because the composer.lock will be overwritten!
Back to your request @ste93cry
Because of the vendors from oro/platform and Symfony 2.8 Standard Edition are simply not compatible and if the oro team would relax the vendors, the oro platform is highly unstable for some users.
I recommend to you the following reads:
http://semver.org/
https://getcomposer.org/doc/articles/versions.md
https://semver.mwl.be/#?package=hwi%2Foauth-bundle&version=~0.3&minimum-stability=stable
Hope I can help you with it

@ste93cry @sbani
Back here with 2.0 release. All PHP vendors were allowed to update patch versions only https://github.com/orocrm/platform/blob/master/composer.json#L19-L110. We tried to update as much as possible.
oro/redis-config is a specific case, not locked because we are not going to break things and it's not enabled without configuration
JS vendors locked to specific versions (a lot of issues in patch releases)

This is great, although at least Symfony should be allowed to update minor releases too because the compatibility is granted by their Backwards Compatibility Promise. It's true that sometimes they made BC breaks, but they reverted all of the changes as soon as they discovered them. In reality, while agreeing with @sbani that indeed many third-parties vendors don't follow SEMVER or don't know how to do it, there are a few other big projects like Doctrine that usually don't break BC, so they should too be allowed to update imho.

@sbani @ste93cry
Do you still have any questions or can I close this issue?

I am now closing this issue as it has been inactive for a while. Please, feel free to reopen it if you have any questions.


[ADD SUGGESTION] vkcache.com

This is an obnoxious domain that is used by either advertising or tracking on some video sharing websites. The worst thing is it makes callbacks literally every second under different domain prefix. I had to block it manually on my pihole.
Example:

One more piece of info - the randomness could suggest some sort of subdomain attack.
https://github.com/galprz/dns-random-subdomains-ddos-attack

Are you joking? That is a CDN for VK, a Russian social network.

Apparently it is not anymore. The domain expired and it got snagged off bz someone else

The lookup on icann confirms it too

Apparently it is not anymore. The domain expired and it got snagged off by someone else

The name is obviously fake and the registrar is DropCatch.com which is domain marketplace for domains about to expire

Are you joking? That is a CDN for VK, a Russian social network.
While it might have been at one point it definitelly is not anymore. The ip it is tied to is hosted in US and the registrar is definitelly not consistent with actual VK.
It is just a very good camuflage.

Thank you for this detailed report Adam @DMNerd.
Looking at this now.  It's too bad hosts files don't support wildcards, but this is a root domain I'm happy to include.  Even the root domain is horrible.

Done.  Thanks Adam @DMNerd.
Closing.


Fixed emojis in calendar by extracting the emoji and discarding the img tag

Fixes #72
I look for the class="emoji" instead of .startsWith('<img')because i think the issue should be fixed just for emojis. Images would be a side-effect and maybe in future they have some use.

Thanks for the fix.
Since image tags could be placed in any field in YAML, the problem should be treated more generally.
Also, we don't want to leave any image tags to the SVG graph building.
Thus, all image tags with alt will be replaced by the text in alt, otherwise replaced by empty string.
A fix has been committed (3810a63ce71502173389ecfc6438f0977b03a25a) for this.


Linux.Clipboard: Use a new method to read text

GirCore provides a new async method to read text from a Gdk.Clipboard.

Pull request type

Please check the type of change your PR introduces:

[ ] Bugfix
[ ] New feature or enhancement
[ ] UI change (please include screenshot!)
[ ] Code style update (formatting, renaming)
[x] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Internationalization and localization
[ ] Other (please describe):

What is the current behavior?

Issue Number: N/A
What is the new behavior?

Other information

Quality check
Before creating this PR:

[x] Did you follow the code style guideline as described in CONTRIBUTING.md
[x] Did you build the app and test your changes?
[ ] Did you check for accessibility? On Windows, you can use Accessibility Insights for this.
[ ] Did you verify that the change work in Release build configuration
[x] Did you verify that all unit tests pass
[x] If necessary and if possible, did you verify your changes on:

[ ] Windows
[ ] macOS
[x] Linux

It looks good to me, thank you for your help! :D


Update: Microsoft.AzureCLI version 2.34.1

Removing old versions to reduce clutter
Microsoft Reviewers: Open in CodeFlow

/azp run


first steps at tls client forwarding

Example configuration
# rules.toml

[backends]
  [backends.backend1]
    [backends.backend1.circuitbreaker]
      expression = "NetworkErrorRatio() > 0.5"
    [backends.backend1.servers.server1]
    url = "https://<IP_ADDRESS>:8443"
    weight = 10
  [backends.backend2]
    [backends.backend2.servers.server1]
    url = "http://<IP_ADDRESS>:50051"
    weight = 10

[frontends]
  [frontends.frontend1]
  backend = "backend1"
  passTLSCert=true
  passHostHeader=true
  entrypoints = ["https"]
    [frontends.frontend1.routes.test_1]
    rule = "Host:kubernetes"
  [frontends.frontend2]
  backend = "backend2"
  passHostHeader=true
  entrypoints = ["http"]
    [frontends.frontend2.routes.test_1]
    rule = "Host:apiserver.local"


[DevTools Network Issues Guide Suggestion]

How does one monitor network response time and health of chrome browser ? Is this tool a good resource?
Or is this only for App Dev

Hey @vini1900 this repo is only used for content on WebFundamentals, I'd suggest posting question to StackOverflow with google-chrome-devtools tag.
Sorry for the confusion !


Update RunAsUsername to beta

This updates the docs to promote RunAsUsername from alpha to beta. See https://github.com/kubernetes/enhancements/issues/1043
/milestone v1.17

/milestone v1.17

/milestone 1.17

The code merged https://github.com/kubernetes/kubernetes/pull/84882, so this doc is ready for review/merge.

Thanks @sftim - updated based on feedback

This is ready for doc review, let me know if any changes are needed. Thanks!

/approve
/lgtm

/lgtm
/approve


feat: add toggle rule (v2)

an alternative pattern to #56, suggested by @Czaki ... where the toggle rule only takes function that returns the current value.  We then call that function on the menu's aboutToShow signal.
This won't work for qtoolbuttons ... but that can be addressed later (and a when clause would work for those buttons anyway)

@Czaki, on the napari side, the ViewerToggleAction also looks simpler:
class ViewerToggleAction(Action):
    def __init__(
        self,
        *,
        id: str,
        title: str,
        viewer_attribute: str,
        sub_attribute: str,
        **kwargs,
    ):
        def get_current(viewer: 'Viewer'):
            """return the current value of the viewer attribute"""
            attr = getattr(viewer, viewer_attribute)
            return getattr(attr, sub_attribute)

        def toggle(viewer: 'Viewer'):
            """toggle the viewer attribute"""
            attr = getattr(viewer, viewer_attribute)
            setattr(attr, sub_attribute, not getattr(attr, sub_attribute))

        super().__init__(
            id=id,
            title=title,
            toggled=ToggleRule(get_current=get_current),
            callback=toggle,
            **kwargs,
        )


gapi.client.* updates

Similar to https://github.com/microsoft/DefinitelyTyped-tools/pull/311 and other PRs by me, removing APIs that are removed from google discovery, and adding new ones

Turns out the 4 deleted lines are actually still in use by packages on DT. I'm going to restore them.

Right, I forgot that it's a two step process.. I'll remove them once my PR to DT to remove them is merged.
If we could add support for wildcards in this file, it'd really help me.

Right, I forgot that it's a two step process

Me too. Sorry for the inconvenience.

If we could add support for wildcards in this file, it'd really help me.

That makes sense, but the point of the file is to prevent people adding dependencies without telling us, which is defeated by a wildcard. Also the code right now is really simple (literally new Set(text.split(/\n/)) followed by if (set.has(name))).


[ggj] Generated classes which implement ResourceName do not use fully qualified names for java.lang classes

Steps to reproduce

Attempt to generate gapic client for storage/v2
Receive compile error[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/CryptoKeyName.java:[173,3] method does not override or implement a method from a supertype
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/CryptoKeyName.java:[175,11] incomparable types: com.google.storage.v2.Object and com.google.storage.v2.CryptoKeyName
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/CryptoKeyName.java:[178,33] incomparable types: java.lang.Class<capture#1 of ? extends com.google.storage.v2.CryptoKeyName> and java.lang.Class<capture#2 of ? extends com.google.storage.v2.Object>
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/CryptoKeyName.java:[179,45] incompatible types: com.google.storage.v2.Object cannot be converted to com.google.storage.v2.CryptoKeyName
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/BucketName.java:[135,3] method does not override or implement a method from a supertype
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/BucketName.java:[137,11] incomparable types: com.google.storage.v2.Object and com.google.storage.v2.BucketName
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/BucketName.java:[140,33] incomparable types: java.lang.Class<capture#3 of ? extends com.google.storage.v2.BucketName> and java.lang.Class<capture#4 of ? extends com.google.storage.v2.Object>
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/BucketName.java:[141,39] incompatible types: com.google.storage.v2.Object cannot be converted to com.google.storage.v2.BucketName
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/ProjectName.java:[125,3] method does not override or implement a method from a supertype
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/ProjectName.java:[127,11] incomparable types: com.google.storage.v2.Object and com.google.storage.v2.ProjectName
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/ProjectName.java:[130,33] incomparable types: java.lang.Class<capture#5 of ? extends com.google.storage.v2.ProjectName> and java.lang.Class<capture#6 of ? extends com.google.storage.v2.Object>
[ERROR] .../src/github.com/googleapis/java-storage/proto-google-cloud-storage-v2/src/main/java/com/google/storage/v2/ProjectName.java:[131,41] incompatible types: com.google.storage.v2.Object cannot be converted to com.google.storage.v2.ProjectName

BucketName, CryptoKeyName and ProjectName all implemented ResourceName and follow a different code generation path than the other proto classes. This different generation path does not use fully qualified class names for java.lang classes. This particular scenario is manifesting because GCS generate a class com.google.storage.v1.Object which takes resolution precedence to java.lang.Object and prevents java.lang.Object#equals(java.lang.Object) from being implemented.
As a temporary work around I'm manually applying a search and replace which can be seen here: https://github.com/googleapis/java-storage/blob/d6acdab54582852b3c14e152a12ea2f923ff85ef/synth.py#L40-L58

Using fully-qualified names for java.lang classes may make things less user-friendly or less readable, but we can fix the name collision for Object. Will look into this next week for FixIt.

It appears protobuf and gRPC exclusively uses fully qualified names (example https://github.com/googleapis/java-storage/pull/922/files#diff-2bb347d8e4f8109169ef2cd49aea4352494276e2264ceca7ecc9a5312c134847R308-R309). At my codes use site either short or fully qualified work equally well.


feat: create card component

create project card component with dummy data
closes Unfinished Card Component #15

Gonna wait on Sarthaks review first since I'm not a frontend person.

Card component looks awesome but looks like the grid is messed up, let me check your code and i'll comment on it.

can somebody else mark this as resolved


feat(config): read autoupdate settings from global config

Description:
Read autoupdate config from global config rather than hardcoding into the autoupdater class.
Motivation and Context:
Consolidate settings and make easier to change config.
How Has This Been Tested?
Manually
Types of changes:
Enhancement
Checklist:

[x] My code follows the code style of this project.
[x] I have reviewed and updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes where needed.
[x] All new and existing tests passed.
[x] My commits have been squashed into a concise set of changes.

Coverage remained the same at 20.212% when pulling 38ffa9ed8ba4a069686e298950612f8e532b54d8 on mrfelton:fix/autoupdate-config into c92c81a19736a76d446ff2a7db45a29b63472ccc on LN-Zap:next.


Accept fields when POSTing to /messages

Hi
I implemented the support  for fields, the same way I did for vars. I should have been more thorough the first time and deal with both in a single PR, sorry about that.
I reused the same tests, since it's really just a lookup in a hash, and I didn't want to add ~70 lines of code just for that.
Please let me know if that suits you.
Thanks

Hello Jeff,
no worries about multiple pull requests, I like to have many small commits that show the thought process.
Thanks a lot!


[Feature request] Emotes in calendar

Hi! I use the calendar together with my family. Some of them use emotes in the title of the event. Sadly it only renders as a box in the text. Is it possible to render the emotes as well?

Emoji are handled by fonts you should be able to install a font to magicmirror that has emoji support. ***I have not attempted this just an assumtion based on my knowledge of fonts.
Go to the fonts folder.
Then npm install the appropriate package
https://fontawesome.com/docs/web/setup/packages
Then change the font in the top of css/custom.css
.normal,
.dimmed,
header,
body {
  color: #fff;
  font-family: "Times New Roman", "Fallback_font_or_font_family"
}


feat(FormGroup): add eager validation

🔗 Linked issue
#891

❓ Type of change

[x] 📖 Documentation (updates to the documentation or readme)
[ ] 🐞 Bug fix (a non-breaking change that fixes an issue)
[x] 👌 Enhancement (improving an existing functionality)
[x] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

Currently form validation doesn't run on the Input and Textarea components until after their initial blur. This change adds an eager-validation prop to the Input and Textarea components. This will allow users to specify if they want validation to run while the end user is typing.
A callout has been added to the Form docs to explain that no validation runs until after blur and that this new prop can be used to override that behavior
Resolves #891
📝 Checklist

[x] I have linked an issue or discussion.
[x] I have updated the documentation accordingly.

Thanks 😊


[Question]: Cover_Path returns local file / Cloud images are encrypted

Ask your question
Hey @bropat ,
Seems like Eufy patched more in their API and now the cover_path is moved from a CDN url to T8410REDACTED~/media/mmcblk0p1/Camera00/20221210230725n.jpg
Any idea if it's possible to fetch an image from the station itself?

same

Hi martijnpoppen,
this seem to be implemented by the firmware version <IP_ADDRESS>. My HomeBase E gives another link to the image (T8002REDACTED~/media/mmcblk0p1/video/20221210235159_c00.jpg). My cams on the HomeBase 2 have not detect any motion since the firmware update.
I think, we have two options: We can use the stationDownload function to download the image (i will try that tonight). Or, if this does not work, we have to implement function like bropat did it in his ioBroker plugin (onStationDownloadStart function in bropat/ioBroker.eusec). There he is downloading the video and create a thumbnail from the downloaded video. But this will add new dependencies (mainly ffmpeg).

Can we download the image as if we download the video?

Can we download the image as if we download the video?

I am not sure. I will try it later this day.

Can we download the image as if we download the video?

I am not familiar with downloading video, can you explain me how to do it with Home Assistant or Node-red?

I will check this soon...

I checked my Eufy Cam 2 and the notification image is still coming from the cloud, except that the image file is encrypted.
The first 41 bytes are a custom header from Eufy that contains info to decrypt the beginning of the image data. Only the first 256 bytes of the image data are encrypted with AES.

Thanks for your checking @bropat. May I ask, what firmware version your homebase is running on? The problem here occurred after the homebase was updated to <IP_ADDRESS>(h) and only for events after the firmware update. Very strange...
DEBUG: Normalized Properties {  deviceSN: 'T8112N2X',  properties: {    motionDetected: false,    personDetected: false,    personName: '',    name: 'Wäscheplatz Tor',    model: 'T8112',    serialNumber: 'T8112N2X',    type: 1,    hardwareVersion: 'HAIYI-IMX323',    softwareVersion: '1.9.3',    stationSerialNumber: 'T8002N2X',    pictureUrl: 'T8002N2X~/media/mmcblk0p1/video/20221210235159_c00.jpg',    recordingClipLength: 0, ...

@PhilippEngler thats the cover_path from the device response
The one @bropat talks about is the cloud image from the notification

@bropat is the decryption something you want to put in this library?
I was thinking that it might be nice to provide a decryption method which can be used by the clients using the library ?

I wish you all success. I hope you all can find a way to retrieve the last motion image (thumbnail). I always used it for sending to my telegram with (home assistant / node-red) because I did this for all my cams. It's fast and much better than opening all kinds of apps.

Thank you for clarification, @martijnpoppen, I have misunderstood that point, sorry.

I think the android app (apk) will show the way to decode the thumbnail ...

@bropat i tried several decryption methods based on the info you gave (first 41 bytes etc) but unfortunately i end up with corrupted images. Hopefully you can come up with a nice solution :D

@martijnpoppen
A few days ago I successfully decrypted the image with the correct keys ;)
But I still need to explore the generation of the correct key.
Unfortunately I had a very stressful month at work and could not spend much time...

Thats moe than i achieved :p
No worries! Take your time and have nice holidays! 🍾🎄

Thank you all for the effort. Have a nice Christmas and lucky new year!

Thank you, you too

I am struggling with the 2fa again. I have to reconfigure it, to see if i get it working again.,

I will ask it again because I never got an answer. Is it possible to retrieve any video from the base station? What can we do with the eufy-security-ws library at this moment other than streaming start/stop and motion and ring detection? Can we retrieve any kind of video from the past?

@Beer17HWAM did you check commands and events of the repository and couldn't find your answer?

@Beer17HWAM did you check commands and events of the repository and couldn't find your answer?

I'am only able to start video streaming. I don't know if I can do more. I like to hear it from someone who knows. Maybe with examples in HA or Node-red. I can't find it myself.

You didn't answer my question? Did you go through the getting started documentation?

You didn't answer my question? Did you go through the getting started documentation?

Yes here:
https://bropat.github.io/eufy-security-ws/#/api_cmds
But I don't know how to do use the commands and I hope someone can show me how to retrieve old videos.

So, your question had changed a lot. So you know now it is capable of doing it but you don't know how to do it.
You can more info below or different previously created issues.
https://github.com/bropat/eufy-security-ws/issues/31

Thanks for the info. I was hoping for a step by step instruction manual and an example. I am familiar with scripting but this is a bit to complex for me.

it is not an easy thing, agree. Let me put more detailed example here, I am planning to add this feature into Home Assistant integration soon (get the latest recorded video)
send_message - {'messageId': '1', 'command': 'driver.get_video_events', 'maxResults': 1}

 _on_message - {'type': 'result', 'success': True, 'messageId': '1', 'result': {'events': [{'monitor_id': 114484245, 'transfer_monitor_id': 0, 'station_sn': 'T8010N2320460480', 'device_sn': 'T8113N63205014E2', 'storage_type': 1, 'storage_path': '/media/mmcblk0p1/Camera01/20221230205117.dat', 'hevc_storage_path': '', 'cloud_path': '', 'frame_num': 149, 'thumb_path': 'T8010N2320460480~/media/mmcblk0p1/video/20221230205117_c01.jpg', 'thumb_data': '', 'start_time':<PHONE_NUMBER>877, 'end_time':<PHONE_NUMBER>784, 'cipher_id': 191, 'cipher_user_id': 'd0da85d5ba183f277c3d7eca940c4cb880c43651', 'has_human': 0, 'volume': 'Anker_B_JICiGmd', 'vision': 1, 'device_name': 'Entrance', 'device_type': 8, 'video_type': 1002, 'extra': 'eyJhdXRvbWF0aW9uX2lkIjowLCJoYXNfbWRldGVjdCI6MSwicHVzaF9tb2RlIjoyLCJtaWNfc3RhdHVzIjoxLCJyZWNvcmRfZm9ybWF0IjoxLCJwaWNrX3RpbWUiOjAsImRlbGl2ZXJfdGltZSI6MH0=', 'user_range_id': 387702, 'viewed': False, 'create_time':<PHONE_NUMBER>, 'update_time':<PHONE_NUMBER>, 'status': 1, 'station_name': '', 'p2p_did': 'HXEUCAM-135914-HKXPL', 'push_did': 'HXEUCAM-135914-HKXPL', 'p2p_license': 'RSEUZH', 'push_license': '', 'ndt_did': 'HXEUCAM-135914-HKXPL', 'ndt_license': '', 'wakeup_flag': 1, 'p2p_conn': '', 'app_conn': '', 'query_server_did': '', 'prefix': 'ZXCAMAA', 'wakeup_key': '', 'ai_faces': None, 'is_favorite': False, 'storage_alias': 1}]}}

send_message - call.data: {'message': {'messageId': '1', 'command': 'device.start_download', 'serialNumber': 'T8113N63205014E2', 'cipherId': 191, 'path': '/media/mmcblk0p1/Camera01/20221230205117.dat'}}

_on_message - {'type': 'result', 'success': True, 'messageId': '1', 'result': {'async': True}}

_on_message - {'type': 'event', 'event': {'source': 'device', 'event': 'command result', 'serialNumber': 'T8113N63205014E2', 'command': 'start_download', 'returnCode': 0, 'returnCodeName': 'ERROR_PPCS_SUCCESSFUL', 'customData': {'command': {'name': 'deviceStartDownload', 'value': {'path': '/media/mmcblk0p1/Camera01/20221230205117.dat', 'cipher_id': 191}}}}}

_on_message - {'type': 'event', 'event': {'source': 'device', 'event': 'download started', 'serialNumber': 'T8113N63205014E2'}}

_on_message - {'type': 'event', 'event': {'source': 'device', 'event': 'download video data', 'serialNumber': 'T8113N63205014E2', 'buffer': {'type': 'Buffer', 'data': [VIDEO BYTES ARE HERE]}, 'metadata': {'videoCodec': 'H264', 'videoFPS': 15, 'videoHeight': 1080, 'videoWidth': 1920}}}

 _on_message - {'type': 'event', 'event': {'source': 'device', 'event': 'download video data', 'serialNumber': 'T8113N63205014E2', 'buffer': {'type': 'Buffer', 'data': [AUDIO BYTES ARE HERE]}, 'metadata': {'videoCodec': 'H264', 'videoFPS': 15, 'videoHeight': 1080, 'videoWidth': 1920}}}

Now, you need to merge video and audio bytes into one video file, save it somewhere so you can watch it later on. Or you can live stream these.

If I am right, the image path in the notification event is only contained when the extended push notification is enabled.
I have taken a deeper look at the app and the p2p communication:
The app use p2p to fetch the preview image from the HomeBase. Therefor the "cover_path" is used. The "~" is to split the path in station serial and the path to the image.
For now, I was able to build the p2p request, sending it to the station and receiving data from the station. What is missing is parsing the data and store the image (and last but not least the confirmation, that the data contains the image).
You can take a look here: https://github.com/PhilippEngler/eufy-security-client/commit/4bed085f413df9fb04ee35e8a71cac59d45011f1

Nice @PhilippEngler
I'll have a look at that too! 👀

fyi: The image data you receive via the p2p command is encrypted in the same way as the images of the cloud push notification.
I think I am close to the result.
The reverse engineering of .so files is much more complicated (libenc.so)... ;)

@bropat can we support in any way to help in the reverse engineering? Missing the pictures of the camera a lot :)

See my post
fuatakgun/eufy_security#685
I start the Eufy Doorbell Stream and grab a part of the stream. I grab directly from the rtsp Eufy stream, not the go2rtc stream. It's working fine but indeed ffmpeg is slow to start, you miss the first seconds. But it works, you can send a few seconds to Telegram. I have it up and running now:
Start streaming
Grab piece of stream:
ffmpeg -y -i rtsp://<IP_ADDRESS>:8554/T8210P0020330C10 -c copy -map 0 -t 6 ffmpeg-snapshot.mp4
Send mp4 to Telegram
Repeat this as long there is motion and keep it running for a half a minute or so when motion stops.
It works great. I tried to Grab one jpg frame but that is not working because the first part is indeed not "full". It will be a distorted jpg because of the delay of a keyframe. The mp4 is perfect. I grab 6 seconds.
I hope you can do something with this now. Good luck.

Sorry guys had a lot to do and so couldn't invest much time here.
Anyone who wants to help is of course welcome. :)
As you know eufy has encrypted the images you receive through the push notification. The same encryption is also used when you try to download the images via P2P (see commit above from @PhilippEngler).
Example Image data:
eufysecurity:T8010P2XXXXXXXX8:0110972768:<256 bytes encrypted image header><unencrypted image data>

The first 41 bytes contain part of the required decryption information.
3 pieces of information are supplied (separated by :):

eufysecurity: If the first 12 bytes contain the string eufysecurity, the image data is encrypted, otherwise not.
T8010P2XXXXXXXX8: serialnumber of the station
0110972768: Secret required to derive the effective encryption key (AES key)

A fourth piece of information is needed and this is fetched directly from the cloud from the properties of the station, namely p2p_did.
The following implementation is found in the code used for image decoding:
public byte[] decryptImage(byte[] image_data) {
        int length = image_data.length - 41;
        byte[] bImageData = new byte[length];
        byte[] bSerialnumber= new byte[16];
        byte[] bSecret = new byte[10];
        // Station serialnumber
        System.arraycopy(image_data, 13, bSerialnumber, 0, 16);
        // Secret
        System.arraycopy(image_data, 30, bSecret, 0, 10);
        // Effective image data
        System.arraycopy(image_data, 41, bImageData, 0, length);
        String serialnumber = new String(bSerialnumber, StandardCharsets.UTF_8);
        String secret = new String(bSecret, StandardCharsets.UTF_8);
        QueryStationData queryStationData = StationDataManager.a().e(serialnumber);
        if (e == null) {
            return null;
        }
        // Get AES key passing station serialnumber, p2p_did and secret to the jndi function in libenc.so
        String aeskey = Encoder.genCheckCode(serialnumber, queryStationData.p2p_did, secret).substring(0, 16);
        byte[] bEncryptedImageHeader = new byte[256];
        System.arraycopy(bImageData, 0, bEncryptedImageHeader, 0, 256);
        // Decrypt first 256 image bytes using the derivated AES key
        System.arraycopy(decrypt(bEncryptedImageHeader, aeskey), 0, bImageData, 0, 256);
        return bImageData;
}

public byte[] decrypt(byte[] bEncryptedData, String aeskey) {
        try {
            SecretKeySpec secretKeySpec = new SecretKeySpec(aeskey.getBytes(StandardCharsets.UTF_8), "AES");
            Cipher cipher = Cipher.getInstance("AES/ECB/NoPadding");
            cipher.init(2, secretKeySpec);
            return cipher.doFinal(bEncryptedData);
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        }
    }
...

The following is the class that loads the native external library libenc.so:
public class Encoder {
    public static native String genCheckCode(String serialnumber, String p2p_did, String secret);

    static {
        System.loadLibrary("enc");
    }
}

This library contains the knowledge of how to derive the AES key to decrypt the image header. For this reason, we need to understand how this works through reverse engineering.
I use ghidra for that.
So I load the library libenc.so into ghidra and have it analysed. From this I get pseudo C code. Here you still have to correct the data types and sizes. Of course, you can also give the variables meaningful names, etc.
Behind the function genCheckCode you will find the following procedures in the native library:
    gen_pic_base_code(p_serialnumber,p_len_serialnumber,p_p2p_did,res_basecode);
    gen_rand_seed(p_p2p_did,p_numericstr,res_seed);
    gen_check_code_v1(res_basecode,res_seed,out_result);

I have tried to understand these procedures and translated them into the following functions in typescript:
import md5 from "crypto-js/md5";
import enc_hex from "crypto-js/enc-hex";
import sha256 from "crypto-js/sha256";

export const getIdSuffix = function(p2pDid: string): number {
    let result = 0;
    const match = p2pDid.match(/^[A-Z]+-(\d+)-[A-Z]+$/);
    if (match?.length == 2) {
        /*const num1 = Number.parseInt(match[1]);
        const num2 = Number.parseInt(match[1].substring(1));
        const num3 = Number.parseInt(match[1].substring(3));
        const num4 = Number.parseInt(match[1].substring(5));*/
        const num1 = Number.parseInt(match[1][0]);
        const num2 = Number.parseInt(match[1][1]);
        const num3 = Number.parseInt(match[1][3]);
        const num4 = Number.parseInt(match[1][5]);

        result = num1 + num2 + num3;
        if (num3 < 5) {
          result = result + num3;
        }
        result = result + num4;
    }
    return result;
};

export const getImageBaseCode = function(serialnumber: string, p2pDid: string): string {
    let nr = 0;
    try {
        nr = Number.parseInt(`0x${serialnumber[serialnumber.length - 1]}`);
    } catch (error) {
    }
    nr = (nr  + 10) % 10;
    const base = serialnumber.substring(nr);
    return `${base}${getIdSuffix(p2pDid)}`;
};

export const getImageSeed = function(p2pDid: string, secret: string): string {
    try {
        const nsecret = Number.parseInt(secret.substring(2));
        const prefix = 1000 - getIdSuffix(p2pDid);
        return md5(`${prefix}${nsecret}`).toString(enc_hex).toUpperCase();
    } catch(error) {
        //TODO: raise custom exception
    }
    return ``;
};

export const getImageKey = function(serialnummer: string, p2pDid: string, secret: string): string {
    const basecode = getImageBaseCode(serialnummer, p2pDid);
    const seed = getImageSeed(p2pDid, secret);
    const data = `10${basecode}${seed}`;
    const hash = sha256(data);
    const hashBytes = [...Buffer.from(hash.toString(enc_hex), "hex")];
    for(let i = 0; i < 32; i++) {
        const byte = hashBytes[i];
        let fixed_byte = hashBytes[10];
        if (i < 31) {
            fixed_byte = hashBytes[i + 1];
        }
        if ((i == 31) || ((i & 1) != 0)) {
            if (i != 31) {
                hashBytes[10] = fixed_byte;
            }
            if ((126 < byte) || (126 < hashBytes[10])) {
                if (byte < hashBytes[10] || (byte - hashBytes[10]) == 0) {
                    hashBytes[i] = hashBytes[10] - byte;
                } else {
                    hashBytes[i] = byte - hashBytes[10];
                }
            }
        } else if ((byte < 125) || (fixed_byte < 125)) {
            hashBytes[i] = fixed_byte + byte;
        }
    }

    return `${Buffer.from(hashBytes.slice(16)).toString("hex").toUpperCase()}`;
};

Unfortunately, I do not get the expected result. I think the error is in the second half of the getImageKey function...
To understand where this effectively is, you have to debug the native library at runtime (e.g. with gdb).
So I wrote a simple Java console program that implements and calls the native class Encoder so that I can test the native library better.
package com.example.enc;

import com.example.enc.Encoder;
import java.util.Scanner;

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Start...");
	String serialnumber = "0123456789000003";
	String p2p_did = "ZXCAMAA-000000-HCUHU";
	String numericstr = "0000000000";
	if (args.length > 0) {
		serialnumber = args[0];
	}
	if (args.length > 1) {
		p2p_did = args[1];
	}
	if (args.length > 2) {
		numericstr = args[2];
	}
	System.out.println("Serialnumber: " + serialnumber);
	System.out.println("p2p_did: " + p2p_did);
	System.out.println("numericstr: " + numericstr);
        String result = Encoder.genCheckCode(serialnumber, p2p_did, numericstr);
        System.out.println("1 Result length: " + result.length());
        System.out.println("1 Result: " + result);

	Scanner sc = new Scanner(System.in);
	sc.nextLine();

        result = Encoder.genCheckCode(serialnumber, p2p_did, numericstr);
        System.out.println("2 Result length: " + result.length());
        System.out.println("2 Result: " + result);

    }

}

The line with sc.nextLine(); was implemented so that I can easily debug the process with gdb and be sure that the native library has already been loaded by the JVM (dalvikVM).
I compiled the Java code and created a jar file from the class files. I then converted this into a dex file using the Android SDK Build Tools (v. 33.0.0) with d8:
.../Android/sdk/build-tools/33.0.0/d8 --output dex helloworld.jar

I then loaded the dex file by adb push onto a rooted device or emulator including the libenc.so file (in the correct architecture of the device or emulator):
adb push classes.dex /data/local/tmp
adb push libenc.so /data/local/tmp

Then I started the program on the device or emulator as follows:
adb shell
su -
cd /data/local/tmp
export LD_LIBRARY_PATH=/data/local/tmp
dalvikvm64 -cp classes.dex com.example.HelloWorld<PHONE_NUMBER>000003 ZXCAMAA-000000-HCUHU<PHONE_NUMBER>

Output of the above command:
Start...
Serialnumber:<PHONE_NUMBER>000003
p2p_did: ZXCAMAA-000000-HCUHU
numericstr:<PHONE_NUMBER>
output:: ts:0000000000, check code: D174D8169229AD39F55E6C3D801E4169
1 Result length: 32
1 Result: D174D8169229AD39F55E6C3D801E4169

<ENTER>

output:: ts:0000000000, check code: D174D8169229AD39F55E6C3D801E4169
2 Result length: 32
2 Result: D174D8169229AD39F55E6C3D801E4169

If you pass the correct input values here, you will get the AES key to decrypt the image header.
Attention: Only the first 16 bytes are used as AES key (see Java code at the beginning).
I have now fetched the gdbserver for the correct architecture of the device or emulator from the Android NDK (older version still supplied with gdb), loaded it with adb push and then started it.

start the HelloWorld Java console program as shown above and do not give an enter command yet.
Find the process id of launched java program:

adb push gdbserver /data/local/tmp
adb shell
su -
ps -edf | grep dalvikvm

Launch gdbserver as follow from another session:

adb push gdbserver /data/local/tmp
adb shell
su -
cd /data/local/tmp
./gdbserver --attach :<port> <pid>
# Example: ./gdbserver --attach :5055 1234

Now you can forward the previously selected port with adb so that you can debug from the PC:
adb forward tcp:5055 tcp:5055

Now you can use gdb or ghidra or radare to connect to the remote gdbserver process via localhost:5055 and start debugging.

So far, I have not been able to read the correct registers or memory areas to understand what was misinterpreted in the typescript code above.
I assume that there is an error in the generated pseudo C code of ghidra. Here you could also check the assembler code using the official documentation from ARM to understand what ghidra has interpreted wrong... Another challenge... ;)
The challenge is now to get the correct result with the following input values and thus solve the mystery ;)
serialnumber = "0123456789000003";
p2pDid = "ZXCAMAA-000000-HCUHU";
code = "0000000000";

Expected outcome: D174D8169229AD39F55E6C3D801E4169
If someone can be helpful here would be great :)

Annex:
I have compiled the Java code with the following command:
javac -source 1.7 -target 1.7 -d bin src/com/example/enc/Encoder.java
javac -source 1.7 -target 1.7 -d bin -cp bin/com/example/enc src/com/example/enc/HelloWorld.java

I'll have a look soon :)

So small update from my side
I spend some long evenings on this.
Unluckily I'm not really helpful...
Java and reverse engineering APK's are not my expertise.
Certainly not reading Peudo Code C files 😅
I did take another approach at getting the genCheckCode result.
I hooked up frida to my emulator and ofcourse I also got D174D8169229AD39F55E6C3D801E4169 back.
Next to that I tried to dissasemble the libhome_security.so (same as libenc.so?) with RetDec and IDA Pro.
The IDA Pro file is better readable than the RetDec variant, but still I couldn't manage to get anything useful out of it.
If it helps I can share you the IDA Pro functions.
Also trying another approach. I downloaded the ROM of my indoorcam. Hoping there would be a encrypt function somewhere which would help us decrypting it. No luck yet, but will dig further.
Might be interesting...: it seems like the Homebase 3 isn't encrypting it's images..

I have also spend some time on this. Yes, @martijnpoppen, the functions from libenc.so were moved to libhome_security.so in v4.5.4 (and the libhome_security.so should also includes the new p2p messages types).
I have also tried to understand the code. From my side, I have a slightly different implementation of the for-statement of @bropat's getImageKey-function. But the result is the same, so that I believe, that the problem is in one of the three functions before. I will have a look on this.
One question I have regarding @bropat's implementation:
Does ${Buffer.from(hashBytes.slice(16)).toString("hex").toUpperCase()} will return us the really the first 16 bytes? I have understand the slice-function so, that in this case we will get the last 16 bytes. ${Buffer.from(hashBytes.slice(0,16)).toString("hex").toUpperCase()} will return the first 16 bytes? But maybe (surely) I am completely wrong.
The non-encrypting thing with the HB3 is interesting... All in all I have the impression, that they are being under time pressure. The app versions v4.5.4 and v4.5.5 for example does not show the mode changing events from HomeBase 2 or E while they are on firmware version below <IP_ADDRESS>. And the app v4.5.3 does not show the mode changing events from HomeBase firmware <IP_ADDRESS>.

Ah thanks @PhilippEngler that makes sense. Saw that GenCheckCode is there since app V2.
I do think you're right about the 3 first functions, also tried some different implementations but got the same result everytime.
Next to that I did just find something in the firmware 😄
There's home_security file there too with a function save_the_enc_pic.
Might be useful ?

Thank you @martijnpoppen for the information about the GenCheckCode occurrence in the app.

I do think you're right about the 3 first functions, also tried some different implementations but got the same result everytime.

That sounds quite good. @bropat has done great work with the implementation.

Next to that I did just find something in the firmware 😄 There's home_security file there too with functions save_the_enc_pic, gen_rand_seed, gen_check_code_v1, gen_base_code Might be useful ?

Yes, this could maybe help. The results  should be the same. Is there a possibility to send me the IDA Pro result of the three functions, I want to check if there are differences to Ghidra (I think we should not share the disassembled code here in the comments).
I hope that you have success on the firmware-way.

Yes i can send you the file. Anywhere we can reach each other? 😄
And also the firmware .so file if you're interested in that. Decompiling that one is a bit harder

Yes i can send you the file. Anywhere we can reach each other? 😄 And also the firmware .so file if you're interested in that. Decompiling that one is a bit harder

From my side, the best way is by email: I would ask you, if you could take a look here, there you can find the address (it's the package.json of my project): PhilippEngler/eufy-security-hm/package.json in the author key. It lead to my private mail, so no other person will receive this email. Many thanks in advance.
I will take what I can get and try my best :smile:.

One question I have regarding @bropat's implementation:
Does ${Buffer.from(hashBytes.slice(16)).toString("hex").toUpperCase()} will return us the really the first 16 bytes? I have understand the slice-function so, that in this case we will get the last 16 bytes. ${Buffer.from(hashBytes.slice(0,16)).toString("hex").toUpperCase()} will return the first 16 bytes? But maybe (surely) I am completely wrong.

The gen_check_code_v1 function produces a SHA256 hash at the end, whereby only the last 16 bytes (of the total 32 bytes) are converted into uppercase hex string (32 characters long) and returned as the result.
From this result, the first 16 characters of the string are then used as the AES key.

@martijnpoppen  can you send me the IDA files too? :)

@bropat @PhilippEngler Sent! :D

The IDA files only contain the relevant functions. Someone else decompiled it for me, as I don't have the pro version

Thank you very much.
I quickly skimmed the code and there is still 1 relevant function j_cal_ppcs_id_suffix missing. Can you ask for these as well? :)

It's in your mail @bropat

@martijnpoppen
I need the following file <IDA installation folder>\plugins\defs.h from the used IDA version.
Thanks again :)

@bropat send it in the mail.
Any luck on the IDA files? :)
I had a hard time understanding it.
For the Firmware SO file, it's really hard to disassemble that. As it uses a ARM architecture. and all my tools are crashing during disassembling

@martijnpoppen
Thank you very much! No, because unfortunately I didn't have time for it last week :(

I had also some troubled weeks and the next two will get not even better.
Taken a look on the file received from @martijnpoppen. Thanks for that again. I didn't found two functions, the first one is the j_cal_ppcs_id_suffix, the second one is the j_gen_rand_seed function. If you have these two functions, could you please send it, @martijnpoppen (or are they there and I did not noticed them?)? Thank you.
Is there a difference between the function starting with j_ and these not starting with j_ (e.g. j_xxx_yyy and xxx_yyy)?
@bropat and @martijnpoppen thank you for your work, I hope, we will solve the problem.

Same here indeed, had some other priorities last week.
Will try to spend some time on it this week.
@PhilippEngler  I'll forward you the same mails I send to @bropat

Thank you, the mails have been well received.

Is there any progress on thie issue?

Been quite busy lately with other priorities, from my side still trying to decompile the Firmware to find something useful there. But this dissamebly and reverse engineering is still a hard task to achieve.

@bropat @PhilippEngler I did some debugging on this topic. I managed to attach Frida to the native library functions and call them with the provided args from bropat.
I got the following results:
gen_rand_seed('ZXCAMAA-000000-HCUHU', '0000000000') // B7A782741F667201B54880C925FAEC4B

This is the same as bropat's getImageSeed implementation :tada:

So i did a lot of debugging and found out that we have the following values:
Doorbell:
the GenCheckCode is called like
GenCheckCode('DEVICE_SN', 'HXUSCAM-430329-HWEPG', '0186410153')
 returns: "E33D0354E55F390AFE6B007EC93EC021"

inside the native library:
the gen_check_code_v1 is called with:
gen_check_code_v1('DEVICE_SN22', 'CBA63CB1FE86078F682FA42FB2FEE090').

argument1 represents the basecode
argument2 represents the seed
you can validate this by running
getImageSeed('HXUSCAM-430329-HWEPG', '0186410153') // CBA63CB1FE86078F682FA42FB2FEE090

Indoorcam:
the GenCheckCode is called like
GenCheckCode('DEVICE_SN', 'HXUSCAM-348024-PHMUP', '0136152141')
returns: "456DBA5721B2345ACF3802B0F9072A2B"

inside the native library:
the gen_check_code_v1 is called with:
gen_check_code_v1('DEVICE_SN11, '03BB6E12CF8481C99B02EBEEFB710C84')

argument1 represents the basecode
argument2 represents the seed
you can validate this by running
getImageSeed('HXUSCAM-348024-PHMUP', '0136152141') // 03BB6E12CF8481C99B02EBEEFB710C84

Concluding:

getImageSeed is ✅
getIdSuffix is ✅
getImageBaseCode should return: ${DEVICE_SN}${suffix}
getImageKey should be fixed, but really can't figure it out. (tried a lot of chatGPT :P )

but if i follow the pattern from above for the keys bropat provide we should have:
basecode =<PHONE_NUMBER>0000030
seed = B7A782741F667201B54880C925FAEC4B

I hope this helps. I'll have another at this this week, but hopefully this is helpful 🤞🏼

@PhilippEngler  @martijnpoppen I would like to do some debugging myself.
Did you follow a tutorial to attach a debugger to the app?

@Palmke not really.
all the information above is done by multiple ways.
so the initial check was done by Bropat ad he found the GenCheckCode in the APK and after that started to dissamble the SO file which also can be found in the APK. For dissambling SO files you can use one of these tools: Ghidra, Retdec, Ida Pro.
I tried it differently and attachted Frida to my Android emulator and wrote a script to execute the SO functions to find out the arguments which are passed.
Any way there are multiple ways to check this.
I have to say that ChatGPT gave me a lot of info on how to do things and explanations on how SO files are working

If anyone is interested I found an interim solution on Android using Autonotification+Tasker to grab the snapshot an post in on a Home Assistant camera using the Push Camera integration. Works flawlessly.

@Zeunas that sounds interesting, at least as a fallback. Could you share it?

Thanks a lot! Your tasker profile was a good starter, though I had to do some adjustments as the curl call did not work on my Fire Tablet house dashboard (apparently no curl installed on these devices...), so I used the tasker http api.
A native implementation decrypting the data in the library would of course be preferred, but at least I got pictures on my tablet again. Hope this works as flawless for me as it does for you. Thanks again! :)

Today I finally had time to dedicate myself to this topic again and was able to decode it correctly 🎉.
I will now implement the remaining part in the library.

Here is a foretaste of the correct functions:
import md5 from "crypto-js/md5";
import enc_hex from "crypto-js/enc-hex";
import sha256 from "crypto-js/sha256";

export const getIdSuffix = function(p2pDid: string): number {
    let result = 0;
    const match = p2pDid.match(/^[A-Z]+-(\d+)-[A-Z]+$/);
    if (match?.length == 2) {
        const num1 = Number.parseInt(match[1][0]);
        const num2 = Number.parseInt(match[1][1]);
        const num3 = Number.parseInt(match[1][3]);
        const num4 = Number.parseInt(match[1][5]);

        result = num1 + num2 + num3;
        if (num3 < 5) {
          result = result + num3;
        }
        result = result + num4;
    }
    return result;
};

export const getImageBaseCode = function(serialnumber: string, p2pDid: string): string {
    let nr = 0;
    try {
        nr = Number.parseInt(`0x${serialnumber[serialnumber.length - 1]}`);
    } catch (error) {
    }
    nr = (nr + 10) % 10;
    const base = serialnumber.substring(nr);
    return `${base}${getIdSuffix(p2pDid)}`;
};

export const getImageSeed = function(p2pDid: string, code: string): string {
    try {
        const ncode = Number.parseInt(code.substring(2));
        const prefix = 1000 - getIdSuffix(p2pDid);
        return md5(`${prefix}${ncode}`).toString(enc_hex).toUpperCase();
    } catch(error) {
        //TODO: raise custom exception
    }
    return ``;
};

export const getImageKey = function(serialnummer: string, p2pDid: string, code: string): string {
    const basecode = getImageBaseCode(serialnummer, p2pDid);
    const seed = getImageSeed(p2pDid, code);
    const data = `01${basecode}${seed}`;
    const hash = sha256(data);
    const hashBytes = [...Buffer.from(hash.toString(enc_hex), "hex")];
    const startByte = hashBytes[10];
    for(let i = 0; i < 32; i++) {
        const byte = hashBytes[i];
        let fixed_byte = startByte;
        if (i < 31) {
            fixed_byte = hashBytes[i + 1];
        }
        console.log(`i: ${i} byte: ${byte} fixed_byte: ${fixed_byte}`);
        if ((i == 31) || ((i & 1) != 0)) {
            hashBytes[10] = fixed_byte;
            if ((126 < byte) || (126 < hashBytes[10])) {
                if (byte < hashBytes[10] || (byte - hashBytes[10]) == 0) {
                    hashBytes[i] = hashBytes[10] - byte;
                } else {
                    hashBytes[i] = byte - hashBytes[10];
                }
            }
        } else if ((byte < 125) || (fixed_byte < 125)) {
            hashBytes[i] = fixed_byte + byte;
        }
    }
    return `${Buffer.from(hashBytes.slice(16)).toString("hex").toUpperCase()}`;
};

@bropat Nice work! 🎉 💪🏼
I just tried it in my script and works like a charm :D
So the expected outputs are indeed as the ones I shared last week  !

Thanks, @bropat and @martijnpoppen. Great work.
I had finished the image download from homebase part, do you need it @bropat?

ok, thank you very much.

soory for the interruption, the updated stuff is now added: previewImageHomeBase_v2-branch.
Please keep in mind, that you will have to adapt the implementation in your euyfsecurity.ts (at the end of the file, that's why I have commented it out) [I do historically some adaptations]. But the rest should work.

@bropat thanks for the update. Works perfect!
One thing: the initial Image is not working. Somehow the command CMD_DATABASE_IMAGE doesn't seem to work.

@bropat thanks for the update. Works perfect!
One thing: the initial Image is not working. Somehow the command CMD_DATABASE_IMAGE doesn't seem to work.

Is it me and did the imageUrl disappeared but no new property of Picture is showing up on the devices?
tried it with both devices that have RTSP stream and devices that dont have this? How to get this new property to show?
Or does that depends on the implentation of fuatakgun?

@kvanbiesen it's a new implementation ;)
imageUrl doesn't exist anymore

Great, appreciated @martijnpoppen

One thing: the initial Image is not working. Somehow the command CMD_DATABASE_IMAGE doesn't seem to work.

@martijnpoppen
I could only test it on my eufycam 2 with station and it works wonderfully ;)
With which device does it not work?
What error are you getting?

@bropat
Having a Doorbell Battery 2K, Indoor Cam PT and Eufy cam 2c here.
response:
[EufyLibrary] Station T8410REDACTED - Result data for command received {
  message: {
    sequence: 1,
    commandType: 1350,
    nestedCommandType: 1308,
    channel: 255,
    acknowledged: true,
    retries: 0,
    returnCode: -133,
    data: <Buffer d1 00 00 01 58 5a 59 48 46 05 d6 00 00 00 01 00 ff 00 00 00 7b 22 61 63 63 6f 75 6e 74 5f 69 64 22 3a 22 62 62 64 37 62 34 66 35 62 34 65 30 37 36 36 ... 184 more bytes>,
    customData: { command: [Object] }
  },
  resultCodeName: 'ERROR_PPCS_SUCCESSFUL',
  resultCode: 0
}
[EufyLibrary] Got p2p command response {
  station: 'T8410REDACTED',
  commandType: 1308,
  channel: 255,
  returnCodeName: 'ERROR_PPCS_SUCCESSFUL',
  returnCode: 0,
  customData: {
    command: {
      name: 'stationDownloadImage',
      value: '/media/mmcblk0p1/Camera00/20230106114405n.jpg'
    }
  }
}

@martijnpoppen
Have you installed an SD card with the Indoor Cam PT (T8410)?

If not, then your behaviour is correct, because the device itself has no memory to store the images/videos locally (no local database is created). You will also no longer see any events in the Eufy app for this device without an SD card.
If so, then it should work as it did for me and you should see events again in the Eufy Apps for this device.

@bropat Yes I have a SDcard :)
But maybe I misunderstood the functionality of this command?
As far as I understood it will download the cover_path and set that into the DevicePicture.

As far as I understood it will download the cover_path and set that into the DevicePicture.

Correct, but the Indoor Cam PT (T8410) hasn't the parameter cover_path ;)
In the next version of the library, the initial image will be fetched from the database (only if an sd card is inserted, formatted and working) for this device (already works):
https://github.com/bropat/eufy-security-client/commit/db5057289962d551af1fae3803fd422cd5fee75c


Refactor icu and init 74.2

ℹ This draft PR is my understanding of how the icu package could be improved in a similar vein as #289690.
TODO
If this PR is headed in the right direction and people see value in pursuing this approach close the following PRs

[ ] #289782
[ ] #289783

Description of changes

Add icu/default.nix containing the icu version specific packages
Remove icu/[0-9]*.nix
Use hash instead of sha256 in icu/base.nix
Includes changes proposed in #289782 icu73: use buildPackages.icu73 for nativeBuildRoot
Update all-packages.nix accordingly using callPackages

I'm uncertain whether the changes proposed here properly affect buildPackages.icu[0-9]{2} or whether more work is needed.
I'd appreciated if someone could chime in on this.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

ℹ icu, icu.passthru.tests, icu74, icu74.passthru.tests on x86_64-darwin failed due to error:** building of '/nix/store/1imr8dj9jf3f6j5sqg0sl1y45pmksi7m-llvm-16.0.6.drv^dev' from .drv file timed out after 3600 seconds

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/3468

I am confused why icu66 hash has changed here (I am fine with any reason, the impact is small, it;s just slightly mysterious…)

Thanks for pointing this out, @7c6f434c, the hashes have changed as I moved from sha256 to hash, the latter using SRI format (nix hash to-sri).
Going over the changes once again I notice that the hashes for icu67 and icu66 are the same, Yikes! Please allow me to doubletriplequadruple check and correct this. Be right back!🫨

Yeah I meant derivation hash change, the rest seem to be preserved (as SRI and non-SRI forms of sha256 are supposed to be treated as the same actual fixed-output requirement and thus not change the output hash). So I guess it's 66 and the new 74 that need the most attention, if the rest has the same builds as before and nobody has complained, probably they are fine…

I've corrected the hash for icu66 and checked the other hashes using:
grep version\ = pkgs/development/libraries/icu/default.nix > versions.txt
for v in $(cut -d'"' -f2 versions.txt); do print "version = \"$v\";"; nix-prefetch-url https://github.com/unicode-org/icu/releases/download/release-${v/./-}/icu4c-${v/./_}-src.tgz | xargs nix hash to-sri --type sha256; print; done 2>/dev/null

and then manually comparing output with the SRI 'hash' in icu/default.nix.
Is the derivation hash for icu66 now unchanged, @7c6f434c ?
During my checks I noticed that the URL (http://bugs.icu-project.org/trac/changeset/39484?format=diff) for the icu58 patch is no longer valid. It seems upstream has replace trac with Jira. Does this need to be addressed too or is the patch cached in Nix using the original sha256?

By the way, why a zero-preexisting-package-rebuild single-addition PR is towards staging? You didn't believe you would succeed to maintain the output paths?
(I hoped to get ofBorg checks on Darwin but apparently parts of staging are broken there)
(If you retarget, draft the PR before and re-ready it afterwards, to avoid triggering CODEOWNER notifications for the difference between staging and master)

Apologies, @7c6f434c, I'm uncertain whether I understand what you are asking for correctly, could rephrase and add more context?
Which branch would you recommend as this PRs merge base?

Sorry, I was assuming too much context here.
This PR does two things: a well done zero-rebuild (now verified to be zero-rebuild) refactoring of old ICU versions; and addition of a package with version 74 which as of this PR has no dependent packages. The total rebuild amount is 1.
For such changes it is usually safe to target master. On the other hand, you do not claim having built it on macOS, we do claim ICU to have wide platform support, and on master ofBorg would check for anything unexpected while on staging it is more complicated. Not a deal-breaker when there are reasons to target staging, but a benefit of targeting master when it is otherwise fine.
So I ask if there is any reason not to retarget to master branch.
If you do, please follow https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#rebasing-between-branches-ie-from-master-to-staging

My sincerest apologies everyone, while changing the merge base from staging to master, you all got pinged, when you shouldn't have. Even though I followed the instructions linked to by @7c6f434c I seem to have made a mistake somewhere along the way.
I'll close this PR and create a new one.

I'll reply to your previous comments here, @7c6f434c, after that—if your onboard with it—let's continue the conversation on the replacement PR #290266.
To answer your initial question "Why staging?" I had assumed that ICU being such an essential package there will be a lot of rebuilds. Is this something I could check upfront locally prior to creating a PR? If yes, how?
Thanks for the context, that's very helpful. Now I understand what you meant with zero-rebuild. Usually I build locally, but with the changes being on staging the rebuild took too long for the patience I had.
With the additional context and especially the verified zero-rebuild: No, I do not see a reason not to retarget to the master branch and have done so in #290266.
As you see from my comment above and this PR being close, something went wrong while rebasing the branches. Hopefully this doesn't disrupt too many peoples day… The 🌙 icon seems to sugggest that have not been pinged, but would have should this PR have been re-ready'd (thanks for the tip! :100:).


Add a method for getting signature of app.

Feature request
Hi, thanks for this great package.
it would be nice to add a simple method for getting signature of the app.

Why it is needed
For security reason and check if the app is decompressed and sign again with different certificate.

There are cloner apps in store that clone a fresh copy of your app in user device and user can enter the app as a new user. for some kind of apps like games it could break your analytics data and other metrics.
Some of these cloners uses this technique to create same app as yours but with different signature.
Possible implementation
A simple method that get your trusted signature value (Developer hard code this value in the app) and compare it with run-time returned value would be enough for check the integrity of the app.

Code sample
@RequiresApi(api = Build.VERSION_CODES.P)
@ReactMethod
  public void checkSignature(int trustedValue, Promise promise) {
    try {
	  Signature[] signatures = null;
	  PackageManager pm = this.reactContext.getPackageManager();
	  signatures = pm.getPackageInfo(this.reactContext.getPackageName(), PackageManager.GET_SIGNING_CERTIFICATES).signingInfo.getSigningCertificateHistory();
	  int value = 1;
	  for (Signature signature : signatures) {
		value *= signature.hashCode();
	  }
      promise.resolve(value == trustedValue);
    }
    catch (Exception ex) {
      ex.printStackTrace();
      promise.reject(null, ex.getMessage());
    }
  }

That's app related, not device-related. You might like a different package. I also do this package over here: https://github.com/mikehardy/react-native-update-apk/blob/eb04c572cc5b3701ce5279c68618e5069fd70385/example/App.js#L191-L197
But there are others

(PRs over there happily accepted, and if I recall from last time I looked there was at least one other module in react-native that handled APK stuff, I'm sure they'd do PRs if the API didn't fit either)

@mikehardy Sorry to bring this to life, but I need a package that does exactly this. That package (react-native-update-apk) suits even less then this just to get that information. Also, I didn't find any package that does this: "Gets the current app signature". If you did find one, can you please indicate me? Otherwise, I think it would be a good feature. Thank you

Package info is not device info, that's pretty clear. If react-native-update-apk doesn't suit, propose something there that does, happy to collaborate, but separation of concerns is an important software concept, I am and will follow it unapologetically


Layouts - Adding layout to empty layout causes original layout to disappear

See video.
http://screencast.com/t/w43o5iRg
Note: This is a UX problem, not necessarily a 'bug'. The area exists, but there's no indication of it until you attempt to drag something over the very narrow area.
http://screencast.com/t/yDyWeGUEbB

This is the expected behavior. All our areas behave this way currently.


Unification algorithm is slow

The current implementation of the blank node unification algorithm is slow. It runs in O(n²) in the mean case and O(2ⁿ) in the worst case where n is the number of blank nodes in the graph. It didn't matter for small examples, but it starting to show now that TreeLDR is used on real life problems.
TreeLDR is now used to generate the JSON-LD Context of Rebase. Because the model contains many anonymous intersection types/layouts giving rise to blank nodes, the unification algorithm takes some noticeable time to run (in the order a second). This model is not small but not particularly large either, we would expect the compiler to finish without noticeable delay.
The cause
The current implementation assumes the worst, where we have a cyclic graph of blank nodes. It attempts to merge all pairs of blank nodes by structurally comparing them (hence the O(n²) mean case). However for each pair of blank nodes a, b such that there is a property prop and two sets of blank nodes A, B such that for all ai ∋ A, a prop ai and for all bi ∋ B, b prop bi, then to prove the equivalence a ≈ b we must find a bijection R between A and B such that R(ai, bi) ⇒ ai ≈ bi. Finding such bijection is done exhaustively in O(2ⁿ). Most of the time the size of A and B will be 1, which mitigates this worst case scenario.
A preprocessing of the blank nodes also computes a coloring C for the blank nodes such that a ≈ b ⇒ C(a) = C(b). This coloring is used to avoid unnecessary comparisons.
Possible optimization: Trees Coloring
For non cyclic blank nodes (trees), it is possible to compute a coloring C such that C(a) = C(b) if and only if a ≈ b. This can be done in O(n) (or more precisely in O(k) where k is the number of transitions in the graph). Fortunately in most of the cases, blank nodes are trees (in fact, it is not possible to define non-trees blank nodes using the TreeLDR syntax).
Implementing a tree unification before the full unification could reduce the mean case from O(n²) to linear time computing.

Wrong repo used by the VSCode extension...


Send an email when the IP address is updated

Add a setting to send an email when the IP address is updated. The settings will be: emails, subject and the text of the message.

The change is done.


2024-12-10_01-37-31_master-next_aws-crt-python

Automatically created.

Successfully created backport PR for kirkstone-next:

#10664

Successfully created backport PR for scarthgap-next:

#10665

Successfully created backport PR for styhead-next:

#10666


VirusTotal

Useful app, but I'm worried that one of your antivirus software reports it as malware https://www.virustotal.com/gui/file/df6de6df2ab6279b623b916e44dc242cad72b1730509a13dee04420bcff16b47/detection.

Thanks for letting me know. I note that only one out of 70 antivirus engines came up as negative. And that one is Cynet, which, looking at their site claims to be a new AI- based approach capable of detecting zero day attacks. I couldn't see any way of reporting possible false negatives to them, so I'm not sure what I can do. I will keep looking.
I'm reassured by the 69 clean bills of health. I know I haven't put anything in there, but I can never rule out the possibility of something sneaky having got on my build machine. Do you get the same results if you build it yourself?

Thanks for letting me know. I note that only one out of 70 antivirus engines came up as negative. And that one is Cynet, which, looking at their site claims to be a new AI- based approach capable of detecting zero day attacks. I couldn't see any way of reporting possible false negatives to them, so I'm not sure what I can do. I will keep looking.
I'm reassured by the 69 clean bills of health. I know I haven't put anything in there, but I can never rule out the possibility of something sneaky having got on my build machine. Do you get the same results if you build it yourself?

It's good that you reported, it's worth eliminating false positive. Unfortunately, I don't know because I don't know how to build software :).

It's good that you reported, it's worth eliminating false positive. Unfortunately, I don't know because I don't know how to build software :).

Here is the response from Cyren:
Your case 00626226 has been closed.
The submitted content is not detected by Cyren. If at any point the content was detected by Cyren, this appears to be already corrected in the latest definition files version.
Please make sure to update to the latest AV definition files version.
So I asked VirusTotal to rescan the file, and now I get not just Cyren, but another positive, from MaxSecure. I will try another false positive submission.

Here is the response from Cyren:
Your case 00626226 has been closed.
The submitted content is not detected by Cyren. If at any point the content was detected by Cyren, this appears to be already corrected in the latest definition files version.
Please make sure to update to the latest AV definition files version.
So I asked VirusTotal to rescan the file, and now I get not just Cyren, but another positive, from MaxSecure. I will try another false positive submission.


Allow editing and mapping identity claims from Console UI

Describe the issue:
Currently, we have blocked the Console UI to not add any mapping to identity claims. We need to allow those mappings
How to reproduce:

Go to Console
Go to Attributes > OpenID Connect
Try to map an identity claim to an oidc attribute
UI will not show any identity claims, since the UI filters the attributes

This is applicable for other attribute dialects too (OIDC/custom/SCIM)
Expected behavior:

Environment information (Please complete the following information; remove  any unnecessary fields) :

Product Version: [e.g., IS 5.10.0, IS 5.9.0]
OS: [e.g., Windows, Linux, Mac]
Database: [e.g., MySQL, H2]
Userstore: [e.g., LDAP, JDBC]

Optional Fields
Related issues:

Suggested labels:

Fixed with https://github.com/wso2/identity-apps/pull/2980


AzureRM Container App Terraform Provider

Just a after a bit of info as to the rational of using the azapi resource in order to provision the Azure Container App Environment over the Terraform AzureRM provider?
It appears the provider meets "most" of the use cases for the service albeit a little buggy. The concern here is that if we adopt the AVM module for the ACA environment, it would make it difficult to move to the native TF provider.
Just really after the rational around the decision. Thanks!

Hi.  I'm the author.  On holiday at the moment.  The rationale was when i started the azurerm provider was lacking significant functionality, that has since been addressed.  I don't have any immediate plans to convert this to azurerm nor see any immediate need to do so, but open to more feedback on specific concerns.

hi @tbarkerwoolworths - just wondering if you can elaborate on the functionality gaps and bugs?  You're correct in that a conversion to azurerm would be a breaking change, but azapi is fully supported.

@matt-FFFFFF i see container apps took the azurerm path - what's your thoughts here - should we convert to azurerm with a breaking change?  i see some of the concerns i previously had with azapi being addressed with v2, but I also see merit in aligning with the container apps res module.


Remove specs for the deprecated service machinelearningexperimentation

Remove specs for the deprecated service machinelearningexperimentation
ARM (Control Plane) API Specification Update Pull Request

[!TIP]
Overwhelmed by all this guidance? See the Getting help section at the bottom of this PR description.

[!NOTE]
As of January 2024 there is no PR assignee. This is expected. See https://aka.ms/azsdk/pr-arm-review.

PR review workflow diagram
Please understand this diagram before proceeding. It explains how to get your PR approved & merged.

Click here to see the details of Step 1, Breaking Changes review
If you are in purview of Step 1 of the diagram, follow the Breaking Changes review process.
IMPORTANT! This applies even if you believe your PR was mislabeled, for any reason, including tool failure.

Click here to see the details of Step 2, ARM review
See https://aka.ms/azsdk/pr-arm-review.

 Click here to see the diagram footnotes
Diagram footnotes
[1] See ARM review queue (for PR merge queues, see [2]).
[2] public repo merge queue, private repo merge queue (for ARM review queue, [1])
The ARM reviewer on-call engineer visits the merge queue twice a day, so the approximate ETA for merges is 12 - 24 hours.

Purpose of this PR
What's the purpose of this PR? Check the specific option that applies. This is mandatory!

[ ] New resource provider.
[ ] New API version for an existing resource provider. (If API spec is not defined in TypeSpec, the PR should have been generated using OpenAPI Hub).
[ ] Update existing version for a new feature. (This is applicable only when you are revising a private preview API version.)
[ ] Update existing version to fix OpenAPI spec quality issues in S360.
[ ] Other, please clarify:

edit this with your clarification

Due diligence checklist
To merge this PR, you must go through the following checklist and confirm you understood
and followed the instructions by checking all the boxes:

[ ] I confirm this PR is modifying Azure Resource Manager (ARM) related specifications, and not data plane related specifications.
[ ] I have reviewed following Resource Provider guidelines, including
ARM resource provider contract and
REST guidelines (estimated time: 4 hours).
I understand this is required before I can proceed to the diagram Step 2, "ARM API changes review", for this PR.

Additional information

 Viewing API changes
For convenient view of the API changes made by this PR, refer to the URLs provided in the table
in the Generated ApiView comment added to this PR. You can use ApiView to show API versions diff.

Suppressing failures
If one or multiple validation error/warning suppression(s) is detected in your PR, please follow the
suppressions guide to get approval.

Getting help

First, please carefully read through this PR description, from top to bottom. Please fill out the Purpose of this PR and Due diligence checklist.
To understand what you must do next to merge this PR, see the Next Steps to Merge comment. It will appear within few minutes of submitting this PR and will continue to be up-to-date with current PR state.
For guidance on fixing this PR CI check failures, see the hyperlinks provided in given failure
and https://aka.ms/ci-fix.
For help with PR workflow diagram Step 2 (ARM review), see https://aka.ms/azsdk/pr-arm-review.
If the PR CI checks appear to be stuck in queued state, please add a comment with contents /azp run.
This should result in a new comment denoting a PR validation pipeline has started and the checks should be updated after few minutes.
If the help provided by the previous points is not enough, post to https://aka.ms/azsdk/support/specreview-channel and link to this PR.

Per ARM RP contact dashboard, the RP microsoft.machinelearningexperiment does not exist. And there is only very old preview (2017-05-01-perview) in the RP folder.  There is no SDK released upon the preview version.  We decide to clean up the RP folder to avoid any waste of maintenance.

No need breaking change review and ARM Review as the RP microsoft.machinelearningexperiment no longer exists.
Add the label of ARM-Signoff to suppress the requirement for ARM-Review

@mikekistler  @JeffreyRichter,  could you help to approve the breaking change?
The RP does not exist for long time, and we are cleaning up the spec for it.


List of Configuration Properties is outdated

Page
https://docs.alluxio.io/os/user/stable/en/reference/Properties-List.html
Summary
The List of Configuration Properties is outdated for Alluxio v2.3.
Many new properties have been added in version 2.3, but their description in the List of Configuration Properties is missing. Among such new properties, that are missing description, are the user properties for streaming:
alluxio.user.streaming.data.timeout
alluxio.user.streaming.reader.buffer.size.messages
alluxio.user.streaming.reader.chunk.size.bytes
alluxio.user.streaming.writer.buffer.size.messages
alluxio.user.streaming.writer.chunk.size.bytes
alluxio.user.streaming.writer.close.timeout
alluxio.user.streaming.writer.flush.timeout
alluxio.user.streaming.zerocopy.enabled

It would be helpful if the list of properties has been updated for the latest production version, so that people can fully use the new features.

Thanks @jpohanka . This should be an easy fix. I'll link a PR with the updates

Thank you very much for the fast response @ZacBlanco :) .

@jpohanka The PR was merged, this should reflect on the website within 24-48 hours


CI test next version

Coverage decreased (-0.3%) to 96.516% when pulling 246a8033c8bcd0f1f982c09d70d98b0837a9e3db on develop into d95a2dfbaa0017cee73493ef3593e0bde5cb5f22 on master.


Test BootableJar using Channel Manifest

This PR:

reintroduces bootable jar tests
adds option to run bootable jar tests with a channel manifest

The test failures seem to be related with the galleon update.

@jbliznak can you have a look?

I guess after @jamezp comes back he will check this.

just rebased on main and have this error (in main too):
$ mvn clean install -DskipTests
...
[ERROR] Failed to execute goal on project resteasy-core-spi: Could not resolve dependencies for project org.jboss.resteasy:resteasy-core-spi:jar:7.0.0.Alpha2-SNAPSHOT: Failed to collect dependencies at jakarta.ws.rs:jakarta.ws.rs-api:jar:4.0.0: Failed to read artifact descriptor for jakarta.ws.rs:jakarta.ws.rs-api:jar:4.0.0: The following artifacts could not be resolved: jakarta.ws.rs:jakarta.ws.rs-api:pom:4.0.0 (absent): Could not transfer artifact jakarta.ws.rs:jakarta.ws.rs-api:pom:4.0.0 from/to central-libs-release (https://repository.eapqe.psi.redhat.com:8081/artifactory/libs-release): Unsupported or unrecognized SSL message -> [Help 1]

just re-base on main and with both main and this branch I get:
$ mvn clean install -DskipTests
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------< org.jboss.resteasy:resteasy-core-spi >----------------
[INFO] Building RESTEasy Core SPI 7.0.0.Alpha2-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central-libs-release: https://repository.eapqe.psi.redhat.com:8081/artifactory/libs-release/jakarta/ws/rs/jakarta.ws.rs-api/4.0.0/jakarta.ws.rs-api-4.0.0.pom
Downloading from libs-snapshot: https://repository.eapqe.psi.redhat.com:8081/artifactory/libs-snapshot/jakarta/ws/rs/jakarta.ws.rs-api/4.0.0/jakarta.ws.rs-api-4.0.0.pom
Downloading from artifactory: https://repository.eapqe.psi.redhat.com:8081/artifactory/all/jakarta/ws/rs/jakarta.ws.rs-api/4.0.0/jakarta.ws.rs-api-4.0.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.514 s
[INFO] Finished at: 2024-04-15T10:49:49+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project resteasy-core-spi: Could not resolve dependencies for project org.jboss.resteasy:resteasy-core-spi:jar:7.0.0.Alpha2-SNAPSHOT: Failed to collect dependencies at jakarta.ws.rs:jakarta.ws.rs-api:jar:4.0.0: Failed to read artifact descriptor for jakarta.ws.rs:jakarta.ws.rs-api:jar:4.0.0: The following artifacts could not be resolved: jakarta.ws.rs:jakarta.ws.rs-api:pom:4.0.0 (absent): Could not transfer artifact jakarta.ws.rs:jakarta.ws.rs-api:pom:4.0.0 from/to central-libs-release (https://repository.eapqe.psi.redhat.com:8081/artifactory/libs-release): Unsupported or unrecognized SSL message -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

We can talk offline about this, but I need to understand the motivation behind this. I've got some opinions on how we test the bootable JAR that I might share alone, but I don't think we do it right :)

Just an FYI I left a review on #4108, but that would apply here too. For now we could focus on one PR so you don't have to keep updating both and cherry-pick it after we're satisfied. I'm happy to handle the cherry-pick as well.


3.1 stable - Lambda support in collections

Changes needed to be able to use lambda type collections

Hi! I'm quite surprise this haven't worked before. Could you support it by writing a test?
Returning the collection early means the label/id transformations won't be applied. Not sure if that is intentional or not.
Also does this exist on master that it targets 3.1 ?

Closing due to inactivity and no test case. Happy to re-open or revisit with a test case (or even a comment that shows some sample code that would help someone else write a test case)!


UI: more linkage

Kargo UI needs more linkage:

to the Argo CD application
to the application git repository tag (if applicable)
to the git repository (and possibly commits)
to the container registry (if docker hub, ghcr.io, quay.io, etc...)
to the helm repository

For 1 (the argo cd application), it requires us to expose a Kargo setting to know the base URL(s) of Argo CD instances.


updating sample tutorial due to sample update

Reasons for PR:
-The sample that this tutorial covers has been updated.
-The app registration experience on the AAD portal has changed.
-Visual Studio references are not really necessary and rather distracting.

@derisen : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@ShannonLeavitt : Thanks for your contribution! The author(s) have been notified to review your proposed change.

rerouting to @hamiltonha

Hi @hamiltonha - Could you review the proposed changes? Thanks.

@derisen : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@derisen : Thanks for your contribution! The author(s) have been notified to review your proposed change.

may I have your approval for this @hamiltonha ?

Thank you, @derisen and reviewers. We'll move this commit to the private repository so we can verify the formatting changes on staging before we merge the PR. We'll post the new PR number here, and then close this PR. We'll try to do this today.
FYI, "Microsoft identity platform" needs to stay as-is for accuracy in branding. See the hub page for Microsoft identity platform: https://docs.microsoft.com/en-us/azure/active-directory/develop/. We'll make this change in the private repo.
Thanks!

The commit in this PR has been moved to the private repo for validation and staging. It should be merged and and the updates live within one day. https://github.com/MicrosoftDocs/azure-docs-pr/pull/105913. We'll close this PR. Thanks.


Feature pytest

Second pull request for feature_pytest (I closed the first one).
- test_flights.py (using unittest framework) replaces run_test.py
- github action now only uploads results folder (.ros/logs and bagfiles already copies in results so no need to upload them separately)
- results folder is now uploaded (step 6) whether test pass or fail (but of course not when the workflow is cancelled)
-  plotter class now corrects the 0.3 m X-axis offset in the pdf of the multitrajectory test (and other details)

Thanks!


Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry.

cognitiveservices service call takes more than 10 minutes also getting below error. Is it azure server related?
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry.'

The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.

@akmichael  Please help with the link to the document on which the above document feedback request is created.
In case your query is not related to document content, open a new forum thread in Microsoft Q&A or Stack Overflow as it could be better place to get help on your scenario. These forum community members could provide their expert guidance on your scenario based on their experience. Thanks.

@akmichael Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

@CHEEKATLAPRADEEP-MSFT   I'm also facing same issue System.AggregateException: 'Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The requested name is valid, but no data of the requested type was found. (myservice.cognitiveservices.azure.com:443)), any idea what could be wrong


Preserve data

Hello, dear reviewer.
By completing this project we met the following requirements:
Project requirements
Checked the documentation for Ruby's JSON library.
On exit from our app saved all the data (books, people, and rentals) into JSON files. One per type of data you are storing and named accordingly (e.g. rentals.json)
On startup of our app load all data from files. Remembered to handle the case when one or more of the files were missing.

Thank you very much for your feedback.


Using czmq package with correct targets

As discussed in #746 I fixed target find and export. I also set building static libraries as default behaviour.
I removed version 3.0.0, a lot of problems with targets. Since hunter czmq package is available since two days ago I think no one would be using such an old version.

Testing:

https://ci.appveyor.com/project/ingenue/hunter/build/1.0.1861
https://travis-ci.org/ingenue/hunter/builds/245334733?utm_source=github_status&utm_medium=notification

Uploading:

https://ci.appveyor.com/project/ingenue/hunter/build/1.0.1862
https://travis-ci.org/ingenue/hunter/builds/245336298?utm_source=github_status&utm_medium=notification

Released: https://github.com/ruslo/hunter/releases/tag/v0.19.19


DMP-3345 Fixing concurrent daily list processing

Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ ] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
Change description
Problem:
The daily list can be trigger using the POST /dailylists/run endpoint. This endpoint is useful for running daily lists for a specific listing courthouse and is (or will be) locked down to SUPER_ADMIN only.
Solution:
Utilise the automated tasks locking mechanism when running the daily list processing, to ensure that it's not already locked by the automated task running on schedule or another manual execution.

the results in calls to POST /dailylists/run getting a 409 response if the task is already locked, preventing concurrent executions.

Implementation:
Exposing some methods from the automated task package through an "api" interface.
Adding a LockService, exposed within the "api" package.

used by the daily list controller to check if the task is locked
used by the daily list processor to execute the task with a lock

Using the new LockService within the automated tasks

to get the locking task executor
to get the default lock times

Additionally:

updating DarNotifyError constructor to call super
using ConcurrentMap in SystemUserHelper - to fix a ConcurrentModificationException that was seen in the logs

Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

@chrisbellingham-hmcts I've rebased this and fixed up conflicts. It's been a while since your first review, but would you mind giving it another review please?