Intregration Daemons and Generic Handlers

This pull request primarily delivers the integration daemon OMAG server and support for the database integrator integration service.  This supports the automatic capture of relational database metadata. (#2512)
Testing this work uncovered a number of issues and missing functions needed to support the expansion out of integration services to supporting asset managers and other third party technologies.
The first was a series of issues in the open types (#3701, #3505, #3332).  This impacted the support for the OCF handlers.  These have been replaced with generic handlers to allow the common functions to be used beyond the types supported by OCF (#3703).
The mechanism for supporting latest changes, templated asset curation, anchor management, external instances and cascaded deletes needed to be improved (#3308).  This has been implemented in the generic handlers.
As the generic handlers were being worked through, a circular dependency between the handlers, metadata security and the OCF occurred because the metadata security module used OCF beans in its API.  This has been address by giving the metadata security module its own tailored beans, simplifying the interface.  I also added support for connection selection to the security module (#3419).
This PR included support for search keywords (#3425).

In total I had 44 review comments but they do not all seem to have made it through review submission - I still have the review page open and will try to get the rest of them across.

Well I guess that's a fail then. I'm not going to repeat the review so just for the record, my comments were generally around Javadoc header comments referring to a different class name (a simple copy-paste problem), toString methods also naming incorrect class (copy-paste problem), and a few other typos in comments and readme files such as word repeats and grammatical typos. The more important ones though were around serialisation of classes that contain a boolean property field. In some places these fields are named isPublic, isPrivate, etc. and the getter and setter methods for isPublic are called isPublic() and setPublic(). This will serialise and deserialise OK but the property field in the serialised JSON will be called 'public' rather than matching the Java field names. This is because Jackson will strip the "is" prefix. If another type of user (via REST or Python) is parsing the JSON, it will not match what they might expect. OK if we document the serialised format, but possibly subtle otherwise. My preference is to use rather explicit getter and setter methods (always prefixed with get or set) and use a simple property name rather than prefixed with "is".
I think it would be a lot better to avoid massive PRs, they are much harder to review both in terms of time and the ease for each reviewer to find the changes they are best placed to review. Also it seems that with a PR with so many files (1399) the GitHub tools start to struggle. I spent a whole day reviewing and although I still have the review page open with my comments, there is no easy way to find the comments within the (collapsed) files that I reviewed (1004). It would be far better to break up work into smaller PRs, if only to help the reviewers.

Thank you for the comments.  I did get them in emails so I think they are there.  On the javadoc and toString problems, most of the ones in the first commit are fixed - they were only in the first commit because intelliJ forces a git add whenever you rename or delete a file and each file refactor often requires manual update of the Javadoc at the top of the file.  This then comes in a later commit.
On the boolean properties in the Java beans, they are only ever used through the Java bean as we do not support direct calls through the REST APIs.  However, I agree we should be prepared to support the REST APIs directly in the future since we see many architects who are evaluating Egeria try to call it through the REST APIs.  We are also using the REST APIs thrugh the notebooks.   I also favour going with getIsPublic() and setIsPublic() style because it is clear then how it maps to the open types rather than continuing with intelliJ's approach.  This issue is reported in #2312 but I will pull the fix into this PR since it is reported in the review comments.
I agree, in general, on the large PRs - however in this case it was unavoidable because integration daemon (actually what this PR is delivering) needed to use the Data Manager OMAS for testing.  This uncovered issues in our support for schemas that impacted OCF and the open types and I was then changing a middle layer of code to create the new handlers.  It was only by migrating the 5 OMASs to the new handlers that I was able to flush out all of the patterns that the generic handlers had to support.  The testing also revealed other issues in the modules I was changing and in those both above and below where I was working.  For example, there were problems in the the admin services, repository services as well as the documentation since new features had been going in without updating it.
The alternative would be to push untested code into master which would have been fine a year ago but I think is less appropriate now.

I take back the comment I made about the javadoc errors in the beans - I have just gone through changing all of the boolean getters and setters in Asset Consumer OMAS and Asset Owner OMAS and found a number of errors in the class name in the Javadoc and toString

I thought it would be useful to build our docker images and validate the container environment.
However when following the git instructions command provided in the PR to test , I get a compile error:
12:33:13,168 [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ tex-view-server ---
12:33:13,176 [INFO] Changes detected - recompiling the module!
12:33:13,180 [INFO] Compiling 6 source files to /Users/jonesn/IdeaProjects/egeria-maven/open-metadata-implementation/view-services/tex-view/tex-view-server/target/classes
12:33:13,319 [INFO] -------------------------------------------------------------
12:33:13,320 [ERROR] COMPILATION ERROR :
12:33:13,320 [INFO] -------------------------------------------------------------
12:33:13,320 [ERROR] /Users/jonesn/IdeaProjects/egeria-maven/open-metadata-implementation/view-services/tex-view/tex-view-server/src/main/java/org/odpi/openmetadata/viewservices/tex/handlers/TexViewHandler.java:[187,68] cannot find symbol
  symbol:   variable VIEW_SERVICE_NULL_PLATFORM_NAME
  location: class org.odpi.openmetadata.commonservices.ffdc.OMAGCommonErrorCode
12:33:13,322 [INFO] 1 error
12:33:13,322 [INFO] -------------------------------------------------------------

This is likely due to more recent changes in master than you currently have in the PR (the instructions you can copy/paste include merging from master -- on the basis that prior to merging a PR it needs to be up to date in any case).

Just checking out this branch (without merging from master) I get the following error in the config notebook:
   ... configuring the dino Integration View Service for this server...
   ...... (POST https://dev:9443/open-metadata/admin-services/users/garygeeke/servers/cocoView1/view-services/dino )
   ...... Response:  {'class': 'VoidResponse', 'relatedHTTPCode': 400, 'exceptionClassName': 'org.odpi.openmetadata.adminservices.ffdc.exception.OMAGInvalidParameterException', 'actionDescription': 'configureViewService', 'exceptionErrorMessage': 'OMAG-ADMIN-400-039 The configureViewService service of OMAG server cocoView1 has been configured with a null root URL for its remote OMAG Server', 'exceptionErrorMessageId': 'OMAG-ADMIN-400-039', 'exceptionErrorMessageParameters': ['configureViewService', 'cocoView1'], 'exceptionSystemAction': 'The system is unable to accept this value in the configuration document because the server would not be able to operate correctly.', 'exceptionUserAction': 'The root URL is supplied by the caller to the OMAG server. This URL value needs to be corrected before the server can operate correctly.'}

I think there was a change added to the admin server. This was known working in the 2.3 release. The dino view service does not utilize the root URL. It appears validation has changed - so either we'll need to set it in the notebook or change the validation.
It's possible this is a different change on master that this branch has picked up.

Step #5 on the 'building a data catalog' notebook appears to fail on 'assetOwnerPrintAssets'

assetOwnerPrintAssets(cocoMDS1Name, cocoMDS1PlatformName, cocoMDS1PlatformURL, petersUserId, ".*file.*")

1 asset found:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-d31c7baee9fc> in <module>
----> 1 assetOwnerPrintAssets(cocoMDS1Name, cocoMDS1PlatformName, cocoMDS1PlatformURL, petersUserId, ".*file.*")

<ipython-input-1-390f80ab5b28> in assetOwnerPrintAssets(serverName, serverPlatformName, serverPlatformURL, userId, searchString)
    183         else:
    184             print (str(len(assets)) + " assets found:")
--> 185         printAssetListSummary(assets)
    186         printAssetListDetail(assets)
    187

<ipython-input-1-390f80ab5b28> in printAssetListSummary(assets)
     38     print(" ")
     39     for x in range(len(assets)):
---> 40         printAssetSummary(assets[x])
     41
     42 def printAssetListDetail(assets):

<ipython-input-1-390f80ab5b28> in printAssetSummary(asset)
      5 def printAssetSummary(asset):
      6     elementType = asset.get('type')
----> 7     assetType = elementType.get('elementTypeName')
      8     assetGUID = asset.get('guid')
      9     assetQualifiedName = asset.get('qualifiedName')

AttributeError: 'NoneType' object has no attribute 'get'

Step #3 on 'understanding an asset' also seems to fail with:
Created Patient Asset
No comment added
Unexpected response from server platform Data Lake Platform at https://datalake:9443

Response:
{
    "timestamp": "2020-10-06T12:49:43.684+00:00",
    "status": 400,
    "error": "Bad Request",
    "message": "",
    "path": "/servers/cocoMDS1/open-metadata/access-services/asset-consumer/users/peterprofile/assets/3781dab8-b5c0-4a18-9339-b65b928fc677/comments"
}

Setting owner on Patient Asset to tanyatidie ...
Setting governance zones on Patient Asset ...
Done.

In the server log I see:
core_1          | Tue Oct 06 12:49:47 GMT 2020 cocoMDS2 Exception OMAG-REPOSITORY-HANDLER-0003 An unexpected error org.odpi.openmetadata.repositoryservices.ffdc.exception.PropertyErrorException was returned to reclassifyEntity by the metadata server during updateAssetZones request for open metadata access service Asset Owner OMAS on server cocoMDS2; message was OMRS-REPOSITORY-400-028 A property called zoneMembership has been proposed for a new metadata instance of category ClassificationDef and type AssetOrigin; it is not supported for this type in open metadata repository cocoMDS2
core_1          | Tue Oct 06 12:49:47 GMT 2020 cocoMDS2 Exception OMAG-REPOSITORY-HANDLER-0003 Supplementary information: log record id 19d3b0a5-dcf2-4c81-a214-1c05025c7ca5 org.odpi.openmetadata.repositoryservices.ffdc.exception.PropertyErrorException returned message of OMRS-REPOSITORY-400-028 A property called zoneMembership has been proposed for a new metadata instance of category ClassificationDef and type AssetOrigin; it is not supported for this type in open metadata repository cocoMDS2 and stacktrace of
core_1          | OCFCheckedExceptionBase{reportedHTTPCode=400, reportingClassName='org.odpi.openmetadata.repositoryservices.localrepository.repositorycontentmanager.OMRSRepositoryContentValidator', reportingActionDescription='updateEntityClassification', reportedErrorMessage='OMRS-REPOSITORY-400-028 A property called zoneMembership has been proposed for a new metadata instance of category ClassificationDef and type AssetOrigin; it is not supported for this type in open metadata repository cocoMDS2', reportedErrorMessageId='OMRS-REPOSITORY-400-028', reportedErrorMessageParameters=[zoneMembership, ClassificationDef, AssetOrigin, cocoMDS2], reportedSystemAction='The system is unable to store the metadata instance in the metadata repository because the properties listed do not match the supplied type definition.', reportedUserAction='Verify that the property name is spelt correctly and the correct type has been used. Correct the call to the metadata repository and retry.', reportedCaughtException=null, reportedCaughtExceptionClassName='null', relatedProperties=null}
core_1          | 	at org.odpi.openmetadata.repositoryservices.localrepository.repositorycontentmanager.OMRSRepositoryContentValidator.validatePropertiesForType(OMRSRepositoryContentValidator.java:2061)
core_1          | 	at org.odpi.openmetadata.repositoryservices.localrepository.repositorycontentmanager.OMRSRepositoryContentValidator.validateClassificationProperties(OMRSRepositoryContentValidator.java:1588)
core_1          | 	at org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.OMRSMetadataCollectionBase.classifyEntityParameterValidation(OMRSMetadataCollectionBase.java:4029)
core_1          | 	at org.odpi.openmetadata.repositoryservices.enterprise.repositoryconnector.EnterpriseOMRSMetadataCollection.updateEntityClassification(EnterpriseOMRSMetadataCollection.java:3277)
core_1          | 	at org.odpi.openmetadata.commonservices.repositoryhandler.RepositoryHandler.reclassifyEntity(RepositoryHandler.java:1833)
core_1          | 	at org.odpi.openmetadata.commonservices.generichandlers.OpenMetadataAPIGenericHandler.setClassificationInRepository(OpenMetadataAPIGenericHandler.java:292)
core_1          | 	at org.odpi.openmetadata.commonservices.generichandlers.OpenMetadataAPIGenericHandler.setClassificationInRepository(OpenMetadataAPIGenericHandler.java:194)
core_1          | 	at org.odpi.openmetadata.commonservices.generichandlers.AssetHandler.updateAssetZones(AssetHandler.java:1042)
core_1          | 	at org.odpi.openmetadata.accessservices.assetowner.server.AssetOwnerRESTServices.updateAssetZones(AssetOwnerRESTServices.java:1468)

Step 3 on the open-discovery notebook fails with:
assetDeduplicatorEngineName = "AssetDeduplicator"
assetDeduplicatorEngineDisplayName = "Asset Deduplicator Discovery Engine"
assetDeduplicatorEngineDescription = "Discovery engine for scanning the asset catalog and identifying which assets are duplicate definitions of the same physical asset."

assetDeduplicatorEngineGUID = createDiscoveryEngine(cocoMDS1Name,
                                                    cocoMDS1PlatformName,
                                                    cocoMDS1PlatformURL,
                                                    petersUserId,
                                                    assetDeduplicatorEngineName,
                                                    assetDeduplicatorEngineDisplayName,
                                                    assetDeduplicatorEngineDescription)

print (" ")
print ("The guid for the " + assetDeduplicatorEngineName + " discovery engine is: " + assetDeduplicatorEngineGUID)
print (" ")

OMAG-REPOSITORY-HANDLER-404-010 Multiple OpenDiscoveryEngine entities where found with a name of AssetDeduplicator: the identifiers of the returned entities are []; the calling method is getDiscoveryEngineByName, the name parameter is name and the server is cocoMDS1
 * The system is unable to process a request because multiple entities have been discovered and it is unsure which entity to use.
 * Investigate why multiple entities exist.  Then retry the request once the issue is resolved.
OMAG-REPOSITORY-HANDLER-500-001 An unexpected error org.odpi.openmetadata.repositoryservices.ffdc.exception.PropertyErrorException was returned to createEntity by the metadata server during createDiscoveryEngine request for open metadata access service Discovery Engine OMAS on server cocoMDS1; message was OMRS-REPOSITORY-400-028 A property called displayName has been proposed for a new metadata instance of category EntityDef and type SoftwareServerCapability; it is not supported for this type in open metadata repository cocoMDS1
 * The system is unable to process the request because of an internal error.
 * Verify the sanity of the server.  This is probably a logic error.  If you can not work out what happened, ask the ODPi Egeria community for help.

Will re-check these tests on master.

@planetf1 do not waste time in testing the notebooks - I am working through them - we only created the PR to allow the build problems to be shared

@planetf1 do not waste time in testing the notebooks - I am working through them - we only created the PR to allow the build problems to be shared

Apologies for the misunderstanding. I'll test again once the PR is ready for final review. Have deleted the test results to avoid cluttering the build discussion

I think using the builder design pattern will make code more efficient and easier to understand.
There are also methods from repository helper  for creation, where using java builder design pattern would help a lot.
For example, instead of create Classification like this, with many parameter and 3 of them  null:
Classification classification = repositoryHelper.getNewClassification(serviceName,
                                                                                null,
                                                                                null,
                                                                                InstanceProvenanceType.LOCAL_COHORT,
                                                                                userId,
                                                                                OpenMetadataAPIMapper.ASSET_OWNERSHIP_CLASSIFICATION_NAME,
                                                                                typeName,
                                                                                ClassificationOrigin.ASSIGNED,
                                                                                null,
                                                                                getOwnerProperties(userId,
                                                                                owner,
                                                                                ownerType,
                                                                                methodName));

it can be used a builder in the sense of builder design pattern.

I noticed the failure
[ERROR] 'dependencies.dependency.version' for org.odpi.egeria:open-metadata-assemblies:pom:distribution is missing. @ line 72, column 21

The other modules were updated in a recent fix for #3752
Remove the specification of the classifier
            <classifier>distribution</classifier>

ie leaving the dependency for the assembly as
        <dependency>
            <groupId>org.odpi.egeria</groupId>
            <artifactId>open-metadata-assemblies</artifactId>
            <type>pom</type>
            <scope>test</scope>
        </dependency>

Update
Regression testing fo Data Engine OMAS is still in progress there are issues in regards to use of deprecated update methods causing UserNotAuthorizedException so far not straight forward to fix. We are still trying to do it and fix forward the code. Testing continues today. Keep you posted.

Thanks @lpalashevski - which update method is causing the problem - if it is problematic, could I restore the method causing the problem to its previous behaviour - or would you prefer to continue to move foreward?
Any call can return UserNotAuthorizedException when the security plugin is enabled so it may be better to continue to work to support this exception?

Thanks @lpalashevski - which update method is causing the problem - if it is problematic, could I restore the method causing the problem to its previous behaviour - or would you prefer to continue to move foreward?
Any call can return UserNotAuthorizedException when the security plugin is enabled so it may be better to continue to work to support this exception?

DE OMAS is using the old updateEntity method from the repository handler, that has no externalSourceName & externalSourceGUID specified. Also, DE is using SchemaTypeHandler - updateSchemaType & updateSchemaAttributes methods that don't specify the externalSourceName&externalSourceGUID. Similar methods need to be created for updating external entities.
I will create an issue to address this and commit the changes after this PR is merged. We're still in the middle of testing and we will come back with a final result later today

We also see  different exception in OMRSRepositoryHelper being used by IGC (IGCOMRSRepositoryEventMapper and IGCRepositoryHelper)
java.lang.NoSuchMethodError: org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector.OMRSRepositoryHelper.getNewClassification
This is breaking also Asset Catalog OMAS and fails related federated calls going to the remote repository.
Looks like we need to align the implementation ing IGC connectors.
@mandy-chessell @cmgrote

I created issue #3843 for handling the changes needed in DE

This is great feedback.  I need to restore any OMRSRepositoryHelper methods that I have been inadvertently deleted since these changes should not affect the IGC connector.  I will also add the update methods back into handlers.

I notice there are some gradle conflicts after I consolidate the version files.
To fix, use intellij search (it should prompt once pasted to use regex) and replace using file pattern 'build.gradle'
Search string: implementation \'([\w\.]+):([\w\-]+):([\w\-]+).*\'
Replace string: implementation '$1:$2'
This strips the version specification and is likely the only reason for those conflicts beyond deleted/moved files
There are separate updates to the top level build.gradle (dependency versions) & settings.gradle (plugin versions) but I would expect auto-merging would work fine there.

I think using the builder design pattern will make code more efficient and easier to understand.
There are also methods from repository helper for creation, where using java builder design pattern would help a lot.
For example, instead of create Classification like this, with many parameter and 3 of them null:
Classification classification = repositoryHelper.getNewClassification(serviceName,
                                                                                null,
                                                                                null,
                                                                                InstanceProvenanceType.LOCAL_COHORT,
                                                                                userId,
                                                                                OpenMetadataAPIMapper.ASSET_OWNERSHIP_CLASSIFICATION_NAME,
                                                                                typeName,
                                                                                ClassificationOrigin.ASSIGNED,
                                                                                null,
                                                                                getOwnerProperties(userId,
                                                                                owner,
                                                                                ownerType,
                                                                                methodName));

it can be used a builder in the sense of builder design pattern.

The repository helper is a released interface since it is used by repository connectors.  It can be extended but not radically changed.  I could turn this particular method into 2 methods to support the two types of classifications (assigned and propagated).  That would eliminate one of the nulls since that parameter would only be needed on one of the new method calls.  The other two nulls are only nulls temporarily, waiting for all OMASs to support provenance properly.  When they do, the first 2 values will be filled out.

I noticed the failure
[ERROR] 'dependencies.dependency.version' for org.odpi.egeria:open-metadata-assemblies:pom:distribution is missing. @ line 72, column 21

The other fvt modules were updated in a recent fix for #3752, so for the data manager fvt remove the specification of the classifier
            <classifier>distribution</classifier>

ie leaving the dependency for the assembly as
        <dependency>
            <groupId>org.odpi.egeria</groupId>
            <artifactId>open-metadata-assemblies</artifactId>
            <type>pom</type>
            <scope>test</scope>
        </dependency>

Where is this error - I am guessing it is in a POM file but wondered which one?

@cmgrote @lpalashevski is this the getNewClassification() method that needs to be reinstated in the RepositoryHelper?
    public Classification getNewClassification(String               sourceName,
                                               String               userName,
                                               String               typeName,
                                               String               entityTypeName,
                                               ClassificationOrigin classificationOrigin,
                                               String               classificationOriginGUID,
                                               InstanceProperties   properties) throws TypeErrorException

The changes I have made to the SchemaTypeHandler is to change it so that updateSchemaType and updateSchemaAttribute calls the original repository handler method that does not do the provenance checking.

The other fvt modules were updated in a recent fix for #3752, so for the data manager fvt remove the specification of the classifier

Where is this error - I am guessing it is in a POM file but wondered which one?

Click on 'details' next to one of the builds like Java 11
Click on link at bottom - view more details on azure pipelines
Click on the maven task in the middle pane
scroll to find error

The current error is about SchemaTypeAttribute
The maven error was relating to use of classifier, so for example one can do:
find . -name 'pom.xml' -print | xargs -n50 grep -y 'classifier'

from the top level -- other than the main pom
The specific error that I saw from an earlier build was in open-metadata-test/open-metadata-fvt/access-services-fvt/data-manager-fvt

@cmgrote @lpalashevski is this the getNewClassification() method that needs to be reinstated in the RepositoryHelper?
    public Classification getNewClassification(String               sourceName,
                                               String               userName,
                                               String               typeName,
                                               String               entityTypeName,
                                               ClassificationOrigin classificationOrigin,
                                               String               classificationOriginGUID,
                                               InstanceProperties   properties) throws TypeErrorException

Yes looks like thats the one judging by the stack trace I see in the logs:
 java.lang.NoSuchMethodError: org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector.OMRSRepositoryHelper.getNewClassification(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Lorg/odpi/openmetadata/repositoryservices/connectors/stores/metadatacollectionstore/properties/instances/ClassificationOrigin;Ljava/lang/String;Lorg/odpi/openmetadata/repositoryservices/connectors/stores/metadatacollectionstore/properties/instances/InstanceProperties;)Lorg/odpi/openmetadata/repositoryservices/connectors/stores/metadatacollectionstore/properties/instances/Classification;

@mandy-chessell Can you please revert the method in the OMRSRepositoryHelper interface itself? .. IGC repository helper is still breaking because of this.
Other than this after reverting the old schema update methods no additional related issues found. From Data Engine OMAS functional perspective we are good to go with this change and update deprecated methods afterwards #3843

Additional method getSkeletonEntitySummary needs to be reverted in OMRSRepositoryHelper to support IGC repository connector, this is the signature
EntitySummary getSkeletonEntitySummary(String                 sourceName,
                                           String                 metadataCollectionId,
                                           InstanceProvenanceType provenanceType,
                                           String                 userName,
                                           String                 typeName) throws TypeErrorException;


Using not depended to codemirror.js

Without loading codemirror, color picker cannot be initiated
//include colorpicker js file and css file then=>

var ColorPicker = CodeMirrorColorPicker.ColorPicker;
var colorpicker = new ColorPicker({
    color: 'blue', // init color code
    type : 'palette' // or 'sketch',  default type is 'chromedevtool'
})

this throws an error "Uncaught ReferenceError: CodeMirrorColorPicker is not defined"
To make it accessible I added one line to the end of ColorPicker method in the js file.
var index = _extends({}, Util, ColorPicker);
// added this line -->
window.CMColorPicker=ColorPicker.ColorPicker;
//--
return index;

and then tried

 var colorpicker = new CMColorPicker({
        color: 'blue',
        position: 'inline',
        container : $('color-container'),
        type : 'ring'
    });

This makes it work, But some types displayed distorted.
please see attached file. This is how "ring" type is displayed.

By the way I think this is one of the best color picker tool on the web. Great job!

hi @zortext
It seems to be a bug. I'll check and contact you.

i fixed this problem.
please  install lastest release  v1.9.35

Thanks... Now picker can be loaded by the following code without loading codemirror.
var colorpicker = new CodeMirrorColorPicker.ColorPicker({
        color: 'blue', // init color code
        position: 'inline',   // default show
        container :$element,
        type : 'ring'
    });

But still appearance  have some issues.

Default, sketch and palette type picker does not have any issues. Only other types have (xd,  ring, macos ...)
You can run this plunk to regenerate the issue
http://plnkr.co/edit/i66DSpcUk21WDWFOj0e0

hi @zortext
please see to https://codepen.io/easylogic/pen/ZqBBWd
The codepen.io works normally.
Something is wrong with the plunker, but I do not know if it is based on angularjs.
I have never played angularjs.

@zortext
haha  my mistake.
see to below code
        var picker = CodeMirrorColorPicker.create({
            type: 'ring',
            ......
        });

you must use  function named by create to makeing  for multi type colorpicker.
CodeMirrorColorPicker.ColorPicker  is default colorpicker  that has a 'chromedevtool' type


Public endpoints with  values

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):
Create deployment and ingress associated with the deployment
Result:
Ingress starts to "flap", getting constantly updated between node IP and  value address as public endpoint. Consequently the deployment gets updated as well with annotations to reflect the changes in the ingress.
Other details that may be helpful:
I have now 4 clusters and 3 of them don't have this issue, but one of them started this behavior after I tried emulating some failover scenarios (especifically deleting one node without draining or cordoning it first).
My ingresses on this cluster have been "flapping" between these two states:
2020/02/06 01:30:51 [INFO] Updating ingress [collabora-app:collabora-load-balancer] with public endpoints [[{"addresses":["<IP_ADDRESS>"],"port":443,"protocol":"HTTPS","serviceName":"collabora-app:collabora","ingressName":"collabora-app:collabora-load-balancer","hostname":"code-homologa.ipea.gov.br","allNodes":true}]]
2020/02/06 01:30:51 [INFO] Updating workload [collabora-app/collabora] with public endpoints [[{"addresses":["<IP_ADDRESS>"],"port":443,"protocol":"HTTPS","serviceName":"collabora-app:collabora","ingressName":"collabora-app:collabora-load-balancer","hostname":"code-homologa.ipea.gov.br","allNodes":true}]]
2020/02/06 01:31:14 [INFO] Updating ingress [collabora-app:collabora-load-balancer] with public endpoints [[{"addresses":["\u003cnil\u003e"],"port":443,"protocol":"HTTPS","serviceName":"collabora-app:collabora","ingressName":"collabora-app:collabora-load-balancer","hostname":"code-homologa.ipea.gov.br","allNodes":true}]]
2020/02/06 01:31:14 [INFO] Updating workload [collabora-app/collabora] with public endpoints [[{"addresses":["\u003cnil\u003e"],"port":443,"protocol":"HTTPS","serviceName":"collabora-app:collabora","ingressName":"collabora-app:collabora-load-balancer","hostname":"code-homologa.ipea.gov.br","allNodes":true}]]

(annotations of Deployment as shown on UI)
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
field.cattle.io/publicEndpoints: '[{"addresses":["<IP_ADDRESS>"],"port":443,"protocol":"HTTPS","serviceName":"collabora-app:collabora","ingressName":"collabora-app:collabora-load-balancer","hostname":"code-homologa.ipea.gov.br","allNodes":true}]'
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
field.cattle.io/publicEndpoints: '[{"addresses":["\u003cnil\u003e"],"port":443,"protocol":"HTTPS","serviceName":"collabora-app:collabora","ingressName":"collabora-app:collabora-load-balancer","hostname":"code-homologa.ipea.gov.br","allNodes":true}]'
It happens every 30 secs or so. I've tried deleting the deployment/ingress and re-creating them, but the issue still persists. I've also tried redeploying the nginx-ingress-controller and no changes as well.
My nodes are all RancherOS, provisioned with RKE using the VMWare ISO.
It seems somehow related to #17876 and #19208, but it happens on v2.3.x
Environment information
Rancher v2.3.5
Single installation on Ubuntu 18.04 with Docker 19.3.5

Cluster information
Cluster type: Infrastructure provider (vSphere on prem)
Machine type: VM with 8 CPU's and 16GB RAM
8 nodes (3 with etcd/control plane and 5 workers, all with RancherOS v1.5.5 and Docker 19.3.5)
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Docker version: 19.3.5

I recently found out that I had a broken ingress controller (from a failed GitLab integration attempt) that was messing around with the cluster. After removing it, everything  went back online.


Add info to readme for conan dependency manager

Hi, raylib is great. I recently added it to the conan dependency manager and thought it might be nice to add a description in the readme after seeing vcpkg on there.

@resttime Nice! Thank you very much for this addition!

@resttime Nice! Thank you very much for this addition!


⚠ GitLab Server has degraded performance

In 2d43494, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 7058 ms

Resolved: GitLab Server performance has improved in f11fda6.


What is the UML Diagram software?

What is the software for the UML Diagram image ?

Guess what, PHPStorm ;)


⚠ AZ-Launcher has degraded performance

In ba01f61, AZ-Launcher (https://az-launcher.nz/) experienced degraded performance:

HTTP code: 200
Response time: 1522 ms

Resolved: AZ-Launcher performance has improved in 1881436 after 11 minutes.


Subjects being cut off in API

This record shows the correct subjects for this work:
https://search.wellcomelibrary.org/iii/encore/record/C__Rb1174240?lang=eng&marcData=Y
650  0 Plants, Edible|xJava (Indonesia)|0sh 85102891
In the API call for this:
https://api.wellcomecollection.org/catalogue/v2/works/ah28bwrz
It appears to be cutting off the end of Indonesia and the closing bracket.
I haven't checked if this is an issue with the Sierra XML.

This is the relevant blob of JSON from the Sierra API:
        {
          "fieldTag": "d",
          "ind1": " ",
          "ind2": "0",
          "marcTag": "650",
          "subfields": [
            {
              "content": "Plants, Edible",
              "tag": "a"
            },
            {
              "content": "Java (Indonesia)",
              "tag": "x"
            },
            {
              "content": "sh 85102891",
              "tag": "0"
            }
          ]
        },

Doesn’t look like it’s being cut off there…

There's two very similar records https://wellcomecollection.org/works/ah28bwrz and https://wellcomecollection.org/works/j357tpex which both point to http://search.wellcomelibrary.org/iii/encore/record/C__Rb1174240.
Subjects are only cut off in https://wellcomecollection.org/works/ah28bwrz

Ah – the source identifier for ah28bwrz is a Miro identifier. Here's the relevant snippet of Miro JSON:
    "image_keywords_unauth": [
      "SCIENTIFIC ILLUSTRATIONS",
      "CHROMOLITHOGRAPHS",
      "BOTANY - JAVA (INDONESIA)",
      "ISLAND FLORA",
      "PLANTS, EDIBLE - JAVA (INDONES",
      "PLANTS, USEFUL - JAVA (INDONES",
      "TROPICAL FRUIT"
    ],

This will be fixed when we merge Miro/Sierra records in a way that's compatible with V0042695, but there's nothing more we can do for now. We already know Miro subjects are a bit wonky, so I'm going to close this specific ticket.


dfu: allow MCUBoot configured without scratch

This adds configuration to disable use of the scratch partition. This option must be disabled ('n') when MCUBoot itself was configured to not use a scratch partition for upgrades (swap with move).

/cc @nvlsianpu @d3zd3z

This allows software that uses mcumgr, eg, smp_svr to work with MCUBoot that is not using a scratch partition. For testing I have also these local changes:
diff --git a/boards/arm/frdm_k64f/frdm_k64f.dts b/boards/arm/frdm_k64f/frdm_k64f.dts
index 432482db6c..6274885d1a 100644
--- a/boards/arm/frdm_k64f/frdm_k64f.dts
+++ b/boards/arm/frdm_k64f/frdm_k64f.dts
@@ -185,15 +185,11 @@ arduino_spi: &spi0 {

                slot0_partition: partition@20000 {
                        label = "image-0";
-                       reg = <0x00020000 0x00060000>;
+                       reg = <0x00020000 0x00070000>;
                };
-               slot1_partition: partition@80000 {
+               slot1_partition: partition@90000 {
                        label = "image-1";
-                       reg = <0x00080000 0x00060000>;
-               };
-               scratch_partition: partition@e0000 {
-                       label = "image-scratch";
-                       reg = <0x000e0000 0x00020000>;
+                       reg = <0x00090000 0x00070000>;
                };
        };
 };
diff --git a/boards/arm/nrf52840_pca10056/nrf52840_pca10056.dts b/boards/arm/nrf52840_pca10056/nrf52840_pca10056.dts
index 3f4528081e..fc20b12a5b 100644
--- a/boards/arm/nrf52840_pca10056/nrf52840_pca10056.dts
+++ b/boards/arm/nrf52840_pca10056/nrf52840_pca10056.dts
@@ -230,15 +230,11 @@ arduino_spi: &spi3 {
                };
        };
 };
diff --git a/boards/arm/nrf52840_pca10056/nrf52840_pca10056.dts b/boards/arm/nrf52840_pca10056/nrf52840_pca10056.dts
index 3f4528081e..fc20b12a5b 100644
--- a/boards/arm/nrf52840_pca10056/nrf52840_pca10056.dts
+++ b/boards/arm/nrf52840_pca10056/nrf52840_pca10056.dts
@@ -230,15 +230,11 @@ arduino_spi: &spi3 {
                };
                slot0_partition: partition@c000 {
                        label = "image-0";
-                       reg = <0x0000C000 0x000067000>;
+                       reg = <0x0000C000 0x000076000>;
                };
-               slot1_partition: partition@73000 {
+               slot1_partition: partition@82000 {
                        label = "image-1";
-                       reg = <0x00073000 0x000067000>;
-               };
-               scratch_partition: partition@da000 {
-                       label = "image-scratch";
-                       reg = <0x000da000 0x0001e000>;
+                       reg = <0x00082000 0x000076000>;
                };

                /*
diff --git a/samples/subsys/mgmt/mcumgr/smp_svr/prj.conf b/samples/subsys/mgmt/mcumgr/smp_svr/prj.conf
index 5ac2554596..6808b9b13b 100644
--- a/samples/subsys/mgmt/mcumgr/smp_svr/prj.conf
+++ b/samples/subsys/mgmt/mcumgr/smp_svr/prj.conf
@@ -6,6 +6,7 @@ CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048

 # Ensure an MCUboot-compatible binary is generated.
 CONFIG_BOOTLOADER_MCUBOOT=y
+CONFIG_MCUBOOT_USE_SCRATCH=n

 # Allow for large Bluetooth data packets.
 CONFIG_BT_L2CAP_TX_MTU=260

@mbolivar This patch is needed as if mcuboot implements swap without scratch algorithm then target might(and should) not contain scratch partition at all.

@mbolivar This patch is needed as if mcuboot implements swap without scratch algorithm then target might(and should) not contain scratch partition at all.

The target could still have a scratch declared in the DTS file, but using the swap without scratch algorithm wouldn't care about it, so it just means that flash space is lost without use. For this PR I want scratch to not be looked at, because even if it is configured in the DTS, it won't have the metadata that MCUBoot needs. Whatever best method to implement it. I could assume that if the scratch is not configured then swap without scratch must be used, but if it is configured I have no way to guess if it is being used or not. Although in this case it could be considered a user misconfiguration...

My proposal:

enable swap without scratch algorithm in MCUBoot project if target doesn't have a scratch partition. If target have scratch partition, keep this still possible to configure via Kconfig (I believe this would add some at last software evaluation value).
do this symmetric here so if target doesn't contain scratch - no Kconfig entry needed at all, otherwise make this configurable.

My proposal:

@nvlsianpu your proposal is entirely based on whether or not the DT_ macro for the scratch partition is defined, right? No changes to Zephyr kconfig needed. If so, I'm agreed.

@nvlsianpu @mbolivar I removed the Kconfig related change.

@nvlsianpu your proposal is entirely based on whether or not the DT_ macro for the scratch partition is defined, right? No changes to Zephyr kconfig needed. If so, I'm agreed.

Yes. I suggested to have kconfig otherwise - but it is still fine to control this via DT only.


Missing search options for regular ePI's

The openapi spec does not reflect the standard search parameters (from https://www.hl7.org/fhir/bundle.html#search) which should be supported by the server (https://github.com/Gravitate-Health/focusing-manager/blob/191bddc324c3386630eaadaa76ea754b956bdf49/openapi.yaml#L10). My need is to be able to search for the unmodified ePI's ahead of annotations and other editorial steps by Identifier.

And how do you intend to model what ePI documents are preprocessed and so on? (FYI - @joofio )

The current existing ePI FHIR service that we have been using is documented here: https://github.com/EuropeanMedicinesAgency/EU-ePI-common-standard

I initially thought of using category to distinguish them. But as we go by,
we might change it of course.
Jens Kristian Villadsen @.***> escreveu no dia terça,
20/06/2023 à(s) 15:20:

The current existing ePI FHIR service that we have been using is
documented here:
https://github.com/EuropeanMedicinesAgency/EU-ePI-common-standard
—
Reply to this email directly, view it on GitHub
https://github.com/Gravitate-Health/focusing-manager/issues/6#issuecomment-1598889906,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADMJVUHF4UDA37VRN7GR5ELXMGWTDANCNFSM6AAAAAAZLDYSJQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

And what about language?

I might make sense to loop in @riksmithies (if he has the time) to say a word or two about how EMA has modelled languages for ePI's (IIRC - it is a FHIR List) and the FOSPS should probably align as much as possbile with the EMA poc.

language is the element i am using for that.

EMA also use Composition.language to say what language a document is in, and we made a change request in R5 to make it searchable (_language).
The Lists thing is a different but related concept. It is what EMA use to say which documents are translations of which. Lists are also used for which documents are about the same (set of) products.

In relation to EMA - @riksmithies would you recall if the MAN is the identifier stated as the bundle.identifier or would it be the composition.identifier?

I don't think the Marketing Authorisation Number would be either of those. That is product specific, and not specific to any document. There can be more than one document about a product and more than one product per document.

So what can be used as unique identifier for documents?

Can we define a set of elements that identify a unique document?
Jens Kristian Villadsen @.***> escreveu em sex.,
23/06/2023 às 14:56 :

So what can be used as unique identifier for documents?
—
Reply to this email directly, view it on GitHub
https://github.com/Gravitate-Health/focusing-manager/issues/6#issuecomment-1604322070,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADMJVUHSLV56BI3ICEOD55LXMWOA3ANCNFSM6AAAAAAZLDYSJQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
João Almeida
MPharm, Msc, PhD Student
LinkedIn https://www.linkedin.com/in/joaofilipealmeida/ | ORCID
https://orcid.org/0000-0003-0882-6547 | Website http://joaofcalmeida.pt
| gitHub https://github.com/joofio |
e-Mais http://e-mais.pt| HL7 Portugal http://hl7.pt | @.***
http://cintesis.med.up.pt/ | IHE Pharmacy http://ihe.net

We should. What does the barcode on medication packages reflect?

Documents are not necessarily about single products, and a product can be in multiple documents. So there is no product id, or product package id that can work as a document id. We should ask what business ids these documents currently have. I did ask that and didn't get a clear answer. It seems they may just not have ids. Maybe they don't need them. What would we use them for if we had them?

I have a product id - I would like to have the corresponding document/leaflet.

Similarly you may want to find a document about a patient. You don't make the document id the patient id, you use the subject.

I would assume any leaflet to carry a business identifier since they go through a quality refinement process, right?

If they do have an existing one then that is fine.
But it seems we are trying to create one even if existing processes don't need one. Things don't always need an identifiying number/string just to be used or referenced. Maybe people just work on the PIL for product xyz.

Lets turn it around and focus on those that actually DO have one (as we cannot force anyone to actually have one).Where would that be placed? As bundle identifier or composition identifier?

Standard FHIR says Bundle.identifier.

So - what is most likely needed (my needs) is the support for search on Bundle.identifier as well as the standard support for _language parameter

Should language be optional? Or both are mandatory?

Language as attribute in the Document model would probably make sense to make mandatory

@aalonsolopez in regards to input parameters in your API - IDK.

@aalonsolopez lets start by making it required

Perfect, noted on my task definition 👌

In addition, I don't know if this feature will be done within the next week because maybe the whole technical team will be focused on doing lenses and finishing the preprocessing service. I will update when I know.

@aalonsolopez all languages have now been added to sample data on https://github.com/hl7-eu/gravitate-health/commit/78e20ca58ef3a662c2d0a16f6ccaeddbeaec6a1c and the profile has been tuned: https://github.com/HL7/emedicinal-product-info/commit/414fa35cd71a9ffe226f708608f8d5bff6464eab

Similarly you may want to find a document about a patient. You don't make the document id the patient id, you use the subject.

This is the way i have been using. However, the idea of product in this thread was kind of confusing for me.
If we speak about product as medicinalProductDefinition, then I would assume we can only have a leaflet for it (taken into account several updates and languages) and this way seems ok. We would need the language for additional filtering only.
And the idea of a leaflet being about multiple products and a product being in multiple leaflets (besides language and version) is weird to me.
If we were talking about products as ingredients, then it all makes more sense, but is less interesting for the case at hand, because of exactly the issue mentioned. They can be in several leaflets and leaflets can be about several products (i.e. ingredients)

I would assume any leaflet to carry a business identifier since they go through a quality refinement process, right?

Regarding this, i agree with Rik. The possibility of forcing an addition of yet another identifier might be more confusing.
cant we properly select documents based on (subject - MedicinalProductIdentifier, language and possibly version?)
check above ccomment

In addition, I don't know if this feature will be done within the next week because maybe the whole technical team will be focused on doing lenses and finishing the preprocessing service. I will update when I know.

I can confirm that our priority for this week is to have more lenses finished, so this issue is delayed.

hi @joofio

And the idea of a leaflet being about multiple products

This is very common. Often different but similar products are documented together. e.g.
Kalydeco 75 mg film-coated tablets
Kalydeco 150 mg film-coated tablets
https://www.medicines.org.uk/emc/files/pil.11974.pdf
(one could argue that this is one product, but normally different strengths are enough for these to be considered different products - certainly EMA and IDMP itself work that way - even though they are clearly the same "product family").

and a product being in multiple leaflets (besides language and version) is weird to me.
Also very common, in fact in almost every case. SmPC and PIL are different documents about the same product(s) for instance.
see https://www.medicines.org.uk/emc/product/11974

If we were talking about products as ingredient
No it is nothing to do with that. Ingredients are not really ever considered to be products (outside of manufacturing scenarios)

re language, clearly it is very important in some places. But also some, in fact perhaps most, countries will never use it. Most projects will not be multi-national either I expect. So it may not be a good idea to make supporting it mandatory.

From a link like https://www.ema.europa.eu/en/documents/all-authorised-presentations/skilarence-epar-all-authorised-presentations_en.pdf - it seems like there are multiple identifiers for the same product - and thereby one PIL. So in essence; one document with multiple identifiers.

That PDF list shows multiple products - because it shows multiple strengths (which usually means more than one product). It also shows multiple packs.
These multiple products may or may not be all on one PIL.
I can see from here that it is at least 2 PILs (https://www.medicines.org.uk/emc/product/752/pil, https://www.medicines.org.uk/emc/product/7480/pil) in this case.
There are also definitely PILs with more than one product (example was given above, Kalydeco).
But regardless, other documents also exist in a wider ePI context. There only be one PIL per product. But that isn't the same as saying only one ePI document per product.

Thanks for the clarification Rik.
I would argue then that, with the subject of the composition (which can have more than one subject, and that is ok)
Then, if we reduce to PIL, we might have just one doc per "identifier". But if we add the SmPC, we can always filter by type/category.
what do you think?

Think about search params, or about List?
If we ignore other languages and versions and statuses then there may be only one PIL and one SmPC per product (each product having one identifier).
We can always reduce the set to one, for a certain use case, but in general there are multiple documents.
Filtering to find e.g. "the currently approved set of documents about X" is not always straightforward.
Which is why EMA wanted a way to assert document sets, rather than compute them.
Whether management of document sets should be in scope of an IG for ePI is a different matter. But we don't want something that clashes with it.

@riksmithies List's is still a thing/the thing in the EMA spec for grouping the ePI's in different languages, correct?

Yes no change on that

So - what is most likely needed (my needs) is the support for search on Bundle.identifier as well as the standard support for _language parameter

Will be added as part of default configurable behaviour as of 7.0.0 of HAPI: https://github.com/hapifhir/hapi-fhir/commit/2da8aafad0dca392728e0710c1e92f6cb57d92ca

We need to update hapi version of the fosps. @jkiddo any objection or issue that you see with that?

Nope

The idea here, in terms of the project, would be to search by text in the all epis right? or just the name? thoughts on this?

that would be https://www.hl7.org/fhir/search.html#_content

so you agree on searching on all fields?

this may require some configuration change on the backend, but the rest is frontend. since they are already supported by hapi


Check and lint workflow

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@2b0c92f). Click here to learn what that means.
The diff coverage is n/a.

@@          Coverage Diff           @@
##             main      #1   +/-   ##
======================================
  Coverage        ?   1.51%
======================================
  Files           ?       3
  Lines           ?      66
  Branches        ?       0
======================================
  Hits            ?       1
  Misses          ?      65
  Partials        ?       0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b0c92f...7a74fe7. Read the comment docs.


Always getting "Android version not supported. Aborting." in MAC

Hello quarkslab, first thank you for create this tool.
I couldnt try it but .... your application worth the time .
I am using Android 11.0 (Google Play)
python3 aeroot.py pid 10614
AERoot (Android Emulator ROOTing system) v. 0.2
_           _         _   _
||      ( \       ( ) ( )
(  /       \ \     | || |
| (|_| \ \   (   _   )
(  )  ()) )   | () |
/_||) |        / /   (   _   )
_)       / /      | | | |
||         (/       () (_)
[!] Android version not supported. Aborting.
[+] Exiting.
Well, I wasn't with my arms crossed. XD
First by touching the  phyton code I have to read the outcome of uname:
uname -rm return
"5.4.47-01061-g22e35a1de440 x86_64"
Second: Install gdb (because you didnt tell me I will need it).
brew install gdb
Third Add the dictionary entry:
"5.4.47-01061-g22e35a1de440 x86_64": {
"name": "Android 11.0 (R) x86_64",
"gdb_arch": "x86_64",
"offset_to_comm": 1808,
"offset_to_parent": 1368,
"offset_to_tasks": 1096,
"offset_to_creds": 1792,
"offset_to_init_ptr": -1379768,
"offset_to_pid": 1352,
"kernel_ptr":<PHONE_NUMBER>4686296388,
"selinux_offset": 1703193,
"ptr_size": 8,
"enforce_size": 1,
"ps_name_cmd": "ps -A",
"ps_pid_cmd": "ps -A -o pid="
},
and ...
the ps command is not working in my OS !!
I try :
"ps_name_cmd": "ps aux | grep -v grep | grep -c [-i] ",
"ps_pid_cmd": "ps -A -o -p "
But didnt work
Do you know how can I do to run it in MacOS ?

Hi! Thank you for the feedback.
I don't think the problem is with your host OS (even if I never tested AERoot on MAC).

Second: Install gdb (because you didnt tell me I will need it).
brew install gdb

I will add the requirement in the README file.

uname -rm return
"5.4.47-01061-g22e35a1de440 x86_64"

and ...
the ps command is not working in my OS !!

Are you running these commands on your host ? They are intented to be runned on Android through ADB.
So. Could you try: adb shell uname -rm (with your Android 11 running), then paste the output here ?
Regards.

Wow, thank you Eric Le Guevel Ha0ris , for your very fast answer !
Here you are the output:
$ adb shell uname -rm
5.4.47-01061-g22e35a1de440 x86_64
Merci beaucoup
Thanks in advance.
Bonne journée.
Have a nice day!

Hello Eric,
Thank you for teaching me and your patience.
I got the PID of __ia32_co+
and after run it
[+] Detected: Android 11.0 (R) x86
[+] Search for [7926] process in memory (this may take a while) ...
[!] Gdb timed out. Make sure gdbserver is running on guest (-qemu -s).
[+] Exiting.
Do you know how to check it on guest (inside the emulator) ?

Hello Eric,
Thank you for teaching me and your patience.
I got the PID of __ia32_co+
and after run it
[+] Detected: Android 11.0 (R) x86
[+] Search for [7926] process in memory (this may take a while) ...
[!] Gdb timed out. Make sure gdbserver is running on guest (-qemu -s).
[+] Exiting.
Do you know how to check it on guest (inside the emulator) ?

Hi. When you start your emulator, you must specify -qemu -s in the command line like that:
emulator -avd API_30 -qemu -s

Without this option, this is no gdbserver running on qemu, so AERoot can't work properly.

Hello Eric,
After run
$ ./emulator -netdelay none -netspeed full -avd API_30_1 -qemu -s
emulator: WARNING: Ignoring invalid http proxy: Bad format: invalid proxy name
emulator: INFO: boot completed
emulator: INFO: boot time 32284 ms
emulator: Increasing screen off timeout, logcat buffer size to 2M.
emulator: Revoking microphone permissions for Google App.
[+] Detected: Android 11.0 (R) x86
[+] Search for [4977] process in memory (this may take a while) ...
[!] [4977] process not found in memory. Aborting.
[+] Exiting.
Do you know what that means?
Thanks in advance.

Hi.
Could you try to run: python3 aeroot -v daemon, then paste the output here ?
Thank you in advance.

Thank you for all your support.
$ python3 aeroot.py -v daemon
AERoot (Android Emulator ROOTing system) v. 0.2
[-] Loading xxxx/AERoot/config.json ...
[+] Detected: Android 11.0 (R) x86
[-] [adbd] process is running
[+] Search for [adbd] process in memory (this may take a while) ...
[-] Init task_struct found at: -0x448
[-] Looking at 0xf000ff53f000de7b - []
[!] [adbd] process not found in memory. Aborting.
[+] Exiting.
Thank you in advance.

I am not 100% sure, but it may be due to the KASLR bypass.
You should try the experimental branch.
And also, since Android 11, this may take some time for the kernel to be correctly mapped into virtual memory, so you should also wait few seconds after your Android starts running to run aeroot.

The issue should be fixed as the experimental has been merged in master.
There still a problem (sometimes) with the emulator cold boot on macOS. A workaround has been added at the end of the README.


examples: do not pack any library with hello world

Hello world example doesn't depend on any library so do not pack it with other avm files.
These changes are made under both the "Apache 2.0" and the "GNU Lesser General
Public License 2.1 or later" license terms (dual license).
SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

Is this just a cleanup?  I was thinking the hello_world example should use io:format instead of the console API.

Is this just a cleanup? I was thinking the hello_world example should use io:format instead of the console API.

I did a double check right now and I think this should be changed to console:print:
I agree that most of the times io:format is the right thing to do, however by using console:print we make sure this is executable without any additional dependency, while io:format requires additional machinery.
So you can either flash it to a real device, run it with ./AtomVM hello_world.avm or even ./AtomVM hello_world.beam
As an alternative we might replace it with erlang:display(hello_world), that might be even better.

So you can either flash it to a real device, run it with ./AtomVM hello_world.avm or even ./AtomVM hello_world.beam

I get that, for the AtomVM developer.  It's not really applicable to the AtomVM user, however, because the end-game is to install AtomVM (make install), which installs the atomvm (case-sensitive) script, which includes atomvmlib.avm on the command line.
But we are still a ways from that.  Once we do a release (now that we have an install target), we will actually be able to build (src and binary) packages for various package managers (apt, yum, brew, etc), so that users can install the binaries and scripts without having the build from source.  But that is well into the future, so probably not relevant to this discussion.

ok for dev purposes only

Changed accordingly for dev purposes only.


fix code block typo in streamfield_migrations.md

Fixes code block typo in streamfield_migrations.md
Please check the following:

[ ] Do the tests still pass?[^1]
[ ] Does the code comply with the style guide?

[ ] Run make lint from the Wagtail root.

[ ] For Python changes: Have you added tests to cover the new/fixed behaviour?
[ ] For front-end changes: Did you test on all of Wagtail’s supported environments?[^2]

[ ] Please list the exact browser and operating system versions you tested:
[ ] Please list which assistive technologies [^3] you tested:

[ ] For new features: Has the documentation been updated accordingly?

Please describe additional details for testing this change.
[^1]: Development Testing
[^2]: Browser and device support
[^3]: Accessibility Target

Thanks @danhayden

Cherry picked to 5.1 3c16eaae7eac7b16b1e975d1bd48eafe2bbb6d87


Add back r2ds.bat

For reference, we had this file in a previous release and it proved useful for novices trying to host dedicated servers.


bilibilitv: download breaks after 8388608

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting that yt-dlp is broken on a supported site
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
EU
Provide a description that is worded well enough to be understood
since 2 days bilibili.ty stops all downloads at exactly 8MB, e.g.:
[download]  55.6% of   14.39MiB at    1.11MiB/s ETA 00:05[download] Got error: Downloaded 8388608 bytes, expected 15092847 bytes. Retrying (1/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (2/10)...
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['https://www.bilibili.tv/en/video/4786903491679744', '-vU']
[debug] User config "/home/nik/.config/yt-dlp/config": ['--no-cache-dir', '--restrict-filenames', '--trim-filenames', '128', '--embed-metadata', '--concurrent-fragments', '1', '--format-sort', 'ext', '--exec', 'timestamp="%(release_date>%s,upload_date>%s|NA)s"; echo "${timestamp}" | grep -Eq "^[^\\-][0-9]+$" && touch -m --date="@${timestamp}" %(filepath)q']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds [6b5d93b0b] (zip)
[debug] Python 3.11.7 (CPython x86_64 64bit) - Linux-6.5.0-5-amd64-x86_64-with-glibc2.37 (OpenSSL 3.0.11 19 Sep 2023, glibc 2.37)
[debug] exe versions: ffmpeg 6.1-5 (setts), ffprobe 6.1-5
[debug] Optional libraries: brotli-1.1.0, certifi-2023.07.22, mutagen-1.46.0, requests-2.31.0, sqlite3-3.44.2, urllib3-1.26.18
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests
[debug] Loaded 1792 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds)
[BiliIntl] Extracting URL: https://www.bilibili.tv/en/video/4786903491679744
[BiliIntl]<PHONE_NUMBER>679744: Downloading webpage
WARNING: [BiliIntl] unable to extract JSON-LD ; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [BiliIntl] unable to extract og:title; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [BiliIntl] unable to extract og:description; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[BiliIntl]<PHONE_NUMBER>679744: Downloading video formats
[debug] Sort order given by user: ext
[debug] Formats sorted by: hasvid, ie_pref, vext, aext, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info]<PHONE_NUMBER>679744: Downloading 1 format(s): 9+1
[debug] Invoking http downloader on "https://upos-bstar1-mirrorakam.akamaized.net/iupxcodeboss/1t/zp/n230425ad38mis41pnvmr621x8tyzp1t-1-131220110000.m4s?e=ig8euxZM2rNcNbdlhoNvNC8BqJIzNbfqXBvEqxTEto8BTrNvN0GvT90W5JZMkX_YN0MvXg8gNEV4NC8xNEV4N03eN0B5tZlqNxTEto8BTrNvNeZVuJ10Kj_g2UB02J0mN0B5tZlqNCNEto8BTrNvNC7MTX502C8f2jmMQJ6mqF2fka1mqx6gqj0eN0B599M=&uipk=5&nbs=1&deadline=1702566832&gen=playurlv2&os=akam&oi=1317955470&trid=388026478eb0499a8e58e5a62c2603c0i&mid=0&platform=pc&upsig=2ffcd8bbd4f4eaa2ea46a8e699104cff&uparams=e,uipk,nbs,deadline,gen,os,oi,trid,mid,platform&hdnts=exp=1702566832~hmac=ef6e7ec4912838145e9a12a61cefc6ddf1147685e8b99b75432d41e686e6334b&bvc=vod&nettype=0&orderid=0,1&logo=00000000&f=i_0_0"
[download] In_Another_Word_with_My_Smartphone_Season_2_Ep4-[4786903491679744].f9.mp4 has already been downloaded
[download] 100% of   20.91MiB
[debug] Invoking http downloader on "https://upos-bstar1-mirrorakam.akamaized.net/iupxcodeboss/1t/zp/n230425ad38mis41pnvmr621x8tyzp1t-1-1c1301000022.m4s?e=ig8euxZM2rNcNbdlhoNvNC8BqJIzNbfqXBvEqxTEto8BTrNvN0GvT90W5JZMkX_YN0MvXg8gNEV4NC8xNEV4N03eN0B5tZlqNxTEto8BTrNvNeZVuJ10Kj_g2UB02J0mN0B5tZlqNCNEto8BTrNvNC7MTX502C8f2jmMQJ6mqF2fka1mqx6gqj0eN0B599M=&uipk=5&nbs=1&deadline=1702566832&gen=playurlv2&os=akam&oi=1317955470&trid=afae0bcbc5f043ab9ef6bbd5ba33640ai&mid=0&platform=pc&upsig=ee07bcd85b5c4a9afbe7a5ddb6d589a5&uparams=e,uipk,nbs,deadline,gen,os,oi,trid,mid,platform&hdnts=exp=1702566832~hmac=fce1b0bec9120579bb05fbaa3d66d612d52e5d775b9ed2f28a8e4275f841136c&bvc=vod&nettype=0&orderid=0,1&logo=00000000&f=i_0_0"
[download] Resuming download at byte 8388608
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (1/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (2/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (3/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (4/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (5/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (6/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (7/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (8/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (9/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (10/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Giving up after 10 retries
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/nik/bin/yt-dlp/__main__.py", line 17, in <module>
    yt_dlp.main()
  File "/home/nik/bin/yt-dlp/yt_dlp/__init__.py", line 1009, in main
    _exit(*variadic(_real_main(argv)))
  File "/home/nik/bin/yt-dlp/yt_dlp/__init__.py", line 999, in _real_main
    return ydl.download(all_urls)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3528, in download
    self.__download_wrapper(self.extract_info)(
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3503, in wrapper
    res = func(*args, **kwargs)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1570, in extract_info
    return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1581, in wrapper
    return func(self, *args, **kwargs)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1737, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1796, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2939, in process_video_result
    self.process_info(new_info)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3387, in process_info
    partial_success, real_download = self.dl(fname, new_info)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3126, in dl
    return fd.download(name, new_info, subtitle)
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/common.py", line 455, in download
    ret = self.real_download(filename, info_dict)
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/http.py", line 362, in real_download
    for retry in RetryManager(self.params.get('retries'), self.report_retry):
  File "/home/nik/bin/yt-dlp/yt_dlp/utils/_utils.py", line 5042, in __iter__
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/common.py", line 401, in report_retry
    RetryManager.report_retry(
  File "/home/nik/bin/yt-dlp/yt_dlp/utils/_utils.py", line 5049, in report_retry
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/common.py", line 404, in <lambda>
    error=IDENTITY if not fatal else lambda e: self.report_error(f'\r[download] Got error: {e}'),
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1048, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 976, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

I tested the sample URL provided and found no issue with the download.

I just tried again. Ut still stops, but now at 16MB:
$ yt-dlp https://www.bilibili.tv/en/video/4786903491679744 -vU
[debug] Command-line config: ['https://www.bilibili.tv/en/video/4786903491679744', '-vU']
[debug] User config "/home/nik/.config/yt-dlp/config": ['--no-cache-dir', '--restrict-filenames', '--trim-filenames', '128', '--embed-metadata', '--concurrent-fragments', '1', '--format-sort', 'ext', '--exec', 'timestamp="%(release_date>%s,upload_date>%s|NA)s"; echo "${timestamp}" | grep -Eq "^[^\\-][0-9]+$" && touch -m --date="@${timestamp}" %(filepath)q']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds [db8b4edc7] (zip)
[debug] Python 3.11.7 (CPython x86_64 64bit) - Linux-6.5.0-5-amd64-x86_64-with-glibc2.37 (OpenSSL 3.0.11 19 Sep 2023, glibc 2.37)
[debug] exe versions: ffmpeg 6.1-5 (setts), ffprobe 6.1-5
[debug] Optional libraries: certifi-2023.07.22, requests-2.31.0, secretstorage-3.3.3, sqlite3-3.44.2, urllib3-1.26.18
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests
[debug] Loaded 1796 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds)
[BiliIntl] Extracting URL: https://www.bilibili.tv/en/video/4786903491679744
[BiliIntl]<PHONE_NUMBER>679744: Downloading webpage
WARNING: [BiliIntl] unable to extract JSON-LD ; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [BiliIntl] unable to extract og:title; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [BiliIntl] unable to extract og:description; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[BiliIntl]<PHONE_NUMBER>679744: Downloading video formats
[debug] Sort order given by user: ext
[debug] Formats sorted by: hasvid, ie_pref, vext, aext, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info]<PHONE_NUMBER>679744: Downloading 1 format(s): 9+1
[debug] Invoking http downloader on "https://upos-bstar1-mirrorakam.akamaized.net/iupxcodeboss/1t/zp/n230425ad38mis41pnvmr621x8tyzp1t-1-131220110000.m4s?e=ig8euxZM2rNcNbdlhoNvNC8BqJIzNbfqXBvEqxTEto8BTrNvN0GvT90W5JZMkX_YN0MvXg8gNEV4NC8xNEV4N03eN0B5tZlqNxTEto8BTrNvNeZVuJ10Kj_g2UB02J0mN0B5tZlqNCNEto8BTrNvNC7MTX502C8f2jmMQJ6mqF2fka1mqx6gqj0eN0B599M=&uipk=5&nbs=1&deadline=1703067800&gen=playurlv2&os=akam&oi=1317955470&trid=f2e88ae99acc4141b736982edcc10947i&mid=0&platform=pc&upsig=c5bed9ff2590a6398c8ffd9c98ca1df8&uparams=e,uipk,nbs,deadline,gen,os,oi,trid,mid,platform&hdnts=exp=1703067800~hmac=561243cb1fc35331ea444cebfdf0090b56c054bb45e34978d94cb2ae51a8e5e7&bvc=vod&nettype=0&orderid=0,1&logo=00000000&f=i_0_0"
[download] Destination: In_Another_Word_with_My_Smartphone_Season_2_Ep4-[4786903491679744].f9.mp4
[download]  76.5% of   20.91MiB at    1.85MiB/s ETA 00:02[download] Got error: Downloaded 16777216 bytes, expected 21924501 bytes. Retrying (1/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (2/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (3/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (4/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (5/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (6/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (7/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (8/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (9/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (10/10)...
[download] Got error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Giving up after 10 retries
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/nik/bin/yt-dlp/__main__.py", line 17, in <module>
    yt_dlp.main()
  File "/home/nik/bin/yt-dlp/yt_dlp/__init__.py", line 1009, in main
    _exit(*variadic(_real_main(argv)))
  File "/home/nik/bin/yt-dlp/yt_dlp/__init__.py", line 999, in _real_main
    return ydl.download(all_urls)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3528, in download
    self.__download_wrapper(self.extract_info)(
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3503, in wrapper
    res = func(*args, **kwargs)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1570, in extract_info
    return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1581, in wrapper
    return func(self, *args, **kwargs)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1737, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1796, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2939, in process_video_result
    self.process_info(new_info)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3387, in process_info
    partial_success, real_download = self.dl(fname, new_info)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3126, in dl
    return fd.download(name, new_info, subtitle)
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/common.py", line 455, in download
    ret = self.real_download(filename, info_dict)
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/http.py", line 362, in real_download
    for retry in RetryManager(self.params.get('retries'), self.report_retry):
  File "/home/nik/bin/yt-dlp/yt_dlp/utils/_utils.py", line 5042, in __iter__
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/common.py", line 401, in report_retry
    RetryManager.report_retry(
  File "/home/nik/bin/yt-dlp/yt_dlp/utils/_utils.py", line 5049, in report_retry
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/common.py", line 404, in <lambda>
    error=IDENTITY if not fatal else lambda e: self.report_error(f'\r[download] Got error: {e}'),
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1048, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 976, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

[debug] Invoking http downloader on "https://upos-bstar1-mirrorakam.akamaized.net/iupxcodeboss/1t/zp/n230425ad38mis41pnvmr621x8tyzp1t-1-1c1301000022.m4s?e=ig8euxZM2rNcNbdlhoNvNC8BqJIzNbfqXBvEqxTEto8BTrNvN0GvT90W5JZMkX_YN0MvXg8gNEV4NC8xNEV4N03eN0B5tZlqNxTEto8BTrNvNeZVuJ10Kj_g2UB02J0mN0B5tZlqNCNEto8BTrNvNC7MTX502C8f2jmMQJ6mqF2fka1mqx6gqj0eN0B599M=&uipk=5&nbs=1&deadline=1703067800&gen=playurlv2&os=akam&oi=1317955470&trid=09bb20294dc44200986fce340a7b6f3fi&mid=0&platform=pc&upsig=4e5eb95b5a83e932590ec494ee7b06fb&uparams=e,uipk,nbs,deadline,gen,os,oi,trid,mid,platform&hdnts=exp=1703067800~hmac=af8a92bde22dfa87cb59b85abc56be502fd5a89f769dab976d452ded23c8fbc9&bvc=vod&nettype=0&orderid=0,1&logo=00000000&f=i_0_0"
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 4069, in urlopen
    return self._request_director.send(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nik/bin/yt-dlp/yt_dlp/networking/common.py", line 114, in send
    response = handler.send(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nik/bin/yt-dlp/yt_dlp/networking/_helper.py", line 204, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nik/bin/yt-dlp/yt_dlp/networking/common.py", line 325, in send
    return self._send(request)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/nik/bin/yt-dlp/yt_dlp/networking/_requests.py", line 341, in _send
    raise HTTPError(res, redirect_loop=max_redirects_exceeded)
yt_dlp.networking.exceptions.HTTPError: HTTP Error 403: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3387, in process_info
    partial_success, real_download = self.dl(fname, new_info)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3126, in dl
    return fd.download(name, new_info, subtitle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/common.py", line 455, in download
    ret = self.real_download(filename, info_dict)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/http.py", line 364, in real_download
    establish_connection()
  File "/home/nik/bin/yt-dlp/yt_dlp/downloader/http.py", line 120, in establish_connection
    ctx.data = self.ydl.urlopen(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nik/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 4101, in urlopen
    raise _CompatHTTPError(e) from e
yt_dlp.networking.exceptions._CompatHTTPError: HTTP Error 403: Forbidden

I believe you're facing the same issue described in issue #6640
Please add --add-header 'Referer: https://www.bilibili.tv/' to the yt-dlp arguments. It should work.

That did the trick, thank you! Just for the record: I had to remove the space, so --add-header 'Referer:https://www.bilibili.tv/' did it.


Fix for issue 361

Sorry for the delay in creating this small fix. Here's the PR for #361.

Thank you @gfmio. This is a good generalization of the types 👍


Package xxxx is not callable on OS X El Capitan

Whenever I use jpype.JPackage() to import a class and create a new object, for example,
DelegateHandler = JPackage("jpype.xml").DelegateHandler
dh = DelegateHandler(None, None, JProxy("org.xml.sax.ContentHandler", inst=ContentHandler()), None)

I constantly get the error message TypeError: Package xxxx is not Callable.
I'm using Java 1.6 and Python 2.7. Anyone has the same problem on a Mac?

a package and class are not the same thing. Are you familiar with Java?


filisters, gravelly (logical-conflict)

This pull request was generated by the 'mq' tool

/trunk merge


Update rules with missing rules from updated pluralize js library

I made the comparison with the actual master branch on the JS pluralize library to update all the changes (not much at the end)

Hi @Jerome2606,
Thank you for taking the time to contribute - appreciate it.
Could you please provide the commit ID up to which you have synchronized the rules? Asking because I could find some extra rules in the latest master branch of the original library.
Thanks & Regards
Sarath

I'll just outline what I observed.

The singular rule - [/(dg|ss|ois|lk|ok|wn|mb|th|ch|ec|oal|is|ck|ix|sser|ts|wb)ies$/i, '$1ie'], is missing .
'species' is removed from uncountables in the original library.
Lots of tests added to test singular words ending in 'ie'.

Kindly update these if you were going for complete sync with the master branch of the original library.
Otherwise no worries, please just update the commit ID up to which your changes reflect in the Readme file, and as long as your changes cover all the changes & the tests up to that commit ID - I'll be happy to merge this PR.
Thanks & Regards
Sarath


Extend the unleash interface with context-data

In the current version of Unleash strategies needs to get their context data via special provides which typically is implemented as some magic-data available at the current request. This works well in most java-based web-applications.
However doing toggles in a standalone web-application this becomes much harder as you typically do not have a user bound to the current request. In this scenario you will typically have the context-data available runtime, at the same location you want to verify if a toggle should be on or of.
It would thus be beneficial if the client also could inject context data at runtime. The unleash API would then be overloaded with a new method, the the user can inject the required context-data.
unleash.isEnabled("app.someToggle", UnleashConext.of("k1", "v1", "k2", "v2"));

Unleash will then in turn inject the context-data to the strategy which would be extended to:
public interface Strategy {
    String getName();

    boolean isEnabled(Map<String, String> parameters);

    boolean isEnabled(Map<String, String> parameters, UnleashConext context);
}

(* if we move the client to java8 we could also provide a default implementation of the new method to avoid breaking the API)
This is also the approach used for the unleash-client-node.

I think newcomers to unleash, who needs to implement a new strategy, would always use UnleashContext if we introduce it, since it's more straight forward. Using UnleashContext is way better thatn using ThreadLocal IMHO.
I'm still a bit sceptic though. Could you give a concrete example when injecting a provider into the strategy would not solve the need for runtime data?

moved discussion to the java-client: https://github.com/finn-no/unleash-client-java/issues/29


Evalution of Llama on Hellaswag

Hi, there, how do you evaluate Llama on Hellaswag? Llama does not have an API that can pass in an argument called echo and returns the log probs. To the best of my knowledge, if we are using Llama from Huggingface, we can only get logits as model output. How to make it work on Hellaswag?

You should be able to use HF's implementation of Llama
Also note there is an issue with tokenization which is currently a work-in-progress.
https://github.com/EleutherAI/lm-evaluation-harness/pull/531

Thanks for your help. I just want to double-check if my understanding is correct. For each question from Hellaswag, there are 4 choices, so you use model(question + choice) for each choice and get the log prob for each choice. Finally, use argmax over the probability distribution of the 4 choices to select the predicted answer. Is this correct?

As en example
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('huggyllama/llama-7b')
model = AutoModelForCausalLM.from_pretrained('huggyllama/llama-7b', device_map='auto', load_in_8bit=True)

test = [
    ('Paris is the capital of', ' England.'),
    ('Paris is the capital of', ' Germany.'),
    ('Paris is the capital of', ' France.'),
    ('Paris is the capital of', ' Japan.'),
]

# encode full sentences, questions, and answers, no padding for simplicity
batched_sentences = tokenizer.batch_encode_plus([q + a for q, a in test], add_special_tokens=False, return_tensors='pt')['input_ids']
batched_questions = tokenizer.batch_encode_plus([q for q, _ in test], add_special_tokens=False, return_tensors='pt')['input_ids']

# run the model on full sentences and get the log probabilities
batched_logprobs = F.log_softmax(model(batched_sentences.cuda())['logits'], dim=-1).cpu()

# take log probabilities corresponding to possible answer tokens
batched_logprobs = batched_logprobs[:, len(batched_questions[0]) - 1 : -1, :]

# get the scores by summing log probabilities corresponding to correct answer tokens, unvectorized
scores = []
for sentence, question, logprobs in zip(batched_sentences, batched_questions, batched_logprobs):
    answer = sentence[len(question):]
    guess = logprobs.argmax(dim=-1)
    print(tokenizer.decode(guess), bool((guess == answer).all()))
    scores.append(float(torch.gather(logprobs, 1, answer.unsqueeze(-1)).sum()))

# predict the answer
test[torch.tensor(scores).argmax()]

Can answer correctly even if the guess in print is wrong.

As en example
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('huggyllama/llama-7b')
model = AutoModelForCausalLM.from_pretrained('huggyllama/llama-7b', device_map='auto', load_in_8bit=True)

test = [
    ('Paris is the capital of', ' England.'),
    ('Paris is the capital of', ' Germany.'),
    ('Paris is the capital of', ' France.'),
    ('Paris is the capital of', ' Japan.'),
]

# encode full sentences, questions, and answers, no padding for simplicity
batched_sentences = tokenizer.batch_encode_plus([q + a for q, a in test], add_special_tokens=False, return_tensors='pt')['input_ids']
batched_questions = tokenizer.batch_encode_plus([q for q, _ in test], add_special_tokens=False, return_tensors='pt')['input_ids']

# run the model on full sentences and get the log probabilities
batched_logprobs = F.log_softmax(model(batched_sentences.cuda())['logits'], dim=-1).cpu()

# take log probabilities corresponding to possible answer tokens
batched_logprobs = batched_logprobs[:, len(batched_questions[0]) - 1 : -1, :]

# get the scores by summing log probabilities corresponding to correct answer tokens, unvectorized
scores = []
for sentence, question, logprobs in zip(batched_sentences, batched_questions, batched_logprobs):
    answer = sentence[len(question):]
    guess = logprobs.argmax(dim=-1)
    print(tokenizer.decode(guess), bool((guess == answer).all()))
    scores.append(float(torch.gather(logprobs, 1, answer.unsqueeze(-1)).sum()))

# predict the answer
test[torch.tensor(scores).argmax()]

Can answer correctly even if the guess in print is wrong.

Hello, I was wondering how could we alter this code for few-shot prompting? Thank you so much.


Hidden Markov Trees

Hi,
Great work, sparked my interest in probabilistic programming!
I'm mostly interested in Hidden Markov Models and in particular in a variant called Hidden Markov Trees (not to be confused with hidden markov decision trees). Basically, the hidden states no longer form a chain, but a (binary) tree.
I wondered how I could implement this in bayespy.
My first idea was to create several 'CategoricalMarkovChain' objects and link their hidden states together:
# a0 is the initial distribution of chain a
# P is the prior over transition probabilities
A = CategoricalMarkovChain(a0, P, states=N_A)

# put the distribution over the last state of A as the initial distribution of B
lastA = A[-1] # somehow get the last variable of chain A (**)
B = CategoricalMarkovChain(lastA, P , states=N_B)

# same for C
C = CategoricalMarkovChain(lastA, P, states=N_C)

e.g. chains B,C "inherit" their initial state from chain A.
However, I have no clue how to access the the array of hidden states within a single chain (the line marked with (**) ). Is this possible somehow?
Alternatively one could of course build up the markov chains directly from Categorical, Gate, and Dirichlet variables, such that the hidden states exist explicitly as a list and can be linked between chains as required. However, I'm not sure if inference is still efficient this way (lot more messages/variables probably).

Hi, thanks for your message! I have never used Hidden Markov Tree but it seems interesting. I may have misunderstood something, but I came up with three possible solutions:
1) Use fully factorizing mean-field approximation.
As you described, you could write something like this to construct a binary tree structure:
A = Dirichlet(np.ones(D), plates=(D,))  # DxD transition matrix
z = Categorical(np.ones(D)/D) # initial state
z_0 = Mixture(z, Categorical, A)
z_1 = Mixture(z, Categorical, A)
z_00 = Mixture(z_0, Categorical, A)
z_01 = Mixture(z_0, Categorical, A)
z_10 = Mixture(z_1, Categorical, A)
z_11 = Mixture(z_1, Categorical, A)
# and so on..

Equivalent constrution is possible with Gate node, doesn't make a difference in this case. You could also have, for instance, different transition matrices for left and right branches if you wanted. The problem with this construction is that the posterior approximation factorizes with respect to the variables. Thus, the approximation might be poor and convergence slow, as you pointed out. Also, you should think in which order you want to update your variables. But at least this is easy to implement and test on small problems.
2) Use separate chains for the paths to each leaf node
In order to reduce the amount of factorization, you could factorize in branch level. That is, each branch is a new categorical Markov chain. I'm not good at explaining but the idea would be that each left "turn" would trigger a new branch and each right "turn" belongs to the same chain as its parent. Thus, sequences 1, 11, 111, 1111, ... would be in the same chain. Also, sequences 0, 01, 011, 0111, ... in the same chain. Also 10, 101, 1011, ... would be a new chain branching at the second level. So, you would have as many chains as you have leaf nodes, that is, 2^(K-1) where K is the height/depth of your tree. The path to each leaf is one chain, and the path starts from the last left "turn" before the leaf. However, this is a bit complicated construction and the factorization is not symmetric so it is difficult at least for me to tell what kind of biases this kind of factorization causes. This shold be possible to implement but there are a few minor details that you need to know: how to access individual states of a chain (the question you marked with (**) ). At the moment, that is slightly complicated, because the indexing is used for (independent) plates only. Thus, you need to do something like this to access individual state variables of a chain:
Z = CategoricalMarkovChain(np.ones(D)/D, np.ones(D,D)/D, states=100)   # dummy chain
from bayespy.inference.vmp.nodes.categorical import CategoricalMoments
Z_tmp = Z._convert(CategoricalMoments)
Z_tmp[-1]  # access the last state
Z_tmp[2]   # access the third state

Yep, I know it's complicated, sorry. Should provide a simpler interface for that.
3) Write custom node for tree (or arbitrary graph) structure categorical variables
Quite often efficient inference and little factorization requires custom nodes that utilize some specific structure. For instance, GaussianMarkovChain implements a Kalman filter/smoother type algorithm and CategoricalMarkovChain implements the alpha-beta recursion. Thus, I guess the best solution is to write a custom node which implements some kind of belief propagation algorithm to perform exact inference for categorical Markov tree given some parameters. Then the challenge is in making the node as general and minimalistic as possible so it can be easily used as a building block in large models. However, I'm not too familiar with inference algorithms for Markov trees so I'm not sure how to implement it without studying a bit.
I hope I understood you correctly and I hope this helps. Please comment or ask further questions if you want.

Hi,
thanks for your detailed response. I'm currently kind of busy with other stuff, but hopefully I'll have some time the next days to look at the ideas you proposed; looks promising!
I discovered some slight mistake in my exampe above: It should of course read
lastA = A[-1] # somehow get the last variable of chain A (**)
linkerNode = Gate(lastA, P)
B = CategoricalMarkovChain(linkerNode, P , states=N_B)
C = CategoricalMarkovChain(linkerNode, P,  states=N_C)

i.e. the last hidden state of A determines the initial distribution of the two following chains.


fix: updated deps

:tada: This PR is included in version 2.1.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


add 17 to lts list

Checklist

[ ] You added tests to cover the change
[ ] mvn clean install build and test completes
[ ] You changed or added to the documentation

Fixes https://github.com/adoptium/api.adoptium.net/issues/172

Codecov Report

Merging #175 (3b17029) into main (68d67a9) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##               main     #175   +/-   ##
=========================================
  Coverage     64.79%   64.79%
  Complexity      199      199
=========================================
  Files           113      113
  Lines          3042     3042
  Branches        446      446
=========================================
  Hits           1971     1971
  Misses          851      851
  Partials        220      220

Impacted Files
Coverage Δ

.../kotlin/net/adoptopenjdk/api/v3/models/Versions.kt
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68d67a9...3b17029. Read the comment docs.


Remove debug console.log statements that I forgot to remove

Remove debug console.log statements that I forgot to remove

in the future, don't worry about running a build for a pr. we can just crank away in ./src, then fire off a build each release.
cheers


Support new gemfile name, and tighten entrypoint checks

Support new gemfile names gems.rb and gems.locked.
Default entrypoint generation checks for config.ru before suggesting an entrypoint that depends on it. Throw an error during dockerfile generation if an entrypoint can't be inferred and must be provided.

Fixes https://github.com/GoogleCloudPlatform/ruby-docker/issues/115


[proposal] Call PreStop hook before a Pod is terminated

Why is this needed:
For some offline Pods, there is a daemon process in the container. We want the daemon to do some cleanup, so the daemon expects to receive SIGTERM before the container terminates, so we want to use the prestop hook to send SIGTERM to the daemon.

/area api-machinery

@FillZpp Please evaluate this functional

@czhenly So you expect the daemon process in Pod to get SIGTERM before Pod terminating or container terminating?

@czhenly So you expect the daemon process in Pod to get SIGTERM before Pod terminating or container terminating?

Yeah, we want to send SIGTERM  to daemon process by prestop hook or other way. Now we send the SIGTERM signal through the prestop hook on K8s, but this way doesn't support koordinator-based