Receive Call Graphs from FASTEN

Hey @vitsalis
I tried to stitch the Call Graphs received from FASTEN (using your pycg-stitch tool), however, I get error messages and I guess they are caused by a different format of the received Call Graphs and the example Call Graphs.
I guess the endpoints are implemented by @MihhailSokolov and currently I am still using the docker deployment which says (if you scroll down) that the basic endpoints work for python. Indeed, I can receive information from there, but the question is, do we need special endpoints for python (like a different format) or am I using the wrong ones?
Thanks for your help!

Hello @amphioxis
I believe that indeed the basic endpoints work for Python. Maybe Mihhail can verify.
It would help a lot if you could provide a sample output call graph and the errors that it generates. Thanks!

Yes of course, this is the Call Graph for the astroid:2.5.7 package created by the callgraph endpoint:
[{"source_id": 77976, "target_id": 78350}, {"source_id": 77975, "target_id": 77976}, {"source_id": 77974, "target_id": 77976}, {"source_id": 77979, "target_id": 78471}, {"source_id": 77978, "target_id": 78020}, {"source_id": 77981, "target_id": 78696}, {"source_id": 77989, "target_id": 78696}, {"source_id": 77997, "target_id": 78696}, {"source_id": 78003, "target_id": 1317}, {"source_id": 78003, "target_id": 77980}]
The error message says:
Traceback (most recent call last):   File "pypi-Plugin-venv/fasten-pypi-plugin/src/fasten/main.py", line 10, in <module>     class Main:   File "pypi-Plugin-venv/fasten-pypi-plugin/src/fasten/main.py", line 37, in Main     StitchCallGraph().stitchCallGraph(pathsToCallGraphs.call_graph)   File "pypi-Plugin-venv/fasten-pypi-plugin/src/fasten/stitchCallGraph.py", line 13, in stitchCallGraph     stitcher = Stitcher(call_graphs)   File "pypi-Plugin-venv/lib/python3.9/site-packages/pycg_stitch-0.0.1-py3.9.egg/stitcher/stitcher.py", line 44, in __init__   File "pypi-Plugin-venv/lib/python3.9/site-packages/pycg_stitch-0.0.1-py3.9.egg/stitcher/stitcher.py", line 91, in _parse_cgs TypeError: list indices must be integers or slices, not str
If I switch to the metadata endpoint I got as "Call Graph":
{"package_name": "astroid", "version": "2.5.7", "metadata": {}}
and the error message says:
Traceback (most recent call last):   File "pypi-Plugin-venv/fasten-pypi-plugin/src/fasten/main.py", line 10, in <module>     class Main:   File "pypi-Plugin-venv/fasten-pypi-plugin/src/fasten/main.py", line 37, in Main     StitchCallGraph().stitchCallGraph(pathsToCallGraphs.call_graph)   File "pypi-Plugin-venv/fasten-pypi-plugin/src/fasten/stitchCallGraph.py", line 13, in stitchCallGraph     stitcher = Stitcher(call_graphs)   File "pypi-Plugin-venv/lib/python3.9/site-packages/pycg_stitch-0.0.1-py3.9.egg/stitcher/stitcher.py", line 44, in __init__   File "pypi-Plugin-venv/lib/python3.9/site-packages/pycg_stitch-0.0.1-py3.9.egg/stitcher/stitcher.py", line 91, in _parse_cgs KeyError: 'product'

I believe the error is due to the first call graph not following the FASTEN format. I am not very familiar with the API though.
Is this the expected behavior @ashkboos  @MihhailSokolov ?

I believe the error is due to the first call graph not following the FASTEN format. I am not very familiar with the API though.
Is this the expected behavior @ashkboos  @MihhailSokolov ?

I don't really know, sorry. I haven't worked with Python CGs. I only know that they have their own format which is described in the wiki.

I mean, pycg is creating the right format and it is running inside the fasten-pycg container.
So the question is:
If I feed the database with package information, this information is send to the kafka container. What happens with that information, does the pycg container use that information to create a Call Graph for those packages and why, how, and where does the Call Graph format changes?
Maybe @MagielBruntink knows?

Hi all,
I am recreating the Python/PyPi pipeline and running into issues with the Python callgraphs. It seems the MetadataDB plugin assumes the new callsite format used by JavaCgOpal, which is not currently produced by PyCG. This leads to NPE's in the MetadataDB plugin, see one of the logs below.
The problematic code is here: https://github.com/fasten-project/fasten/blob/142891de4147f9bb50503b3a3b08727d9be96cfa/core/src/main/java/eu/fasten/core/data/graphdb/ExtendedGidGraph.java#L82
I think that currently only works for Java, and fixes should be implemented for Python and C. @ashkboos @MihhailSokolov Please provide your thoughts on what should happen there.
{
   "input": {
      "input": {
         "requires_dist": [],
         "product": "namesss",
         "version_timestamp":<PHONE_NUMBER>,
         "version": "0.1.0"
      },
      "plugin_version": "0.0.1",
      "payload": {
         "product": "namesss",
         "metadata": {
            "loc": 21,
            "max_rss": 20048,
            "time_elapsed": 0.07,
            "num_files": 3
         },
         "nodes": 9,
         "forge": "PyPI",
         "generator": "PyCG",
         "depset": [],
         "version": "0.1.0",
         "modules": {
            "internal": {
               "/gen_names_bozydar/": {
                  "sourceFile": "gen_names_bozydar/__init__.py",
                  "namespaces": {
                     "2": {
                        "metadata": {
                           "last": 0,
                           "first": 0
                        },
                        "namespace": "/gen_names_bozydar/"
                     }
                  }
               },
               "/gen_names_bozydar.gen_names_bozydar/": {
                  "sourceFile": "gen_names_bozydar/gen_names_bozydar.py",
                  "namespaces": {
                     "1": {
                        "metadata": {
                           "last": 11,
                           "first": 1
                        },
                        "namespace": "/gen_names_bozydar.gen_names_bozydar/"
                     }
                  }
               },
               "/setup/": {
                  "sourceFile": "setup.py",
                  "namespaces": {
                     "0": {
                        "metadata": {
                           "last": 18,
                           "first": 1
                        },
                        "namespace": "/setup/"
                     }
                  }
               }
            },
            "external": {
               "names": {
                  "sourceFile": "",
                  "namespaces": {
                     "6": {
                        "metadata": {},
                        "namespace": "//names//names.get_full_name"
                     }
                  }
               },
               "setuptools": {
                  "sourceFile": "",
                  "namespaces": {
                     "3": {
                        "metadata": {},
                        "namespace": "//setuptools//setuptools.setup"
                     },
                     "4": {
                        "metadata": {},
                        "namespace": "//setuptools//setuptools.find_packages"
                     }
                  }
               },
               "argparse": {
                  "sourceFile": "",
                  "namespaces": {
                     "7": {
                        "metadata": {},
                        "namespace": "//argparse//argparse.ArgumentParser"
                     }
                  }
               },
               ".builtin": {
                  "sourceFile": "",
                  "namespaces": {
                     "5": {
                        "metadata": {},
                        "namespace": "//.builtin//len"
                     },
                     "8": {
                        "metadata": {},
                        "namespace": "//.builtin//print"
                     }
                  }
               }
            }
         },
         "graph": {
            "internalCalls": [],
            "externalCalls": [
               [
                  "0",
                  "3",
                  {}
               ],
               [
                  "0",
                  "4",
                  {}
               ],
               [
                  "1",
                  "5",
                  {}
               ],
               [
                  "1",
                  "6",
                  {}
               ],
               [
                  "1",
                  "7",
                  {}
               ],
               [
                  "1",
                  "8",
                  {}
               ]
            ],
            "resolvedCalls": []
         },
         "sourcePath": "/mnt/fasten/sources/n/namesss/0.1.0",
         "timestamp": "1581173493"
      },
      "created_at":<PHONE_NUMBER>,
      "plugin_name": "PyCG"
   },
   "plugin_version": "0.1.2",
   "err": {
      "stacktrace": [
         "java.base/java.util.Objects.requireNonNull(Unknown Source)",
         "java.base/java.util.Arrays$ArrayList.<init>(Unknown Source)",
         "java.base/java.util.Arrays.asList(Unknown Source)",
         "eu.fasten.core.data.graphdb.ExtendedGidGraph.lambda$toJSON$2(ExtendedGidGraph.java:82)",
         "java.base/java.util.HashMap.forEach(Unknown Source)",
         "eu.fasten.core.data.graphdb.ExtendedGidGraph.toJSON(ExtendedGidGraph.java:78)",
         "eu.fasten.analyzer.metadataplugin.MetadataDBExtension.produce(MetadataDBExtension.java:188)",
         "eu.fasten.server.plugins.kafka.FastenKafkaPlugin.handleProducing(FastenKafkaPlugin.java:275)",
         "eu.fasten.server.plugins.kafka.FastenKafkaPlugin.processRecord(FastenKafkaPlugin.java:261)",
         "eu.fasten.server.plugins.kafka.FastenKafkaPlugin.handleConsuming(FastenKafkaPlugin.java:186)",
         "eu.fasten.server.plugins.kafka.FastenKafkaPlugin.run(FastenKafkaPlugin.java:145)",
         "eu.fasten.server.plugins.kafka.FastenKafkaPlugin.start(FastenKafkaPlugin.java:163)",
         "java.base/java.util.ArrayList.forEach(Unknown Source)",
         "eu.fasten.server.FastenServer.run(FastenServer.java:206)",
         "picocli.CommandLine.executeUserObject(CommandLine.java:1919)",
         "picocli.CommandLine.access$1100(CommandLine.java:145)",
         "picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)",
         "picocli.CommandLine$RunLast.handle(CommandLine.java:2326)",
         "picocli.CommandLine$RunLast.handle(CommandLine.java:2291)",
         "picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)",
         "picocli.CommandLine.execute(CommandLine.java:2058)",
         "eu.fasten.server.FastenServer.main(FastenServer.java:352)"
      ],
      "error": "NullPointerException"
   },
   "consumed_at":<PHONE_NUMBER>,
   "host": "ec934779f2ae",
   "created_at":<PHONE_NUMBER>,
   "plugin_name": "MetadataDBPythonExtension"
}

I mean, pycg is creating the right format and it is running inside the fasten-pycg container.
So the question is:
If I feed the database with package information, this information is send to the kafka container. What happens with that information, does the pycg container use that information to create a Call Graph for those packages and why, how, and where does the Call Graph format changes?
Maybe @MagielBruntink knows?

Some further comments. The proper endpoint to download a callgraph would be /rcg not /callgraph but it won't work currently because the REST API will not properly handle the PyPi package name. The REST API is built for Maven, and while some of it works also for Python, not everything...
Fixes would be needed eg. here:
https://github.com/fasten-project/fasten/blob/142891de4147f9bb50503b3a3b08727d9be96cfa/analyzer/restapi-plugin/src/main/java/eu/fasten/analyzer/restapiplugin/mvn/api/impl/PackageApiServiceImpl.java#L124

Closed because the \rcg endpoint for python if working. Thx!


Multi-world calendar is missing modern

@Julexar would it be possible for you to implement the modern calendar?
I would really like to use this script :)

Modern is currently not implemented - I am working on adding ot alongside the other options though ^^

@Stexinator the Modern calendar has been implemented with the newest version (1.2).


Tutorial 2-structured-data - Internal Server Error

Fallowing tutorial on newly created project, after successful deploy there is only "Internal Server Error"
In app engine current version logs there is such error:
Error: 7 PERMISSION_DENIED: Missing or insufficient permissions.
at Object.exports.createStatusError (/srv/node_modules/grpc/src/common.js:87:15)
at Object.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:1188:28)
at InterceptingListener._callNext (/srv/node_modules/grpc/src/client_interceptors.js:564:42)
at InterceptingListener.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:614:8)
at callback (/srv/node_modules/grpc/src/client_interceptors.js:841:24)
Do anyone known what is the reason of this error ?

Hi, thanks for raising the issue. Did you follow the setup instructions and enable the required APIs?

I just tried tutorial 2. What does your config.json look like?

config.json I had created according to tutorial with own project id
{
  "GCLOUD_PROJECT": "something-something-333",
  "DATA_BACKEND": "datastore"
}

In the end I did manage to successfully deploy and run sahat/hackathon-starter
I'm not sure now what exactingly was the case, but I thing it was one of fallowing:

package.json

"engines": {
  "node": "10",
  "npm": ">=5.6.0"
}

app.yaml

runtime: nodejs
env: flex
manual_scaling:
  instances: 1

or enabling "Compute Engine" (I'm not sure when, and if I activated it, or it's activated by default, or it activated itself in one of my invalid configurations)

The most problematic from my perspective was error message, which wasn't telling anything specific more than "PERMISSION_DENIED"

Glad you got it running. Could you try to use this app.yaml:
runtime: nodejs10

That will use AppEngine Standard instead of Flex, which is much faster to deploy to. Thank you!

Thank you @fhinkel! It indeed worked with just this single line in app.yaml

Awesome. I'm closing this. Let me know if you run into more issues.


Question: error while building

Please check for duplicates or similar issues before asking your question.
What is your question?
i am wondering what is the reason for this  error?
note: Using new build system

note: Planning build

note: Constructing build description

error: Signing for "Funkin" requires a development team. Select a development team in the Signing & Capabilities editor. (in target 'Funkin' from project 'Funkin')

** BUILD FAILED **

warning: OpenGLES is deprecated. Consider migrating to Metal instead. (in target 'Funkin' from project 'Funkin')

never mind, relised that macos latest doesn’t support openGL


Check for already existing entries, when add mount

When a user adds a new mount there is a possibility, that entry with this name already exist.
Example:
gopass generate test 24
gopass mount add test ~/.test-password-store

After that one could not reveal test.gpg in the root store, until mount test is unmount.

Thanks for your PR. I'm not yet sure if should merge it.
When we designed gopass we modeled the mounts feature after regular Unix filesystem mounts. These act similar to how gopass does right now: If you mount a device to a mountpoint everything that was at the mountpoint before is invisible and inaccessible until the device is unmounted. To me this feels very natural and is a powerful feature.
But I admit that it may confuse some users. I think we should warn the users if they are attempting to shadow a mountpoint, but I wouldn't prevent it.

Hello, thanks for the detailed feedback! What do you think if we change to a warning and just tell a user that he/she shadowing another entry? Does that sound good?

Yes, if you change the error to a warning I'd merge it.

IMHO that sounds perfect. We shouldn't deny shadowing, as it allows users to build up their secrets tree individually. But a warning when shadowing is definitely a good thing.

I approve @zwopir's comment! 👍

@dominikschulz Changed to a warning :)

Thanks a lot!


feat(catalog): support rw_catalog.rw_sources/rw_materialized_views etc

Is your feature request related to a problem? Please describe.
In order to support the display of some richer information of different relations including mv, source, sink, index etc. i.e. support for columns including name, connector type, data format for source.
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response

Cc @Gogomoe , I have introduced some system tables under rw_catalog, could you please check and help to switch those queries on system tables under pg_catalog to these ones in cloud side? Feel free to ping me if you find any missing requirements like more columns or tables that could be added under rw_catalog. After that, I will do some refactoring on current implementation of system tables under pg_catalog to to ensure they are fully compatible with PostgreSQL. 🙏

Thanks @yezizp2012 !  corresponding cloud issue is created


Reworked CSS: Grid, RWD, font-size relativity

Let me know if you have any questions!

You are amazing - thank you. I just pulled the Minimal theme, and it was good, but it hasn't been updated in a while. :)


Camera interface white SDK: 22, the user manually shut down Camera permissions

Camera authority is not open, the camera interface is white, how to solve?
PermissionsAndroid only works at SDK> = 23
Will the SDK <23 how to detect whether the user manually closed by the user?
windows 10
phone: Samsung
Android ：5.0

Hey, thanks for reporting this issue!
It looks like your description is missing some necessary information. Can you please add all the details specified in the template? This is necessary for people to be able to understand and reproduce the issue being reported.
I am going to close this, but feel free to open a new issue with the additional information provided. Thanks!


uninitialized variable for ones_like

import tinyflow as tf

v0 = tf.Variable(tf.zeros([10, 10]))
v1 = tf.Variable(tf.ones_like(v0))
sess = tf.Session()
sess.run(tf.initialize_all_variables())

The code above will occur an uninitialized variable error. It is because that v0 will be added into read_var_nids due to ones_like, which is uninitialized at this moment and will be checked. (https://github.com/tqchen/tinyflow/blob/master/src/session.cc#L328)

fixed by https://github.com/tqchen/tinyflow/pull/10
position: https://github.com/tqchen/tinyflow/pull/10/files#diff-a39713a7caf4db9d7b5a9a5e778339b6R345
TODO: more strict rule


RSC in R4, and update some permission names

Docs Build status updates of commit af89066:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/graph-api/rsc/resource-specific-consent.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 6e041ed:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/graph-api/rsc/resource-specific-consent.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


add func and FUNCTION

add func and FUNCTION
#1
These two seem to be defined everywhere but I can't anything specifying where its guaranteed. The __func__ seems to be standardised in c++11 https://stackoverflow.com/a/4384825. While FUNCTION seem to be older and I can't find any info on when or where its specified.
@boris-kolpackov I think you have to decide what todo with these.

Hm, I see that this implementation makes __func__ only available in C++ but it seem like it's also available in C since C99 (and even earlier on compiler-specific basis): https://stackoverflow.com/questions/7008485/func-or-function-or-manual-const-char-id
I think we should just always define it and see what happens. Maybe also add a note that it's definitely available from C99 and C++11 but likely even earlier.

@boris-kolpackov sure, always defined it to 1 now :)


[Bug] Codeunit 7600 - ReverseSign

In codeunit 7600 Calendar Management, function ReverseSign, we believe we found a standard bug causing run-time errors. It could be fixed as follows:
EVALUATE(Formula,DateFormulaExpr);
NewDateFormulaExpr := CONVERTSTR(FORMAT(Formula),'+-','-+');
IF NOT (DateFormulaExpr[1] IN ['+','-']) THEN
  // >> ----- NEW
  IF NewDateFormulaExpr <> '<0D>' THEN
  // >> ----- NEW
    NewDateFormulaExpr := '-' + NewDateFormulaExpr;
EXIT(NewDateFormulaExpr);

Hi @AlexanderYakunin , this change did it not make into the latest bc insider images for BC 14 and BC 15.
Is it only postponed, or did you solve it differently (if so: where)? Thank you.

This is in but you will see it in 15.1 or 14.6

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.

Thanks for submitting this issue. We will publish a fix for it in the next minor update for release 15.

Thanks for submitting this issue. We will publish a fix for it in the next minor update for release 15.


Hide dock icon

Love this app. Thanks for making it free. I try to limit the number of apps appearing in my dock. In the next revision, if possible, would you be able to add an option in the clutch preferences to make it not add a visible clutch icon to the dock when clutch is open and running? Given that there's the clutch menubar icon always present when the agent is up, it would be nice to not also have the app icon appear in the dock.
Thanks

Hi, there shouldn’t be an icon in the dock unless the Clutch setup app is running. There are 2 parts, the Clutch setup app and Clutch Agent. Clutch Agent runs in the background and has the icon in the top bar. If you click the icon in the bottom dock, the Clutch setup app should appear. You can then quit the Clutch setup app (press Command-Q) and Clutch Agent will continue to run in the background.
If you already know this and the icon is appearing in the dock anyway: (1) what version of Mac OS are you running? (2) does this still happen if you restart? (3) what is the output when you run this command in Terminal: “ps aux | grep Clutch”?
Glad you like the app! I made it for myself but I see many people are using it now.

On Apr 13, 2020, at 5:04 PM, Jottle<EMAIL_ADDRESS>wrote:
Love this app. Thanks for making it free. I try to limit the number of apps appearing in my dock. In the next revision, if possible, would you be able to add an option in preferences to make to not add an icon to the dock when clutch is open and running? Given that there's the clutch menubar icon always present when the agent is up, it would be nice to not also have the app icon appear in the dock.
Thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Ah. Thanks for clarifying. I had the setup window open as you mentioned. No issue! Please delete this comment.

Ah okay; glad it's working properly. I'll leave it up in case anyone else has this question.


Use key value from events instead of codes

This should make the game work with different layouts.
I'm not sure how this would cope with non-latin alphabet layouts (e.g. Cyrillic or Hebrew) but it's working for me just fine switching between QWERTY and Colemak

Looks pretty good! We still probs want some filtering on the keyboard -- there are a few nonsensical consequences to the changes as-is. I'm guessing filtering for (alphabet + ' ') or something like that should resolve them :)


Egeria release 1.2 fix

This fix should just change OMRSMetadataCollectionBase at lines 5667 and 5921

This is not right - pulled content merged from master


How to fetch new location after the marker is drag to another position

Hi @tristandunn
Great plugin, really easy to implement
I found a way to drag the marker after the map is rendered thanks to this post http://dailyjs.com/2012/02/21/jquery-roundup/
What I'm not able to do is to grab the new location values after the marker is moved. Is there a event to bind this action ?
Thanks

The library itself doesn't support any sort of dragging events, but since you have access to the Google Maps objects it should be fairly easy to do. I have not tested this out, but based on the documentation it should be something like:
$("#location")
  .autoGeocoder()
  .bind("auto-geocoder.onGeocodeResult", function() {
    var marker = this.marker;

    marker.setDraggable(true);

    google.maps.event.addListener(marker, "position_changed", function() {
      var position = marker.getPosition();
    });
  });

Check out the API and Event references.
Also, note that I reuse the marker, so you may want to bind to another event or keep track of your event binding to avoid multiple listeners. And that the position_changed event is likely to be triggered by the input changing the position as well.

Thanks for your response.
I'll try this approach


Revert "Ignore CppIncrementalBuildIntegrationTest for now"

This reverts commit 983644ae0dbae1c6aff8955756a75bdf2d57cfd4.
To check if #16876 fixes the problems in CppIncrementalBuildIntegrationTest as well
Fixes https://github.com/gradle/gradle-private/issues/3365

@bot-gradle help

Currently I support the following commands:

@bot-gradle test {BuildStage} to trigger a build

e.g. @bot-gradle test SanityCheck plz
SanityCheck/CompileAll/QuickFeedbackLinux/QuickFeedback/ReadyForMerge/ReadyForNightly/ReadyForRelease are supported
test this means test ReadyForMerge
test and merge to trigger a pre-tested commit build
SanityCheck can be abbreviated as SC, ReadyForMerge as RFM, etc.
Specific builds:

AST: AllSmokeTestsReadyForMerge
AFT: AllFunctionalTestsReadyForMerge
ASB: AllSpecificBuildsReadyForMerge
ACC: AllConfigCacheTestsReadyForMerge
ACT: AllCrossVersionTestsReadyForNightly
APTN: AllPerformanceTestsReadyForNightly
AFTN: AllFunctionalTestsReadyForNightly
ACTR: AllCrossVersionTestsReadyForRelease
AFTR: AllFunctionalTestsReadyForRelease

@bot-gradle help to display this message
@bot-gradle clean to clear the conversation history

@bot-gradle test AFT

OK, I've already triggered AllFunctionalTestsReadyForMerge build for you.

@bot-gradle test and merge

OK, I've already triggered a build for you.

OK, I've already triggered a build for you.


multiple property bindings in a mapping can not be saved

I am working with ontop Protege plugin 1.7.1.
I have two mappings, both on the same SQL statement:
select * from XStudent
The mappings are:
:Person-{IDCode} a :Student .
:Person-{IDCode} :studentName {Name} .
This works.
Now I want to group both mappings together. I edit the first one into:
:Person-{IDCode} a :Student , :studentName {Name} .
When I press Update, a lengthy error message appears in the Protege log (see report.txt).
Even if I am doing something wrong, the messages should be more polite ..
Thanks,
Karlis Cerans
report.txt

Hi,
thanks for reporting the bug, we will provide a better error message to avoid the generation of this kind of mapping.
The union of the two mappings together is possible but it needs a semicolon instead of a colon:
:Person-{IDCode} a :Student ; :studentName {Name} .
The colon can be used to shorten the mappings when they have the same subject and predicate and different object for example:
:Person-{IDCode}  :studentName {Name} , {Surname} .


downloads.dockerslim.com doesn't resolve (binary releases are broken)

$ curl https://downloads.dockerslim.com/releases/1.37.3/dist_linux.tar.gz
curl: (6) Could not resolve host: downloads.dockerslim.com

$ dig dockerslim.com

; <<>> DiG 9.16.1-Ubuntu <<>> dockerslim.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 3244
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;dockerslim.com.			IN	A

;; Query time: 0 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>)
;; WHEN: Tue Nov 15 15:20:57 UTC 2022
;; MSG SIZE  rcvd: 43

I just ran into this as well, as osener points out the DNS doesn't seem to resolve to anything.
I do see that the domain was just updated recently (2022-11-15 05:31:57 UTC) (~12 hours ago).
I also see the nameservers are NS1.HOVER.COM and NS2.HOVER.COM if this domain lapsed payment, they may have disabled the dns records, I had this happened with a client and they had to get support to fix the DNS records as it completely wiped them out on the UI side.


Pretrained model

Can someone please provide a link to download the resnet101_caffe.pth if possible. I want to use it for a personnal project and i dont have a powerful machine. Thanks!

Jut found link for it. https://drive.google.com/drive/folders/1n2hWpTEWe3LwfOYq0VUslok-EmdqrMQP

Jut found link for it. https://drive.google.com/drive/folders/1n2hWpTEWe3LwfOYq0VUslok-EmdqrMQP

thanks !!!!!


[Fabric Bot] Reflect CODEOWNERS for Digital Twins

Summary
The focus of these changes is to update the "Service Attention" contacts for Digital Twins to reflect the most recent updates to CODEOWNERS.
References and Related

[DigitalTwins] Add Code Owners (#26718)

/check-enforcer override


Update tasmota.bin to version <IP_ADDRESS>

Just flashed ESP-01 with no issues.
Tollbringer

Thanks again!


Reusing EVP_CIPHER_CTX on ZLinux does not return same ciphertext for same input

We are trying to reuse an EVP_CIPHER_CTX object for multiple doing encryption/decryption operations in multiple iterations. The Key and IV doe snot change for us in each of those iterations.
After creating and initializing an EVP_CIPHER_CTX object using "EVP_CIPHER_CTX_new()", followed by  "EVP_EncryptInit(cipher_ctx, cipher, key_data, iv_data)" call, we call "EVP_CipherInit(cipher_ctx, NULL, NULL, NULL, -1)" to reset the EVP_CIPHER_OBJECT for the next encryption/decryption operation and then call EVP_EncryptUpdate() and EVP_EncryptFinal() APIs to do the encryption in each iteration.
I observed that it works perfectly fine on Linux and other platforms for all AES modes. But specifically on ZLinux with AES-CFB mode, we have an issue that, we are getting different cipher text in each iteration, for the same input text. The key and IV remains the same in each iterations. Linux and other platforms are returning same cipher text for AES_CFB mode as expected. On ZLinux also, XTS, GCM and other modes are also returning the same cipher text as expected. So, this is the problem specific to ZLinux and specific to AES-CFB mode.
Another observation was that, if I pass same "key_data" in my reset call as mentioned above, (like, EVP_CipherInit(cipher_ctx, NULL, key_data, NULL, -1)), instead of passing NULL for the key data, then I start getting the same cipher text as expected.
I checked the assembler code for AES-CFB for ZLinux in file "providers/implementations/ciphers/cipher_aes_hw_s390x.inc", and found that we are setting "adat->plat.s390x.res" to "0" in s390x_aes_cfb128_initkey() API, which gets called by the reset call EVP_CipherInit(). But after doing the encryption operation in API s390x_aes_cfb128_cipher_hw(), we are setting "adat->plat.s390x.res" to "n". I tested and confirmed that this is causing the problem I explained above. If I set "adat->plat.s390x.res" to "0" in s390x_aes_cfb128_cipher_hw(), then I start getting the expected results on ZLinux with AES-CFB.
Can you please review the change below in file "providers/implementations/ciphers/cipher_aes_hw_s390x.inc" which helped me to get the correct expected results and merge it to the OpenSSL codeline if it looks good or please suggest a fix for this issue and the reason for keeping "adat->plat.s390x.res" value as "n" in s390x_aes_cfb128_cipher_hw(). Please let me know if you think that the below change may break any other usecase.
@@ -164,7 +164,7 @@
     }

     memcpy(dat->iv, adat->plat.s390x.param.kmo_kmf.cv, dat->ivlen);
-    adat->plat.s390x.res = 0;
+    adat->plat.s390x.res = n;
     return 1;
 }

Thanks for reporting this issue.
It is difficult to read your patch from your comment above since you did not surround it with triple-backticks.
I think this is what you suggested:
diff --git a/providers/implementations/ciphers/cipher_aes_hw_s390x.inc b/providers/implementations/ciphers/cipher_aes_hw_s390x.inc
index c8282dbd08..1d82e3e1a1 100644
--- a/providers/implementations/ciphers/cipher_aes_hw_s390x.inc
+++ b/providers/implementations/ciphers/cipher_aes_hw_s390x.inc
@@ -164,7 +164,7 @@ static int s390x_aes_cfb128_cipher_hw(PROV_CIPHER_CTX *dat, unsigned char *out,
     }

     memcpy(dat->iv, adat->plat.s390x.param.kmo_kmf.cv, dat->ivlen);
-    adat->plat.s390x.res = n;
+    adat->plat.s390x.res = 0;
     return 1;
 }

We will review your change and report back here.

I edited the original post to add markers for the diff section.  Hope that clarifies things

@holger-dengler @ifranzki Could you please look at this?

ok, I'll try to understand whats going wrong.

After a first look, I'm not sure, that the proposed fix is the solution for this problem. We had a similar issue in 2021 with the iv in the hw-specific part of the context (commit 732a4d15b0da7c04437ea828b2915a691b6e38db).
The proposed fix may work for single-shot messages, but it will cause problems for multi-chunk cases. I'll try write some code, reproduce the problem and try to develop a proper fix for it.

@ashwanikadian According to the man Page, EVP_CIPHER_CTX_reset() has to be called on re-using the context.
From man EVP_ENCRYPTINIT:
...
EVP_CIPHER_CTX_reset()
    Clears all information from a cipher context and free up any allocated memory associated with it,  except  the  ctx  itself.  This  function
    should be called anytime ctx is reused by another EVP_CipherInit() / EVP_CipherUpdate() / EVP_CipherFinal() series of calls.
...

Are you aware of that?

@holger-dengler IMO that documentation is fairly outdated. We support calling EVP_CIPHER_CTX_init() without _reset().
However in practical use-cases you always reinitialize at least with a different IV and not call EVP_CIPHER_CTX_init()  with all NULL arguments. Nevertheless it would be better to support even this kind of reinitialization if other implementations support it.

@t8m Thanks for this hint. It looks like we're tracking partial processed blocks in the platform-specific part of the context. I've currently no idea, how this information can be reset with the current set of registered provider functions.

@ashwanikadian Can you please have a look at the following test code? Is this similar to your usecase?
If I run this test code below on my test env (s390x, openssl 3.0.9), I get the same ciphertext in all iterations. I also get the same results on my local machine (x86_64, openssl 3.1.4). Which openssl version do you use? There was a bug in the s390x-specific code, which has been fixed by commit 732a4d15b0da7c04437ea828b2915a691b6e38db. Can you check, if your openssl version has this commit? If not, can you please check, if the commit is fixing your problem?
cfb.c
#include <stdio.h>
#include <string.h>
#include <openssl/evp.h>

#define AES_BSIZE	16

static const unsigned char
key[16] = { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
	    0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f };

static const unsigned char
iv[16] =  { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
	    0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f };

static const unsigned char
plain[] = "This is the plain text";

static unsigned char
cipher[sizeof(plain)];

void hexdump(const char *msg, const unsigned char *b, size_t blen)
{
	size_t i, part;

	if (msg)
		printf("%s\n", msg);

	for (i = 0, part = 0; i < blen; i++) {
		printf("0x%02x ", (int)(b[i] & 0xff));
		part = (part + 1) % 8;
		if (!part)
			printf("\n");
	}
	if (part)
		printf("\n");
}

int main(void)
{
	EVP_CIPHER_CTX *ctx;
	int outl, i;

	memset(cipher, 0, sizeof(cipher));

	printf("init:\n");
	hexdump("plain", plain, sizeof(plain));
	hexdump("cipher", cipher, sizeof(cipher));

	ctx = EVP_CIPHER_CTX_new();
	if (!ctx) {
		fprintf(stderr, "EVP_CIPHER_CTX_new failed\n");
		return(99);
	}
	if (!EVP_EncryptInit(ctx, EVP_aes_128_cfb8(), key, iv)) {
		fprintf(stderr, "EVP_EncryptInit failed\n");
		return(99);
	}

	for (i = 1; i < 5; i++) {
		memset(cipher, 0, sizeof(cipher));
		if (!EVP_CipherInit(ctx, NULL, NULL, NULL, -1)) {
			fprintf(stderr, "EVP_CipherInit failed\n");
			return(i);
		}
		if (!EVP_EncryptUpdate(ctx, cipher, &outl, plain, sizeof(plain))) {
			fprintf(stderr, "EVP_EncryptUpdate failed\n");
			return(i);
		}
		if (!EVP_EncryptFinal(ctx, cipher, &outl)) {
			fprintf(stderr, "EVP_EncryptFinal failed\n");
			return(2);
		}
		printf("\nround %d:\n", i);
		hexdump("cipher", cipher, sizeof(cipher));
	}
	return 0;
}

Output:
$ uname -a
Linux a3545040.lnxne.boe 6.5.5-100.fc37.s390x #1 SMP Sat Sep 23 22:29:55 UTC 2023 s390x s390x s390x GNU/Linux
$ ./cfb
init:
plain
0x54 0x68 0x69 0x73 0x20 0x69 0x73 0x20
0x74 0x68 0x65 0x20 0x70 0x6c 0x61 0x69
0x6e 0x20 0x74 0x65 0x78 0x74 0x00
cipher
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x00 0x00

round 1:
cipher
0x5e 0xb4 0x4a 0x15 0x57 0x7e 0xd9 0xf3
0x80 0x33 0x2d 0x7f 0xd1 0xcf 0xd1 0xb2
0x0e 0xfc 0x9e 0x40 0x85 0x1e 0x6d

round 2:
cipher
0x5e 0xb4 0x4a 0x15 0x57 0x7e 0xd9 0xf3
0x80 0x33 0x2d 0x7f 0xd1 0xcf 0xd1 0xb2
0x0e 0xfc 0x9e 0x40 0x85 0x1e 0x6d

round 3:
cipher
0x5e 0xb4 0x4a 0x15 0x57 0x7e 0xd9 0xf3
0x80 0x33 0x2d 0x7f 0xd1 0xcf 0xd1 0xb2
0x0e 0xfc 0x9e 0x40 0x85 0x1e 0x6d

round 4:
cipher
0x5e 0xb4 0x4a 0x15 0x57 0x7e 0xd9 0xf3
0x80 0x33 0x2d 0x7f 0xd1 0xcf 0xd1 0xb2
0x0e 0xfc 0x9e 0x40 0x85 0x1e 0x6d

Hi @t8m and @holger-dengler
Thanks for you comments.
@holger-dengler I am using OpenSSL 3.0.12 for my testing. I verified that the commit you mentioned is already available in OpenSSL 3.0.12 code.
I tested you code and found it to be returning the same ciphertext as you also mentioned. But then I compared that with our code found that the difference is only in the EVP_CIPHER object.
You are fetching the cipher object EVP_aes_128_cfb8(), but in our code, we are fetching it like, EVP_CIPHER_fetch(NULL, "AES-128-CFB", NULL) and EVP_CIPHER_fetch(NULL, EVP_CIPHER_name(EVP_aes_128_cfb()), NULL). The ciphers fetched like the one mentioned now are not working as I had mentioned before.
So, if you change your following line
if (!EVP_EncryptInit(ctx, EVP_aes_128_cfb8(), key, iv)) {
to
if (!EVP_EncryptInit(ctx, EVP_CIPHER_fetch(NULL, "AES-128-CFB", NULL), key, iv)) {
then it starts failing with OpenSSL 3.0.12.

@ashwanikadian Ok, thanks. I'll have a look at it.


Correct choice of words in Getting started.

Actual Behaviour
I also noticed that the "Getting started" component references  “article” rather than Documentation.

Expected Behaviour
This section of the Documentation should be carefully reviewed and improved upon to become self-explanatory to users reading through. The words should be corrected too

Steps to reproduce it

Screenshots of the issue

Would you like to work on the issue?

I also noticed that the "Getting started" component references “article” rather than Documentation.

Hi @Anita-ihuman, I'm not clear what's article rather than docs 🤔 Could you please provide more details?

The documentation has a recurring use of the word "article", instead of documentation.

I want to work on this , could you please assign it to me.

@juzhiyuan if you can, please assign this task to @123liubao


Ref docs: Storage Blob - Blob item properties competes with Ref Doc properties

In the Azure/storage-blob ref docs, looking at the BlobItem doc, the object has a property named "properties", and the ref doc page also has a section named "Properties". Both are the HTML bookmarks, or at least function that way so that in order to find the properties, I have to scroll down. If I wasn't aware of how the ref docs worked (or didn't work), I would think something was missing.
Can I get a general sense of how many SDKs in the ref docs may be affected by this issue so it can get appropriately prioritized to fix?

@diberry I dont believe there is a straight forward way to get this information. For the new set of libraries, we have the api extractor to generate a file which has the user facing api. So, it would be an easy search here. But none of the management libraries that also live in this repo use the api extractor.

But all such packages that do not use the api extractor are auto generated using the swagger file that represents the service. So, you can search the swagger files to get a count.
Also, all this information is only for the packages that live in this repo, there are potentially other JS packages in Azure and elsewhere in other repos that all have their reference docs rendered in docs.microsoft.com


add OpenShift PSa featuregate to the default profile

PSa should be in enforcement mode in master
/assign @deads2k

/lgtm
/approve


Allow for non-word aligned mtvec

I just wanted to raise awareness that the RISC-V compliance suite currently seems not respect coarser alignment restrictions on the BASE field for mtvec of a given target implementation.
The minimum alignment mandated by the RISC-V Privileged Spec v.1.11 is 4-byte but it explicitly allows for coarser restrictions:

If mtvec is writable, the set of values the register may hold can vary by implementation. The value in the BASE field must always be aligned on a 4-byte boundary, and the MODE setting may impose additional alignment constraints on the value in the BASE field.
An implementation may have different alignment constraints for different modes. In particular, MODE=Vectored may have stricter alignment constraints than MODE=Direct.
Allowing coarser alignments in Vectored mode enables vectoring to be implemented without a hardware adder circuit.

Ibex requires mtvec to be 256-byte aligned. Since the RISC-V compliance suite currently requires a finer alignment, the I-EBREAK and I-ECALL tests keep failing.
We are not sure what your view on this is, but we thought this should maybe be tracked here. For further information, please also have a look here lowRISC/ibex#428, lowRISC/ibex#99, lowRISC/ibex#100.

I am also pinging @eroom1966 as we have discussed this a while ago.

The riscof compliance framework (currently under review) does have a method
of specifying restrictions on WARL fields, and the example you give is a
prime example of why.
However, the current compliance suite has poor coverage of any privileged
mode operations (and interrupts are an example of that, as are any
privileged mode CSRS),
And, the existing privileged mode tests are self-checking; they flag
mismatched expected result as a literal in the test, and the were written
to expect a Rocket core (or so I understand) , which doesn't have those
restrictions.
The upcoming compliance framework will instead compare an implementation
(Ibex in this example) with a formal model.
The implications of this is that the

the framework must be supplied with a description of the implementation
options
the formal model must use that description to restrict what it models
(in this case, it would ignore writes to MTVEC[7:0] and keep them as 0)
So, when the formal model is released with those capabilities (I believe it
is being worked on) then tests which compare instead of self-check should
run correctly.

More details are available here:
Targets and framework (incl. tests ) are now in separate repositories.
 RISCOF:              https://gitlab.com/incoresemi/riscof . YAML based

framework, test-pool and relevant docs.
RISCOF-Targets: https://gitlab.com/incoresemi/riscof-plugins plugins
for various targets
YAML description:https://github.com/riscv/riscv-config
On Fri, Oct 25, 2019 at 8:26 AM Pirmin Vogel<EMAIL_ADDRESS>wrote:

I am also pinging @eroom1966 https://github.com/eroom1966 as we have
discussed this a while ago.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/riscv/riscv-compliance/issues/72?email_source=notifications&email_token=AHPXVJXHHGH5FXKG67G427TQQMFZLA5CNFSM4JFEYYD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECIWI4Q#issuecomment-546399346,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHPXVJXRU626VY2L7BR4XZDQQMFZLANCNFSM4JFEYYDQ
.

Thanks @allenjbaum for your quick response. I am happy to learn that this problem is being addressed.

Well, it will be addressed in the future. It isn't clear to me that it is possible to fix this in v.1, though.

Therer are currently 4 tests: I-EBREAK, I-ECALL, I-MISALIGN_JMP and I-MISALIGN_LDST which require access and update of mtvec. While riscof accounts for capturing the platform specifc constraints for mtvec, these tests would also have to be updated along with a linker script changes as well to account for this issue.

Is this possible to fix in the current framework to handle arbitrary alignments?
if we only need to modify linker scripts which are already implementation specific, that is doable
If we need to modify tests, that will be implementation specific, and until we can import YAML parameters, we can't have a general test, in which case this should be deferred until we get to v.2.
I don't think we have a way of "deferring" issues; should this get a "fixed in riscof" tag?

I think the issue here is that the idiom adopted by Andrew Waterman was used early in the test development, so that the mtvec address is set to be the address of an instruction after the faulting address, in other words (pseudo code)
l1: some_instruction
l2: set mtvec = &l4
l3: faulting_instruction
l4: some_instruction
if the intention is to use a READONLY MTVEC, then a trap handler must be written and located at the fixed MTVEC address, this traphandler could read the EPC and perform a jump to EPC+2 (compressed) or EPC+4 (uncompressed) thus resuming execution at l4
I think it is a test issue, not a framework issue
Thx
Lee

Not only do these tests need to be different depending on whether the DUT
supports unaligned ops or not, but also requires different trap handling
depending on the WARL characteristics of mtvec. The latter might be able to
be handled with model specific linker script (and test changes), but the
former requires test changes or new tests (plus a way to select different
tests, plus different reference signatures) and that can't be done in v.1.
Unless someone volunteers to write more tests and linker scripts, I'd
rather defer this to v.2.
A better approach would be to replace the instruction at the MTVEC location
with a branch to the handler (so we don't care about mtvec alignment or
writability constraints), and restore it when the test ends. (that breaks
if the MTVEC location isn't RW - but then we can't test any of this anyway)
(It's actually more complicated than that, because for fixed MTVEC we can't
guarantee that the branch offset is long enough to get to the handler,
which means we need to either save a register first using mscratch, or
mscratch points to the actual handler)
I've been talking with Neel about a standard interrupt handling macro that
would alleviate this in general. In addition to saving a trap signature (at
the end of signature area, running backwards) it always returns to a 4B
aligned instruction past EPC, regardless of the length or alignment of the
op itself. Those tests will need NOP padding after an op that is expected
to trap. A standard test prolog, included by the RVTEST_CODE_BEGIN macro
can perform the MTVEC op save and replacement, and RVMODEL_HALT can restore
it. Not for v.1, though.
On Wed, Apr 1, 2020 at 8:02 AM eroom<EMAIL_ADDRESS>wrote:

I think the issue here is that the idiom adopted by Andrew Waterman was
used early in the test development, so that the mtvec address is set to be
the address of an instruction after the faulting address, in other words
(pseudo code)
l1: some_instruction
l2: set mtvec = &l4
l3: faulting_instruction
l4: some_instruction
if the intention is to use a READONLY MTVEC, then a trap handler must be
written and located at the fixed MTVEC address, this traphandler could read
the EPC and perform a jump to EPC+2 (compressed) or EPC+4 (uncompressed)
thus resuming execution at l4
I think it is a test issue, not a framework issue
Thx
Lee
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/riscv/riscv-compliance/issues/72#issuecomment-607302397,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHPXVJQCLOC4NXLRGPWX6MLRKNJR5ANCNFSM4JFEYYDQ
.

OK, the tests that have just merged have a standard trap handler that any test can use.
The prolog/setup sets mtvec to a trampoline in the handler that enables saving trap state.
If it is unable to set mtvec to that handler, it instead replaces whatever mtvec points to with the trampoline code.
Only if that fails does it give up & the test fails
A future version may allow implementors to point to a RWX area to place the trampoline table. So:

An implementation that has a fixed MTVEC that vectors to ROM will always fail
An implementation that has partially writable MTVEC that is initialized point to ROM when the initialization routine is entered will fail.
Otherwise, it should work.

Has this been retested with the updated RFQ tests?
If not, I suggest you reclone the entire riscv-arch-test (the new name for the old riscv-compliance repo) repo and try again
The new tests should save and overwrite the region that MTVEC points to with a trap vector table, and restore it after the test. Any traps will then get vectored to ther standard trap handler, save trap state, and return past the trap point.

Seeing no comments, I believe this has been fixed in the latest version of the tests; that is, the tests that were failing because of mtvec alignment shouldn't be failing now. I am closing this, but please reopen if you see further failures.


Replace reference to upstream builds

Upstream builds at AdoptOpenJDK are discontinued. Temurin is now the upstream build.
Suggest that users test with an alternate JDK publisher from the marketplace.

Will wait for a review


Flicker on Orientation Change

Description
When you switch from portrait to landscape mode and back, the buttons and screen flicker wildly.
CodeSandbox/Snack link
https://snack.expo.dev/@miles542/switch-orientation-bug
Steps to reproduce

Open snack on real device
Rotate device from portrait mode into landscape mode
Rotate device back into portrait mode
See flicker

NativeBase Version
3.4.7
Platform

[X] Android
[ ] CRA
[X] Expo
[X] iOS
[ ] Next

Other Platform
No response
Additional Information
https://user-images.githubusercontent.com/30126023/176980871-38f76866-e2ce-4ff9-9e45-b92577faed99.mp4

Hi @mmfire118, Thank you for reporting this issue. We will try to fix it at the earliest.


Gh/reusable wf

Description

Development notes

Checklist

[ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[ ] Added a description of this change in the relevant RELEASE.md file
[ ] Added tests to cover my changes


Implement refunds API functions

https://docs.mollie.com/reference/refunds-api

What a surprise. I already implemented it https://github.com/AdGoji/mollie/issues/20

I'll update README as part of this issue.


Remove @@servername from atatuservar tests (#3211)

Description
Removing @@servername from atatuservar tests since the actual server name may be different in other environments which would cause the tests to fail.
Signed-off-by: Rob Verschoor<EMAIL_ADDRESS>Test Scenarios Covered

Use case based - N/A

Boundary conditions - N/A

Arbitrary inputs - N/A

Negative test cases - N/A

Minor version upgrade tests - N/A

Major version upgrade tests - N/A

Performance tests - N/A

Tooling impact - N/A

Client tests - N/A

Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>5
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 74.809%

Totals

Change from base Build<PHONE_NUMBER>0:
0.0%

Covered Lines:
46469

Relevant Lines:
62117

💛  - Coveralls


MonoRepo

Merge everything into a monorepo, add a backend server, drag and drop, testing, new CI flow, staging servers and so much more.
@rtluu If all the tests pass, I think we can merge this and start branching off develop.

Let’s do it! 🚀


Robots may get more power than battery capacity

The current program seems to allow the robot to get more power than its battery capacity. Is this correct behavior?
The patch to correct this is as follows:
diff --git a/luxai2022/env.py b/luxai2022/env.py
index 8ddb15d..acd88dc 100644
--- a/luxai2022/env.py
+++ b/luxai2022/env.py
@@ -580,6 +580,7 @@ class LuxAI2022(ParallelEnv):
                 for agent in self.agents:
                     for u in self.state.units[agent].values():
                         u.power = u.power + math.ceil(self.env_cfg.ROBOTS[u.unit_type.name].CHARGE * weather_cfg["power_gain_factor"])
+                        u.power = min(u.power, u.battery_capacity)
             for agent in self.agents:
                 for f in self.state.factories[agent].values():
                     # Factories are immune to weather thanks to using nuclear reactors instead

Fixed in #79


Connect-MicrosoftTeams from Microsoft Teams PowerShell does not work on Azure Automation

I am attempting to connect to Microsoft Teams using Azure Automation. I am using the following code in Azure Automation PowerShell Runbook.
# Get the credential from Automation
$credential = Get-AutomationPSCredential -Name 'O365'
$userName = $credential.UserName
$securePassword = $credential.Password

$psCredential = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $userName, $securePassword

# Connect to Microsoft Teams
Connect-MicrosoftTeams -Credential $psCredential

I receive the following error:
Connect-MicrosoftTeams : One or more errors occurred.: Object reference not set to an instance of an object.
At line:9 char:1
+ Connect-MicrosoftTeams -Credential $psCredential
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : AuthenticationError: (:) [Connect-MicrosoftTeams], AadAuthenticationFailedException
    + FullyQualifiedErrorId : Connect-MicrosoftTeams,Microsoft.Open.Teams.CommonLibrary.ConnectMicrosoftTeams

Connect-MicrosoftTeams : One or more errors occurred.
At line:9 char:1
+ Connect-MicrosoftTeams -Credential $psCredential
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : AuthenticationError: (:) [Connect-MicrosoftTeams], AggregateException
    + FullyQualifiedErrorId : Connect-MicrosoftTeams,Microsoft.Open.Teams.CommonLibrary.ConnectMicrosoftTeams

Connect-MicrosoftTeams : Object reference not set to an instance of an object.
At line:9 char:1
+ Connect-MicrosoftTeams -Credential $psCredential
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : AuthenticationError: (:) [Connect-MicrosoftTeams], AdalException
    + FullyQualifiedErrorId : Connect-MicrosoftTeams,Microsoft.Open.Teams.CommonLibrary.ConnectMicrosoftTeams

Connect-MicrosoftTeams : Object reference not set to an instance of an object.
At line:9 char:1
+ Connect-MicrosoftTeams -Credential $psCredential
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : AuthenticationError: (:) [Connect-MicrosoftTeams], NullReferenceException
    + FullyQualifiedErrorId : Connect-MicrosoftTeams,Microsoft.Open.Teams.CommonLibrary.ConnectMicrosoftTeams

Connect-MicrosoftTeams : One or more errors occurred.: Object reference not set to an instance of an object.
At line:9 char:1
+ Connect-MicrosoftTeams -Credential $psCredential
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Connect-MicrosoftTeams], AadAuthenticationFailedException
    + FullyQualifiedErrorId : Microsoft.Open.Teams.CommonLibrary.AadAuthenticationFailedException,Microsoft.Open.Teams.C
ommonLibrary.ConnectMicrosoftTeams

When I run a simple 'Connect-MicrosoftTeams' locally it works fine.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1ee082c9-735d-3f8a-7bc4-b2d4f175c784
Version Independent ID: 50e0366c-46b0-71da-b6f6-f422adaad785
Content: Connect-MicrosoftTeams (teams)
Content Source: teams/teams-ps/teams/Connect-MicrosoftTeams.md
Service: teams-powershell
GitHub Login: @kenwith
Microsoft Alias: kenwith

@officedocsbot assign @yogkumgit

Hello @AndrewCi Andrew, how are you?
As here we are limited to documentation fixes, I tested your scenario in my lab to verify if we need to maybe add a note in the article. I successfully connected and ran Get-Team from an azure runbook.
Hope this info helps you.

Forgot to recommend to open a support ticket directly in your tenant in case you still face issues.

@get-itips  Thank you very much for the contribution and sharing this explanation. @AndrewCi
Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.

Hello @AndrewCi Andrew, how are you?
As here we are limited to documentation fixes, I tested your scenario in my lab to verify if we need to maybe add a note in the article. I successfully connected and ran Get-Team from an azure runbook.
Hope this info helps you.

can you share your connection code? it's still not working for me

@get-itips Thank you very much for the contribution and sharing this explanation. @AndrewCi
Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.

can you please re-open this issue? it's not resolved.

can you share your connection code? it's still not working for me. i believe something needs to be added to the documentation to illustrate the different between prompted connection and programmatic connection.

Hello, it is no different from what you are doing I think.
$credential = Get-AutomationPSCredential -Name '365Demo'
$userName = $credential.UserName
$securePassword = $credential.Password

$psCredential = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $userName, $securePassword

Connect-MicrosoftTeams -Credential $psCredential

Get-Team

If you still face issues, I would recommend reinstalling the module into the automation account and/or opening a support ticket directly in your tenant.
Hope this info helps you.

@AndrewCi Did you get get a resolution for this? I'm running into the same problem.


[1.10] Removing a network during shutdown leaves Docker inoperable

Description of problem:
If you have a Docker Network plugin that tries to remove its network on shutdown, and you restart the Docker daemon, it then restarts continually.
docker version:
Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:36:33 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:36:33 2016
 OS/Arch:      linux/amd64

docker info:
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 631
Server Version: 1.10.0
Storage Driver: aufs
 Root Dir: /home/docker/aufs
 Backing Filesystem: extfs
 Dirs: 564
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 3.13.0-71-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.829 GiB
Name: bbvlx1
ID: IVMU:DIYM:GNIF:DNSW:GCHY:42Z5:R573:MDYB:YWOK:W57O:P4WY:2LQS
Username: bboreham
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

uname -a:
Linux bbvlx1 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):
VirtualBox
How reproducible:
100%
Steps to Reproduce:
Using Weave Net 1.4.3:
$ weave launch-router && weave launch-plugin
3c19d3e02d07e13967690044b27e6029bd3a6d0f2623984200a0e4ff8f3a355d
e183a8e9973cc1331627386ad34158602cb13a3f1bf27b28a0793a8a47871b03
$ sudo service docker restart

Actual Results:
Docker log from the restart onwards:
INFO[0017] Processing signal 'terminated'
ERRO[0017] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
ERRO[0017] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
INFO[0027] Container 975bf48d15c0d8128e47bf27d1ed9615226f3012de4fea7233e694629b844a71 failed to exit within 10 seconds of SIGTERM - using the force
WARN[0027] Unable to connect to plugin: /run/docker/plugins/weavemesh.sock, retrying in 1s
ERRO[0027] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
ERRO[0027] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
ERRO[0027] Handler for GET /v1.21/containers/weave/json returned error: write unix /var/run/docker.sock->@: write: broken pipe
2016/02/09 10:23:21 http: multiple response.WriteHeader calls
WARN[0028] Unable to connect to plugin: /run/docker/plugins/weavemesh.sock, retrying in 2s
ERRO[0028] Error during layer Store.Cleanup(): device or resource busy
Waiting for /var/run/docker.sock
INFO[0000] [graphdriver] using prior storage driver "aufs"
/var/run/docker.sock is up
INFO[0000] Graph migration to content-addressability took 0.00 seconds
INFO[0000] Firewalld running: false
WARN[0000] Could not get list of networks during endpoint cleanup: could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
ERRO[0000] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
INFO[0000] Default bridge (docker0) is assigned with an IP address <IP_ADDRESS>/16. Daemon option --bip can be used to set a preferred IP address
ERRO[0001] Error during layer Store.Cleanup(): device or resource busy
FATA[0001] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (<IP_ADDRESS>): Address already in use
/var/run/docker.sock is up
INFO[0000] [graphdriver] using prior storage driver "aufs"
INFO[0000] Graph migration to content-addressability took 0.00 seconds
INFO[0000] Firewalld running: false
WARN[0000] Could not get list of networks during endpoint cleanup: could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
ERRO[0000] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
INFO[0000] Default bridge (docker0) is assigned with an IP address <IP_ADDRESS>/16. Daemon option --bip can be used to set a preferred IP address
ERRO[0001] Error during layer Store.Cleanup(): device or resource busy
FATA[0001] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (<IP_ADDRESS>): Address already in use
/var/run/docker.sock is up
INFO[0000] [graphdriver] using prior storage driver "aufs"
INFO[0000] Graph migration to content-addressability took 0.00 seconds
INFO[0000] Firewalld running: false
WARN[0000] Could not get list of networks during endpoint cleanup: could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
ERRO[0000] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store
INFO[0000] Default bridge (docker0) is assigned with an IP address <IP_ADDRESS>/16. Daemon option --bip can be used to set a preferred IP address

ad infinitum
Expected Results:
Calm restart of Docker process; perhaps one or two error messages noting that the plugin did something inadvisable.
Additional info:
The crux of the problem seems to be that the Weave Net plugin attempts a RemoveNetwork via the API in the middle of the shutdown sequence.  Removing this call avoids the issue.
The same plugin code didn't break Docker 1.9.

ping @mavenugo

@bboreham restarting continuously is due to the systemd configuration on failed start. So the question here is the reason for the daemon failure during the start.
ERRO[0001] Error during layer Store.Cleanup(): device or resource busy
FATA[0001] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (<IP_ADDRESS>): Address already in use

I think I have an idea.... We will push a fix soon.

This issue is not full resolved. reopening.


Feature #769: Reject duplicate properties with different capitalization

Closes getodk/central#769
What has been done to verify that this works as intended?
Added tests and manual verification.
Why is this the best possible solution? Were any other approaches considered?
I initially changed the database column type to citext to stop duplicate property names (case insensitive) but that prevented creation even when existing properties were unpublished. With the implemented approach we can have two unpublished Forms with same properties but different capitalization. Error is thrown when user tries to publish second one.
How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?
Change affects already complicated area of Form and EntityList creation, that should be tested thoroughly.
Does this change require updates to the API documentation? If so, please update docs/api.yaml as part of this PR.
Yes
Before submitting this PR, please make sure you have:

[x] run make test and confirmed all checks still pass OR confirm CircleCI build passes
[x] verified that any code from external sources are properly credited in comments or that everything is internally sourced

I initially changed the database column type to citext to stop duplicate property names (case insensitive) but that prevented creation even when existing properties were unpublished.

I think it's also possible that there are already entity lists in production with property names that differ only on case. We want to prevent new properties like that, but I don't think we can add a database constraint if there could be existing properties like that.


Docker compose no longer strips dashes and underscores, Dockstation still does

As of the 4/10 change in Docker Compose:
"Dashes and underscores in project names are no longer stripped out."
From: https://github.com/docker/compose/blob/master/CHANGELOG.md
There is now a breaking inconsistency between containers accessed with Docker Compose and Dockstation in names like "my-project":
Docker Compose references container: my-project_service
Dockstation references container: myproject_service

Hi. Thanks for feedback. We will fix this issue in next release.


Visdom cannot plot small numbers?

I was using viz.line() tool to plot learning curves. When I am trying to overfit on a small dataset, the loss quickly converges to 0 from around 1. But visdom seem not to plot very small numbers. I am talking about something in 1e-6 or 1e-9 in magnitude.
Is this a normal issue?

What do you mean by "visdom [does] not ... plot very small numbers"? It rounds them to zero?
I presume this is related to what happens to numbers when they're serialized by json.dumps: https://github.com/facebookresearch/visdom/blob/master/py/init.py#L240
There are several topics about how to change the rounding behavior of this function on StackOverflow, but most of them seem to involve getting less precision rather than more precision in the serialized numbers...


Backwards RSI logic?

Hi, As I understand it when trading using RSI indicators:
Buy when RSI rises above the low limit (30 in the default logic)
Sell when RSI drops below the upper limit (70, in the default logic)
It appears as though Zenbot is buying when the price raises above the upper limit (70), and selling when it drops below the lower (30).
I can see below thatat 12:40:15 RSI hit 70, and triggered an attempt to purchase, which doesn't make sense because that indicates the market is oversold (why would we purchase here)?
`01/19/2017 12:40:15 PM PST [    reporter]    m24747639       2 trades.           ++++       RSI: 70 CLOSE:      $907.00 USD gdax.BTC-USD BAL:       $99.76 ROI:1.000 [link]'
'01/19/2017 12:40:15 PM PST [      trader] gdax.BTC-USD trend:  UP not enough funds ($1.75 USD) to execute min. buy 0.01, aborting trade! [link]'
'01/19/2017 12:40:15 PM PST [      trader] gdax.BTC-USD attempting to buy 0.00189704 BTC for $1.72 USD [link]'
'01/19/2017 12:40:15 PM PST [      trader] gdax.BTC-USD RSI:70 trend: null -> UP [link]`

I noticed the same thing. You are correct in that the buy/sell signals are backwards. In fact, one of the first things I tried doing was reversing them and running against the historic data. Funny enough, It gave nearly the same results. My opinion is that buying and selling your full bankroll based only on RSI and the time since the last transaction is not a very good strategy. Sure you can come up with decent returns on a particular dataset, but that means nothing for next week. I found it a little more adaptable to base the signals off MACD and compare that to the RSI strength to help determine how much to wager. I played around with other TA indicators but I am a believer that adding more indicators is just adding more noise. Pure price is king IMHO. I recommend writting a logic that is based off your strategy. Really just more of a conditional statements to trigger the buys or sells you would do when the price reaches certain areas instead of buying/selling based on what a TA tells you. Besides, watching the market I think whales really gobble up TA based bot orders.

And for that I am planning to spend a few days slogging through this code to understand everything I am looking at...  ;-)
Right now it looks like Greek.

For me, playing with zenbot was just a motivating excuse to learn node. Once I can have more time to test I will submit some pull requests to this project
In the meantime, I found this package useful to add other indicators such as MACD:
https://www.npmjs.com/package/technicalindicators

Sorry to butt in since I won't be submitting any code, but I wanted to comment that I think the logic of the original code wasn't necessarily backwards, but just that the mindset behind was the opposite of what most people would think of when using ZenBot. Basically, as the code is written, the focus is more on increasing the USD value of holdings, rather than the BTC/ETH/etc amount.
I think the point was more that the original code is focused on growing USD holdings whereas we'd rather increase crypto holdings, if that makes any sense. Point is I'm in support of the new request.

@robsteady Hi!
I watched the trader for a few days and all it appeared to do was buy high ($ value) and sell low.
Since reversing the logic it now sells high and buys low, increasing USD value (and BTC holdings).
At least I think that's what it is doing.
Adviser mode's ROI has done some screwy things on me so I can't trust it's results.  Yesterday before I implemented my changes it looks like it attempted to "buy" a negative amount of BTC and that somehow skyrocketed my ROI to 10.5 with a USD value of over 100k, haha.
But with me manually making trades based on the advisers logic, my USD value has gone up around 1% (after 18 hours of work).
So...going to run a live test with Zenbot trading today and see how it manages $100.

I also grabbed the new logic code and switched to it last night. It made a more sensible purchase on a low RSI. After this is verified to be a proper move, maybe we should have a look at the 24 hour restrictions.

Agreed, I've actually had my own copy set a lot lower just to see how it performs when allowed to purchase/sell more often.

So considering this afternoon's crash, I'm wondering if that's why the logic was written the way it was. Maybe the idea was that RSI was being used as a safety against a major crash (selling when RSI is down) and making sure we profit off of a spike (buying when RSI is up). It does seem counter-intuitive, but when bigger moves happen, it does actually make sense. Just figured I'd throw out some food for thought.

As long as the upper and lower limits of RSI were set accordingly the
"crash" this afternoon wouldn't affect anything.
My bot purchased at 890 (RSI 23), price rose to 894 (RSI 55, didn't trigger
purchase), price dropped to 880 or so (RSI 15 or so, bot tried to purchase
more but didn't have the cash to), price just raised back up to 895 (RSI
75) triggering a sell as it breached the RSI upper limit 65.
I made $13 :-)
I can't imagine a circumstance where you would trade RSI in reverse, and
can't find any reason in my reading of RSI trading techniques.
On Jan 24, 2017 4:46 PM, "R. Samuel Tiedemann"<EMAIL_ADDRESS>wrote:

So considering this afternoon's crash, I'm wondering if that's why the
logic was written the way it was. Maybe the idea was that RSI was being
used as a safety against a major crash (selling when RSI is down) and
making sure we profit off of a spike (buying when RSI is up). It does seem
counter-intuitive, but when bigger moves happen, it does actually make
sense. Just figured I'd throw out some food for thought.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/carlos8f/zenbot/issues/102#issuecomment-274986371,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJqpeUtJeRj0xgde5IORSwpS4V5CZKQdks5rVptIgaJpZM4Lom3C
.

Did we come to a consensus here on the backwards logic? Buying when RSI is low and selling when RSI is high seems like the right strategy here but I think zenbot is doing the exact opposite...

Actually upon closer inspection (actually just careful reading of the README), it seems like zenbot is using a very short window RSI (14-hours) to calculate its RSI. I think that given this very short window, the RSI can be seen as more of a predictor of major swings and thus when the strength is high, an uptrend is just beginning and vice versa.

It may look counter-intuitive, but simulations throw a better ROI as it is.
By the way, a RSI lower threshold of 25, instead of 28, makes a better ROI than default settings.

Yup, i also understand that this logic buys/sells based on trends, not on absolute peaks/pits. So when things start to go down the drain, i would consider it a rather good idea to dump my assets before they hit rock bottom. Same for buying: how do you know it's the lowest low, the right time to buy? I think it's better to buy when things are just starting to peak again, because then you have a higher chance of actually having assets when that peak hits.
But the 24 hours seem a bit long, indeed.

That is why the bot simulation was tested in a 3 months lapse.

Zenbot 4 is out, so I'm cleaning up 3.x issues. Please use and test 4, thanks!


"Layout Default" box is seen at the top of CONTRIBUTING.md document & Table of contents start from 0th index

"Layout Default" box is seen at the top of CONTRIBUTING.md document & Table of contents start from 0th index. Screenshot attached for reference.

I can fix this if it's issue.

This is not an issue.


interactive transactions with extended client for RLS in postgres causes blocking queries in postgres

We are implementing RLS and we decided to do a test before actually applying any polices. So we extended the prisma client and ran our code in our services with the new client.
We started noticing some odd blocking behavior that didn't exist before. When we investigated further it looks like the extending prisma client is executing parts of interactive
transactions within separate transactions only when we have extended the the model and override query: {   $allModels: {
If we use the doNothing version that extends the client but doesn't actually do any work everything works as expected.
How to reproduce

Within a transaction do a select for update like the example code that also does an update on the record it has selected
Insert a record with a null processed date to cause the code to find a row
Run this query in postgres to find blocked queries

SELECT
    activity.pid,
    activity.usename,
    activity.query,
    blocking.pid AS blocking_id,
    blocking.query AS blocking_query
FROM pg_stat_activity AS activity
         JOIN pg_stat_activity AS blocking ON blocking.pid = ANY(pg_blocking_pids(activity.pid));

If you have 10 records that all get pulled at the same time you will see 10 update queries that are blocked now.
Expected behavior
Extended client should behave the same as the regular client. Within the same transaction I should be able to update rows I pulled out with a row lock. The behavior I am seeing from prisma makes me believe these queries are running in separate transactions when we have extended the client.
Prisma information

generator client {
  provider      = "prisma-client-js"
  binaryTargets = ["native", "linux-arm64-openssl-1.1.x"]
}

generator types {
  provider = "prisma-generator-types"
  output   = "../../../client/src/interfaces/schema.ts"
}

datasource db {
  provider = "postgresql"
  url      = env("PORTAL_DB_URL")
}

model Notification {
  id                           Int       @id @default(autoincrement())
  userId                       Int       @map("user_id")
  content                      String?   @db.VarChar
  toAddress                    String    @map("to_address") @db.VarChar
  fromAddress                  String    @map("from_address") @db.VarChar
  subject                      String?   @db.VarChar
  processedDate                DateTime? @map("processed_date") @db.Timestamptz(6)
  createdDate                  DateTime? @default(now()) @map("created_date") @db.Timestamptz(6)
  updatedDate                  DateTime? @updatedAt @map("updated_date") @db.Timestamptz(6)
  deliveryId                   String?   @map("delivery_id") @db.VarChar
  notificationDeliveryStatusId Int       @map("notification_delivery_status_id")
  notificationMethodId         Int       @map("notification_method_id")

  @@map("notification")
}

prisma-client
import {PrismaClient, Prisma} from '@prisma/client';

/**
 * We expose a single Prisma client so that we do not have multiple connection pools. This will limit us to exactly 1
 * connection pool.
 * @see https://github.com/prisma/prisma/issues/1983
 */
// you can use this to see the actual PSQL queries that prisma is running
export const prisma = new PrismaClient({
    log: ['info', 'warn', 'error'],
    // log: ['query', 'info', 'warn', 'error'],
    transactionOptions: {
        timeout: 60000,
    },
});

/**
 * This function is used to bypass the RLS policies for a given transaction. Use carefully
 */
export const bypassRLS = () =>
    Prisma.defineExtension((iprisma) =>
        iprisma.$extends({
            query: {
                $allModels: {
                    async $allOperations({args, query}) {
                        const [, result] = await iprisma.$transaction([
                            iprisma.$executeRaw`SELECT set_config('app.bypass_rls', 'on', TRUE)`,
                            query(args),
                        ]);
                        return result;
                    },
                },
            },
        })
    );

export const doNothing = () =>
    Prisma.defineExtension((iprisma) =>
        iprisma.$extends({})
    );

/**
 * This function sets our tenant id for the transaction, this allows our RLS policies to work.
 * @param tenantId
 */
export const forTenant = (tenantId: number) =>
    Prisma.defineExtension((iprisma) =>
        iprisma.$extends({
            query: {
                $allModels: {
                    async $allOperations({args, query}) {
                        const [, result] = await iprisma.$transaction([
                            iprisma.$executeRaw`SELECT set_config('app.tenant_id', '${tenantId}', TRUE)`,
                            query(args),
                        ]);
                        return result;
                    },
                },
            },
        })
    );

export const getDbClient = (tenantId: number) =>
    // type cast needed because of https://github.com/prisma/prisma/issues/19949
    prisma.$extends(forTenant(tenantId)) as typeof prisma;

export const getDbClientBypassRLS = () =>
    // type cast needed because of https://github.com/prisma/prisma/issues/19949
    prisma.$extends(bypassRLS()) as typeof prisma;

export const getDBClientNothing = () =>
    // type cast needed because of https://github.com/prisma/prisma/issues/19949
    prisma.$extends(doNothing()) as typeof prisma;
export default prisma;

email poller
import {from, interval, Subscription, switchMap} from 'rxjs';
import {getDBClientNothing} from './prisma-client';

import {
    exponentialBackoffRetry,
    getRecordsOfType,
    markNotificationAsBounced,
    markNotificationInitiated,
} from './poll-utils';

const NOTIFICATIONS_POLL_DELAY_MS = 2000;
const NOTIFICATIONS_POLL_BATCH_SIZE = 500;
export default function setupEmailSubscriber(): Subscription {

    console.log(
        `Listening for Email Notifications ${NOTIFICATIONS_POLL_DELAY_MS}ms ${NOTIFICATIONS_POLL_BATCH_SIZE}`
    );

    const processRecords$ = interval(NOTIFICATIONS_POLL_DELAY_MS).pipe(
        switchMap(() =>
            from(
                getDbClientBypassRLS().$transaction(async (tx) => {
                    console.log('email transaction');
                    const records = await getRecordsOfType(
                        1,
                        tx
                    );

                    const trySendEmail = async (record: any) => {
                        try {
                            await markNotificationInitiated(
                                1,
                                tx,
                                record.id
                            );

                            if (record.content && record.subject) {
                                console.log(`Sending email to ${record.to_address}`);
                            }
                        } catch (e) {
                            console.log(`Email Polling Error: ${e}`);
                            // TODO: handle failed messages / pick a status or retry
                            await markNotificationAsBounced(record.id, getDbClientBypassRLS());
                        }
                    };

                    await Promise.all(records.map(trySendEmail));
                    console.log('email transaction done');
                })
            )
        ),
        exponentialBackoffRetry()
    );

    return processRecords$.subscribe();
}

poll-utils
import {retry, timer} from 'rxjs';
import {Prisma} from '@prisma/client';

import {
    Notification,
} from '@prisma/client';

type CamelToSnakeCase<S extends string> = S extends `${infer T}${infer U}`
    ? `${T extends Capitalize<T> ? '_' : ''}${Lowercase<T>}${CamelToSnakeCase<U>}`
    : S;

export type KeysToSnakeCase<T> = {
    [K in keyof T as CamelToSnakeCase<string & K>]: T[K];
};

export interface SnakeCaseNotification extends KeysToSnakeCase<Notification> {
}

export const markNotificationInitiated = async (
    notificationMethod: number,
    tx: Prisma.TransactionClient,
    recordId: number,
    deliveryId?: string
): Promise<void> => {
    console.log(`Marking notification as initiated: ${recordId}`);
    await tx.notification.updateMany({
        where: {
            id: recordId,
        },
        data: {
            notificationMethodId: notificationMethod,
            processedDate: new Date(),
            notificationDeliveryStatusId: 1,
            deliveryId,
        },
    });
    console.log(`Notification marked as initiated: ${recordId}`);
};

// TODO: we will want this to retry on bounce
export const markNotificationAsBounced = async (
    recordId: number,
    tx: Prisma.TransactionClient
): Promise<void> => {
    console.log(`Marking notification as bounced: ${recordId}`);
    await tx.notification.updateMany({
        where: {
            id: {
                in: [recordId],
            },
        },
        data: {
            processedDate: new Date(),
            notificationDeliveryStatusId: 2,
        },
    });
    console.log(`Notification marked as bounced: ${recordId}`);
};

export const getRecordsOfType = async (
    notificationMethod: number,
    tx: Prisma.TransactionClient
): Promise<SnakeCaseNotification[]> => {
    console.log(`Polling for ${notificationMethod} notifications`);
    const records = await tx.$queryRaw<SnakeCaseNotification[]>`
        SELECT *
        FROM global.notification
        WHERE processed_date IS NULL
          AND notification_method_id = ${notificationMethod}
        ORDER BY created_date ASC
        LIMIT 500 FOR UPDATE SKIP LOCKED;
    `;
    console.log(
        `Found notifications ${records.length} for type ${notificationMethod}`
    );
    return records;
};

export const exponentialBackoffRetry = (count: number = 5) =>
    retry({
        count,
        resetOnSuccess: true,
        delay: (err, currentRetry) => {
            const delayMs = 2 ** currentRetry * 1000;
            console.log(`Error processing notification: ${err}`);
            console.log(`Retrying notification poll with delay ${delayMs}ms`);
            return timer(delayMs);
        },
    });

Environment & setup

OS: MacOS
Database: Postgres version: 15.4
Node.js version: v16.16.0

Prisma Version
5.11.0

While trying to narrow this down further, I seem to be observing that any queryRaw causes that query to be run on a separate db connection & transaction. Modifying the above code to only query and update a record and then try to query the updated record and validate the value is in fact changed within the transaction fails. Here is the example code I used to cause this:
I just replace the email-poller code
import {from, interval, Subscription, switchMap} from 'rxjs';
import {getDbClientBypassRLS} from './prisma-client';

import {
    exponentialBackoffRetry,
    getRecordsOfType,
    markNotificationAsBounced,
    markNotificationInitiated,
    SnakeCaseNotification,
} from './poll-utils';

const NOTIFICATIONS_POLL_DELAY_MS = 2000;
const NOTIFICATIONS_POLL_BATCH_SIZE = 500;
export default function setupEmailSubscriber(): Subscription {

    console.log(
        `Listening for Email Notifications ${NOTIFICATIONS_POLL_DELAY_MS}ms ${NOTIFICATIONS_POLL_BATCH_SIZE}`
    );

    const processRecords$ = interval(NOTIFICATIONS_POLL_DELAY_MS).pipe(
        switchMap(() =>
            from(
                getDbClientBypassRLS().$transaction(async (tx) => {
                    console.log('email transaction');
                    const records = await tx.notification.findMany(
                        {
                            where: {
                                notificationMethodId: 1,
                                processedDate: null,
                            },
                            take: NOTIFICATIONS_POLL_BATCH_SIZE,
                        }
                    );

                    const trySendEmail = async (record: any) => {
                        try {
                            await markNotificationInitiated(
                                1,
                                tx,
                                record.id,
                                undefined,
                                5
                            );

                            const first = await tx.$queryRaw<SnakeCaseNotification>`SELECT *
                                                                                FROM notification
                                                                                WHERE id = ${record.id}
                                                                                limit 1` as any
                            console.log(`status: ${first?.notificationDeliveryStatusId}`)
                            if (first?.notificationDeliveryStatusId !== 5) {
                                throw new Error('Notification transaction invalid');
                            }

                            if (record.content && record.subject) {
                                console.log(`Sending email to ${record.to_address}`);
                            }
                        } catch (e) {
                            console.log(`Email Polling Error: ${e}`);
                            // TODO: handle failed messages / pick a status or retry
                            await markNotificationAsBounced(record.id, getDbClientBypassRLS());
                        }
                    };

                    await Promise.all(records.map(trySendEmail));
                    console.log('email transaction done');
                })
            )
        ),
        exponentialBackoffRetry()
    );

    return processRecords$.subscribe();
}

Raw queries are not attached to a model, so they are not affected by $allModels.$allOperations . You need to use top-level $allOperation:
prisma.$extends({
  query: {
    $allOperations() { ... }
  }
})

Thanks for the help, that worked for me! Any chance we could update the examples, so that someone else doesn't run into this issue as well?
https://github.com/prisma/prisma-client-extensions/blob/main/row-level-security/script.ts

@vrecan glad I could help!
You are right, we need to update the example - it was written before we had top-level $allOperations, i think. I've notified the team, we'll do it soon. Thanks for pointing that to us!
Since original problem is solved, i am going to close this now - feel free to comment if you have anything else to say on the topic.


Receive unexpected status code 400 from the API.

I am running this in Python and my first PDF worked fine with the API. The first PDF was 38 pages.
I am trying now with a PDF which is 98 pages and I receive the error: "Receive unexpected status code 400 from the API.".
Any idea why?

Are you able to share the pdf that caused this?

shell-sustainability-report-2022.pdf
It was the attached, thanks.

Thanks for the file - I'm able to reproduce the 400. The message from the API is that the file is encrypted. That's coming from this check, which needs to be tweaked. The file does have a password, but it's still readable.

Fix has been deployed!


Flutter version 1.25.0-8.0.pre

/C:/flutter/.pub-cache/hosted/pub.dartlang.org/velocity_x-1.0.1/lib/src/flutter/theme.dart:27:9: Error: No named parameter with the name 'isMaterialAppTheme'.
isMaterialAppTheme: isMaterialAppTheme

Hey @coimstark, thanks for raising the issue. The isMaterialAppTheme was removed 7 days ago from the framework. For more details https://github.com/flutter/flutter/pull/71861. If you build is failing, I would suggest to downgrade the branch until the parameter is removed from the VelocityX.

I downgraded to Flutter 1.22.5 and it worked again. In the meantime, I submitted a pull request.

ok thanks mate, awesome jobs


Keeps reporting "Websocket Error"

I tried to build a docker and run it on azure but it keeps reporting same error.
Same on my own machine.
Error log
2024-05-25 11:26:50,590 WARNING Websocket Error: scheme https is invalid
2024-05-25 11:26:50,591 INFO scheme https is invalid - reconnect

and keeps printing same thing.
Filesystem
root@3f6d8af29414:/app# ls
Dockerfile  LICENSE  README.md  config.json  config.sample.json  main.log  main.py  requirements.txt

Config
{
  "tg_bot_token": "1005******:AA********************************k",
  "channel_chat_id": -100198*******,
  "pm_chat_id": 143******,
  "mastodon_host": "mastodon.online",
  "mastodon_api_access_token": "fb8i98sRSyF********************************",
  "mastodon_app_name": "Mastodon2TG",
  "mastodon_username": "Misaka_0x447f",
  "scope": ["public", "unlisted"],
  "add_link_in_telegram": true,
  "add_link_in_mastodon": true
}

Sorry for late reply, I'm no longer using this project so I cannot ensure you my code is still working for you.
But I can offer you some idea to debug.
The error may occur at: https://github.com/littlebear0729/Mastodon2TG/blob/668d0f30c036336e5c27420a9a7f3547138d5a29/main.py#L63
You may try check the API document of your Mastodon instance and change the URL?


Use EntityCategory enum instead of strings

Using the entity category strings doesn't work anymore starting in 2022.4. Using the enums instead fixes #503 .
The enums have been accepted since 2021.12 (PR) which covers your supported versions so it is safe to make immediately, don't have to wait for 2022.4.

Are we sure it's backwards competitive and won't break the integration for those using latest stable HA image? 🤔 :)

Well, I don't think so. EntityCategory is added in home-assistant/core#60720, so all compilations before that one will not contain that class, resulting on an error, as far as I see it. Is there any way to check the version, and choose accordingly?
If not, maybe we should wait until the stable version is released, and then use the new method, which won't be compatible with HA <2022.4.
However, this should be tested.

After a bit of searching I've found this on StackOverflow, so maybe something like this can be used:
if version >= "2022.4":
    from homeassistant.const import ENTITY_CATEGORY_CONFIG
else:
    from homeassistant.helpers.entity import EntityCategory.CONFIG as ENTITY_CATEGORY_CONFIG

Which will allow to keep using
_attr_entity_category = ENTITY_CATEGORY_CONFIG

Not sure about this, it has to be tested and further investigation must be performed, just a suggestion that might work out until it's marked as not backward-compatible by HA.

@ArnyminerZ This repository lists version 2022.2 as it's minimum supported version of home assistant which is after the linked pr.
https://github.com/leikoilja/ha-google-home/blob/master/hacs.json#L4
That's why I said it would be fine to merge immediately. All versions of home assistant this integration supports are after that was merged into core and the enum shipped.
I'll fix the formatting and stuff though sorry about that. Was cranking through a couple of these based on users reporting the same issues in the beta channel of ha

Also just to get everyone on the same page, 2022.4 is the release when providing an entitycategory as a string instead of an enum goes from being deprecated to being unsupported. It's not the release when HA introduced the entitycategory enum, that was 2021.12 (the pr we both linked was committed into core on 12/1/2021).
So essentially this component no longer works at all starting in 2022.4 without this change. With this change it works fine for all releases of ha >= 2021.12 which covers all supported releases of this component according to it's HACS.json.

Okey @mdegat01, thanks for making it clear.  Then we can merge this without any issues. Agree @leikoilja ?

Since no further issues have been reported, merging this one 🚀

This issue does not appear to be resolved, check https://github.com/leikoilja/ha-google-home/issues/503 for update

Awesome, thank you, gents 💪 💥 🔥

Thanks very much for your efforts resolving this.