401 error of text analytics

@hemokwang commented on Wed Jan 30 2019
I wrote an app in the light of this tutorial, copied the source code and change the API key to the correct one.
But somehow it came with an error 401. I tried to change the endpoint, but it did not work.
The sentence with the problem is as below:
HttpWebResponse response = (HttpWebResponse)detectLanguageWebRequest.GetResponse();
The whole string:
// ***** DETECT LANGUAGE OF TEXT TO BE TRANSLATED
private string DetectLanguage(string text)
{
string uri = TEXT_ANALYTICS_API_ENDPOINT + "languages?numberOfLanguagesToDetect=1";
        // create request to Text Analytics API
        HttpWebRequest detectLanguageWebRequest = (HttpWebRequest)WebRequest.Create(uri);
        detectLanguageWebRequest.Headers.Add("Ocp-Apim-Subscription-Key", TEXT_ANALYTICS_API_SUBSCRIPTION_KEY);
        detectLanguageWebRequest.Method = "POST";

        // create and send body of request
        var serializer = new System.Web.Script.Serialization.JavaScriptSerializer();
        string jsonText = serializer.Serialize(text);

        string body = "{ \"documents\": [ { \"id\": \"0\", \"text\": " + jsonText + "} ] }";
        byte[] data = Encoding.UTF8.GetBytes(body);
        detectLanguageWebRequest.ContentLength = data.Length;

        using (var requestStream = detectLanguageWebRequest.GetRequestStream())
            requestStream.Write(data, 0, data.Length);

        HttpWebResponse response = (HttpWebResponse)detectLanguageWebRequest.GetResponse();

        // read and parse JSON response
        var responseStream = response.GetResponseStream();
        var jsonString = new StreamReader(responseStream, Encoding.GetEncoding("utf-8")).ReadToEnd();
        dynamic jsonResponse = serializer.DeserializeObject(jsonString);

        // fish out the detected language code
        var languageInfo = jsonResponse["documents"][0]["detectedLanguages"][0];
        if (languageInfo["score"] > (decimal)0.5)
            return languageInfo["iso6391Name"];
        else
            return "";
    }

文档详细信息
⚠ 请勿编辑此部分。 docs.microsoft.com ➟ GitHub 问题链接必须具有此部分。

ID: 2bfeb62f-8c6b-ed9b-3563-0514ccf05b8c
Version Independent ID: 64a9dfae-cade-7425-d858-eb2d33e75fba
Content: 教程：使用 C# 创建 WPF 应用程序 - 文本翻译 API - Azure Cognitive Services
Content Source: articles/cognitive-services/Translator/tutorial-wpf-translation-csharp.md
Service: cognitive-services
GitHub Login: @erhopf
Microsoft Alias: erhopf

@erhopf commented on Thu Jan 31 2019
@Jann-Skotdal - Is there someone on your team that can take a quick look at this user's issue?

@LisandroSu commented on Thu Jan 31 2019
Hello, @hemokwang

Thank you for your feedback.

@erhopf let us know if we should move the issue to a different repo or close it. Thanks!
Kind regards,
Microsoft DOCS International Team

@LisandroSu commented on Thu Jan 31 2019
Hello, @hemokwang

Thank you for your feedback.

@erhopf let us know if we should move the issue to a different repo or close it. Thanks!
Kind regards,
Microsoft DOCS International Team

@erhopf commented on Thu Jan 31 2019
@LisandroSu - I've asked the PM of the project to locate a resource to support this issue.

Hi team,
I moved this one here since the issue refers to the code example
best regards

@CeciAc Hi, we are fixing this document bug. Please just remove "?numberOfLanguagesToDetect=1" in the code sample. It should work. Thank you.

#please-close
@CeciAc + @YutongTie-MSFT - We've made some major adjustments to the tutorial.

The tutorial now uses the Detect resource from Translator Text instead of Text Analytics
The tutorial has been updated to meet MVC
The full source code is now available on GitHub


Redirect middleware seems to be incompatible with ASGI

I've been investigating ASGI.  I am observing this error from the redirect middleware.
Traceback (most recent call last):
   File "/var/lib/integration/django/core/handlers/exception.py", line 47, in inner
     response = get_response(request)
   File "/var/lib/integration/django/utils/deprecation.py", line 116, in __call__
     response = self.process_response(request, response)
   File "/var/lib/integration/wagtail/contrib/redirects/middleware.py", line 37, in process_response
     if response.status_code != 404:
 AttributeError: 'coroutine' object has no attribute 'status_code'

I've been looking through how these work and its got nothing to do with something you would fix


Bigtable: refactoring read_rows infrastructure

This work is on behalf of understanding the issues behind issue #5876 where InvalidChunk is thrown due to row keys being processed out of order.
About this PR

introducing PartialRowsData._process_chunk to break up logic in __iter__, and allow for additional testability.  __iter__ is responsible to get chunks, call _process_chunk, and manage last_scanned_row_key.
removed _copy_from_current
_state is now explicitly set
fixed a bug in PartialRowsData.counter which was counting chunks instead of rows.  This counter is used in reconstructing requests for "smart retries" that update the request on transient exceptions

@tseaver, I ran git push upstream minor_read_changes.  Fingers crossed...

@tseaver, I'll open a new PR from the local branch.


Got main wallet visual components completed

:tada: This PR is included in version 1.0.3 :tada:
Your semantic-release bot :package::rocket:


Shared-root example and IndexRoute

Help me, please. I try to change routing in shared-root example.I want App(SignedIn(Home)) components at "/" path. But it doesn't work! I can not understand why. Code:
<Router history={history}>
  <Route path="/" component={App}>
    <Route component={SignedOut}>
      <Route path="signin" component={SignIn} />
      <Route path="forgot-password" component={ForgotPassword} />
    </Route>
    <Route component={SignedIn}>
      <IndexRoute component={Home} />
    </Route>
  </Route>
</Router>

IndexRoute only applies to its immediate ancestor (it works by setting route.indexRoute), so you can't nest routes in that manner.
You can't directly re-use the shared root example in this manner.


[sdl2-mixer] Fixes for installing on Linux

I encountered a few issues while installing sdl2-mixer on Ubuntu (x64-linux triplet):

libFLAC was referred to as "flac" without capitalization in the CMakeLists file, resulting in the library not being found. I capitalized FLAC to fix this.
The CMakeLists file always built native_midi_win32.c which does not work on Linux. There does not seem to be an alternative for Linux in the SDL mixer repository so I simply skipped building the native MIDI functionality for non-Windows systems.
libmodplug fails to install on Ubuntu because the upstream CMakeLists file has a couple of issues. I have submitted an upstream PR to fix this yesterday and have a patch for it in my vcpkg fork, but didn't include that patch in this PR since I'm hoping it can be fixed upstream instead.

The end result of this PR is that you still can't install sdl2-mixer unless you also patch libmodplug, but it's one step closer. Let me know if you want the libmodplug patch included in this PR as well.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: JackBister sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the improvements!


Resolution Issue

When is the resolution issue going to be fixed? iIs been 3 years and Sharepoint list forms are still barely usable without experimenting for hours for a variety of screen sizes. How is anyone expected to replace Infopath forms with 1/4 of the small screen and brower ZOOM out is the only solution for Large screens while its leaves a huge amount of white space. Flexible screen size needs to be implemented in Sharepoint Lists forms already.

@ddeborja Sorry to hear about your frustration. However, we are a documentation team and are not in the loop with regard to such matters. If you have any questions or issues about the documentation, please provide a link to the page. Thanks


Plans for PHP 7.1+ support?

Hey,
Thanks for your hard work! Do you have plans on supporting php 7.1+ at some point? Would love to continue using this package, but 4.5 is broken now w/ 7.1.3 ;[
`
Your requirements could not be resolved to an installable set of packages.
Problem 1
- Installation request for shift31/laravel-elasticsearch 4.5.0 -> satisfiable by shift31/laravel-elasticsearch[v4.5.0].
- shift31/laravel-elasticsearch v4.5.0 requires php >=5.6 <=7.1 -> your PHP version (7.1.3) does not satisfy that requirement.
Installation failed, reverting ./composer.json to its original content.
`

Actually I didn't have time to test with PHP 7.1+ thats why I lock it. If you test it and works without limitation, you can make a pull request for changes.

Its already fixed on 4.5 branch.


Programmatic usage?

I need to use micro programatically, i.e. require('micro').
Is there any counterpart for this in micro-dev?

micro-dev is a development CLI. For programmatic use, take advantage of the micro package. ☺

@leo I think the doubt is more about using micro-dev when using micro programmatically. You can just use micro-dev index.js if it's not exporting a function.
That mean if you use micro programmatically you can't get the advantages of using micro-dev

The advantages of micro-dev can only be provided over the CLI. micro-dev is only meant to be used as a CLI in development. If you want its advantages, you need to use its CLI. There is no way we can provide this programmatically in a proper way.

@leo Is there a way to use micro-dev from CLI while using micro programmatically? Great tool / love using it on other projects.
Currently receive this error:
micro: The file "/.../index.js" does not export a function.
micro: https://err.sh/micro/no-export

I tried @sergiodxa 's recommendation of appending the filename (ie. micro-dev index.js) but I received the same export error.

I know this is kind of an off-topic question. But I'm wondering why you guys want to use micro programmatically in the first place? I'm just trying to think about this issue another way around.

@tungv Good question.
Why programmatically?
I'm attempting to homogenize builds for multiple projects as much as possible.
What about micro CLI makes it difficult to homogenize builds?

Inability to specify listen port (port exposure is relevant for Docker mapping)
Needing to call micro instead of traditional node index.js equivalent (requires changes in both NPM builds and Dockerfile)

These are by no means hard blockers (I caved yesterday, reverted to non-programmatic build and specialized build for micro), but it's definitely frustrating to not be able to use micro-dev programmatically.
If that's a requirement, why not just use something like Express?
A few soft reasons (in order of high-low weighted relevance)

Emotionally bought into the zeit ecosystem. I love now & hyper
Micro-dev is beautiful & simple. Default hot reload and pretty logs save time
I'm interested in exploring; I've used Express for every microservice prior to this
Express has always seemed verbose to me

Inability to specify listen port (port exposure is relevant for Docker mapping)

You can use micro -p $PORT to customize the port.

Needing to call micro instead of traditional node index.js equivalent (requires changes in both NPM builds and Dockerfile)

With npm scripts I don't see how this is an issue, you can also use npx micro to run micro outside an npm script

I'm not sure I understand the relevance of npx micro suggestion. As I pointed out in previous post, I've already configured a specialty build for Micro. The mere fact of needing to alter my standard build/start script in the first place is the problem.
Having said that, given that I do need to specialize the build for micro, the port tip is helpful. Thank you, @sergiodxa

Sounds like there's a lack of concrete use cases in this issue, so I'd like to throw in mine.
I currently use micro to serve a Typescript app. I have a src/index.ts file which gets compiled to a build/index.js file, which I point micro toward using the main key of my package.json. In order to run micro-dev in development, I have the following scripts set up:
{
  "main": "dist/index.js",
  "scripts": {
    "start": "micro",
    "build": "tsc",
    "dev": "ts-node --typeCheck node_modules/.bin/micro-dev src/index.ts",
    "postinstall": "npm run build"
  },
  "dependencies": {
    "micro": "^9.1.4"
  },
  "devDependencies": {
    "@types/micro": "^7.3.1",
    "@types/node": "9.4.7",
    "micro-dev": "^2.2.1",
    "ts-node": "^5.0.1",
    "typescript": "^2.7.2"
  }
}

I've also been testing a version that simply uses ts-node for production:
{
  "main": "index.ts",
  "scripts": {
    "start": "ts-node node_modules/.bin/micro",
    "dev": "ts-node --typeCheck node_modules/.bin/micro-dev"
  }
}

Being able to require a module before running the app might make this a bit cleaner, something like mocha's or nodemon's --require module flag: micro-dev --require ts-node/register. Or, being able to call micro-dev as a package could let someone to import the module in their typescript app and do a check like:
// script/dev.ts
import microDev from "micro-dev"
import app from "index.ts"

const server = microDev(app)

server.listen(3000)

I hope this is helpful — something to keep in mind as you build out the micro ecosystem!

My use-case is I'm using websockets as per the micro examples.

I can definitely echo the websockets usecase—need to pass the server instance to socket.io so need to use micro programmatically.

So I was making something similar work with tape, which doesn't have built in support for typescript/babel/etc. One of the workarounds to work with typescript (the others include --register flags like the ones mentioned above) is to call the tape binary from ts-node via ts-node node_modules/tape/bin/tape. This seems to work fine with micro-dev (as tested with sucrase):
sucrase-node node_modules/micro-dev/bin/micro-dev ./micro.ts -p 3001

Here I'm running a typescript file (types, imports, es7 syntax, etc) with sucrase, and passing args (port) to micro-dev, and it all seems to work as it should. Versions used:
micro-cors: "^0.1.1",
micro-dev: "^3.0.0",

Hopefully that helps someone! Can confirm it's soooooooo much faster and concise than a separate tsc -w script.


Update SDK to v12

There's been a major release on the directus sdk due to a small breaking change: https://github.com/directus/directus/releases/tag/v10.6.2.
Shouldn't need any changes in this module except from increasing the version. V12 also has some (type) optimisations and fixes.

Thanks for reporting this. This should be solved on v2.0.5


AxisInfoVisitor for LoadOp constancy calculation

If you call result = load(x, mask) where x and mask have some constancy properties, then you can infer some constancy properties for result.

@ptillet I had a lint failure, could you re-approve the tests?

unfortunately this has triggered a large performance regression (~30%) in some of our internal tests. I will dig a little to see what I can find.

Update: not this PR's fault. In fact, we had a bug on our side where we forgot a tl.multiple_of and the compiler just set it too optimistically before this PR! Thanks again :)


Feature / Add Juz / Siparah marks for Ayat.

Title of change

Add juz/siparah marks for ayaat. from request #652 and #371

Checklist

[x] Unit tests written
[x] Manually tested
[x] Prettier & ESLint were run
[x] New dependencies are included in package-lock.json

Screenshot

@ahmedre @mmahalwy @ahmedelgabri can you guys have a look at this, and review the UI.thanks guys.

Deployed to: http://staging.quran.com:33010

rebuild

@Dhul-Husni this is so close! :) excited for it to ship

Deployed to: http://staging.quran.com:33011

Deployed to: http://staging.quran.com:33013

@mmahalwy  will fix the tooltip bug in the next PR and make necessary changes inshaAllah

rebuild

Deployed to: http://staging.quran.com:33021


Customize guacamole

We should customize Guacamole:

Add custom text messages
Serve guacamole at /sessions path
Write custom guacamole-ext: https://guacamole.apache.org/doc/gug/guacamole-ext.html
Add custom authentication provider

"Add custom text messages" done in https://github.com/DSD-DBS/capella-collab-manager/pull/891
"Serve guacamole at /sessions path " tracked in https://github.com/DSD-DBS/capella-collab-manager/issues/942
"Write custom guacamole-ext" done in https://github.com/DSD-DBS/capella-collab-manager/pull/891
"Add custom authentication provider" tracked in https://github.com/DSD-DBS/capella-collab-manager/issues/943


Refactor internal query graph representation

Previously, API response records were refactored to streamline related code interactions and to simplify testing.
It appears that our internal representation of the query graph could use a similar treatment. Currently, the query graph (as supplied by the query) is validated and transformed into first qNodes and qEdges, and then qXEdges (query execution edges) are created as a "wrapper" of sorts around qEdges. Given the actual lifetime of a qEdge and limited use outside of being part of a qXEdge, I think it makes sense to roll the two into simply just a qEdge.
Merging qEdge and qXEdge would simplify the structure and interface. Beyond this, however, refactoring the class along the lines of Record in order to allow freezing/unfreezing of an instance, where both the frozen and unfrozen data (and therefor class constructor) have the same structure and property names, would also dramatically simplify both testing and further development.

I could work on this

Deployed to Prod.


[Metal] Dispatch numerically stable tanh for metal

Prior to this PR, tanh(x)returns NaN on metal when x > 45.0.
Metal's built-in tanh is implemented as (t - 1.0) / (t + 1.0), where t = exp(2.0 * x). Hence for large x, t becomes inf, causing tanh(x) to be NaN.
A numerically stable tanh is implemented for llvm, this PR lifts it to src/target/intrin_rule.cc and apply the same rule for metal as well.

cc @MasterJH5574 @tqchen

Observed the same issue on webgpu as well; updated to fix.


Always keep top N items option and prevent animated scrolling

Is there an option to always keep top page of items, as users can always hit the Home key? At the moment whole list executes an animated scroll. If list is supposed to be loaded asynchronously, this would make many unnecessary HTTP requests to the server while scrolling would have to be paused. Having top page always loaded could simply jump to the top of the list (and maybe just animate this portion as it should hold more than a visible page of items). but it would definitely prevent HTTP requests.
How should this be solved?

Hi Robert,
I am sorry but I didn't get your idea.
Clusterize doesn't support http requests, asynchronous loading logic and etc. It's intended to render rows. Think of it as 'view'.
Main idea is to create one array of rows (empty of filled with some initial data, no matter).
Init clusterize with this array. Make asynchronous request to load more rows, if needed.
Add loaded rows to our array and call clusterize_instance.update(our_array);

I didn't imply that Clusterize does any async magic. I was thinking (for whatever reason) that clusterize only works with a subset of the whole data set. But that's not the case. Clusterize uses the whole set and only displays a specific cluster of rows at a time.
My original question was if there was any possibility to keep the first (and last) cluster always part of the DOM in case user hits the Home or Enter key? Because if we look at the possible scenarios of a long list user may either:

scroll up (or down) whicm means that sooner or later clusterize will have to display a different cluster which is sibling to current one - in case user is scrolling down it will be next cluster, if scrolling up it will be previous
hitting PgUp/PhDown/Space which is similar to first
hitting Home/End which forces clusterize to display first/last cluster and recalculate dummy empty containers

Now if clusterize would always keep in DOM first, last and current cluster it wouldn't have to do any specific processing when third event would happen, which would result in an instant view it first/last cluster items.
But my question was also related to animated scrolling. Because if user hits the Home button and browser has some sort of animated scrolling feature, then browser will execute a series of onscroll events meaning there will be much more processing for each step in between current cluster and first one.
Did I explain myself better this time?

Thanks, now it's much clearer.
Regarding Home buttons question - It is slightly artificial problem. I don't really think that additional top/bottom clusters worth the effort. Cluster chunks are replaces so fast that user will not even notice this. Replacing of first or last cluster it's an ordinary action for this plugin. Moreover, it will pollute the DOM with redundant markup.
Animated scrolling - Look, if user has enabled smooth scrolling in his browser - this is the usual behaviour for him and I do not want to break this behaviour. I understand that because of the smooth scrolling may be made more DOM inserts (in series), than if the transition was instant. But the clusters are changing very quickly, so I'm willing to sacrifice performance to keep customary UI behaviour.


does cardano support sign transaction with secp256k1?

is there any secp256k1 transaction on mainnet ?

I'm not aware of any. I think it's still something that was built for some future plan, such as Midnight integration.


fix: Require tokio/rt feature for sync scenarios

This should fix #191 but I'd feel better if there is a test/configuration where the CI detects such bugs.
@uklotzde do you have an idea how to provoke the issue?

Just fixing the issue without a regression test that triggers the dependency/feature error is brittle.

cargo check --no-default-features --features rtu-sync fails.
Ideally, all or at least selected feature combinations should be checked by the CI builds. Using all possible combinations does not make much sense, but checking single features in isolation like demonstrated here would help.

Another argument to avoid overusing features for crate extensions. Using separate sync and server crates would be more appropriate.
The sync stuff doesn't fit in here at all. Offering a sync-over-async bridge layer is pointless for an async-focused crate IMHO.

The sync stuff doesn't fit in here at all. Offering a sync-over-async bridge layer is pointless for an async-focused crate IMHO.

I agree!

After force pushing GitHub noticed the new workflow action.


Access to general postMessages

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce
<!DOCTYPE html>
<html>
<head>
    <title>Beans SDP Deeplink</title>
</head>
<body>
    <button onclick="close()">Close</button>
    <script>
      function close() {
        postMessage('close');
      }
    </script>
</body>
</html>

Expected results
I'm expecting to be able to listen to the general post message. I already tried listening to the window javascript channel but that does not work.
If I add Print.postMessage('close'); I can listen to it via a Print javascript channel. And I think we should also be able to listen to these general postMessages.
Actual results
Not being able to receive the close post message.
Code sample
Code sample
    _webViewController = WebViewController()
      ..setJavaScriptMode(JavaScriptMode.unrestricted)
      ..addJavaScriptChannel(
        'window', // This is the channel name you will use in your JS code
        onMessageReceived: (message) {
          log(
            message.message,
            name: 'WebViewController.windowJavaScriptChannel.onMessageReceived',
          );
          if (message.message != 'close') {
            return;
          }
          // Handle your logic here when the 'close' message is received
          // Navigator.of(context).pop();
        },
      )
      ..addJavaScriptChannel(
        'Print', // This is the channel name you will use in your JS code
        onMessageReceived: (message) {
          log(
            message.message,
            name: 'WebViewController.printJavaScriptChannel.onMessageReceived',
          );
          if (message.message != 'close') {
            return;
          }
          // Handle your logic here when the 'close' message is received
          // Navigator.of(context).pop();
        },
      );

Screenshots or Video

Screenshots / Video demonstration
[Upload media here]

Logs
Logs
[Paste your logs here]

Flutter Doctor output
Doctor output
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.10.6, on macOS 13.3.1 22E772610a darwin-arm64, locale en-GB)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
[✓] Android Studio (version 2022.1)
[✓] Android Studio (version 2022.1)
[✓] VS Code (version 1.81.1)
[✓] Connected device (2 available)
[✓] Network resources

• No issues found!

Found a work around myself.
    _webViewController = WebViewController()
      ..setJavaScriptMode(JavaScriptMode.unrestricted)
      ..setNavigationDelegate(NavigationDelegate(
        onPageFinished: (_) async {
          _webViewLoaded = true;
          _updatePages();
          setState(() {});
          await _webViewController.runJavaScript(
            'window.addEventListener("message", (event) => YourName.postMessage(event), false);',
          );
        },
      ))
      ..addJavaScriptChannel(
        'YourName',
        onMessageReceived: (message) {
          if (message.message != 'close') {
            return;
          }
          // Handle your logic here when the 'close' message is received
        },
      );


Dynamic Allocation: Too many executors than desired?

On running a Spark Pi with dynamic allocation enabled, I see the following somewhat consistently.. When I run the job with these arguments
bin/spark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi --master k8s://https://<IP_ADDRESS>:6443 --kubernetes-namespace default --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.kubernetes.shuffle.namespace=default --conf spark.kubernetes.shuffle.labels=app=spark-shuffle-service,spark-version=2.1.0 --conf spark.app.name=spark-pi --conf spark.kubernetes.driver.docker.image=docker:5000/spark-driver:varun_latest31 --conf spark.kubernetes.executor.docker.image=docker:5000/spark-executor:varun_latest31 --conf spark.kubernetes.initcontainer.docker.image=docker:5000/spark-init-container:varun_latest31 local:///opt/spark/examples/jars/spark-examples_2.11-2.1.0-k8s-0.2.0-SNAPSHOT.jar 61000

The number of pods launched seem to be a much higher number than what is reported in the logs..
..
spark-pi-1498671730530-driver                    1/1       Running   0          2m
spark-pi-1498671730530-exec-1                    1/1       Running   0          2m
spark-pi-1498671730530-exec-10                   1/1       Running   0          1m
spark-pi-1498671730530-exec-11                   1/1       Running   0          1m
spark-pi-1498671730530-exec-12                   1/1       Running   0          1m
spark-pi-1498671730530-exec-13                   1/1       Running   0          1m
spark-pi-1498671730530-exec-14                   1/1       Running   0          1m
spark-pi-1498671730530-exec-15                   1/1       Running   0          1m
spark-pi-1498671730530-exec-16                   1/1       Running   0          1m
spark-pi-1498671730530-exec-17                   1/1       Running   0          59s
spark-pi-1498671730530-exec-18                   1/1       Running   0          59s
spark-pi-1498671730530-exec-19                   1/1       Running   0          59s
spark-pi-1498671730530-exec-2                    1/1       Running   0          2m
spark-pi-1498671730530-exec-20                   1/1       Running   0          59s
spark-pi-1498671730530-exec-21                   1/1       Running   0          59s
spark-pi-1498671730530-exec-22                   0/1       Pending   0          38s
spark-pi-1498671730530-exec-23                   0/1       Pending   0          38s
spark-pi-1498671730530-exec-24                   0/1       Pending   0          38s
spark-pi-1498671730530-exec-25                   0/1       Pending   0          38s
spark-pi-1498671730530-exec-26                   0/1       Pending   0          38s
spark-pi-1498671730530-exec-3                    1/1       Running   0          2m
spark-pi-1498671730530-exec-4                    1/1       Running   0          2m
spark-pi-1498671730530-exec-5                    1/1       Running   0          2m
spark-pi-1498671730530-exec-6                    1/1       Running   0          2m
spark-pi-1498671730530-exec-7                    1/1       Running   0          1m
spark-pi-1498671730530-exec-8                    1/1       Running   0          1m
spark-pi-1498671730530-exec-9                    1/1       Running   0          1m

..
In the driver logs I see the following, I see only upto 5 launched
...
└─[0] <git:(master 29acb7e✱✈) > kubectl logs spark-pi-1498671730530-driver | grep -i backend
2017-06-28 17:42:30 INFO  KubernetesClusterSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
2017-06-28 17:42:35 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 1
2017-06-28 17:42:46 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(null) (<IP_ADDRESS>:51308) with ID 1
2017-06-28 17:42:47 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 2
2017-06-28 17:42:47 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 3
2017-06-28 17:42:47 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 4
2017-06-28 17:42:47 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 5
...

I tried using the published images from: https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html
My logs say:
~/g/src k logs spark-pi-1498677689366-driver | grep 'Requesting a new executor, total executors'
2017-06-28 19:21:52 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 1
2017-06-28 19:22:02 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 2
2017-06-28 19:22:02 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 3
2017-06-28 19:22:02 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 4
2017-06-28 19:22:02 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 5
2017-06-28 19:22:02 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 6
2017-06-28 19:22:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 7
2017-06-28 19:22:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 8
2017-06-28 19:22:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 9
2017-06-28 19:22:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 10
2017-06-28 19:22:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 11
2017-06-28 19:22:20 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 12
2017-06-28 19:22:20 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 13
2017-06-28 19:22:20 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 14
2017-06-28 19:22:20 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 15
2017-06-28 19:22:21 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 16
2017-06-28 19:22:29 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 17
2017-06-28 19:22:29 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 18
2017-06-28 19:22:29 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 19
2017-06-28 19:22:29 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 20
2017-06-28 19:22:29 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 21
2017-06-28 19:22:37 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 22
2017-06-28 19:22:37 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 23
2017-06-28 19:22:37 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 24
2017-06-28 19:22:37 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 25
2017-06-28 19:22:37 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 26
2017-06-28 19:22:45 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 27
2017-06-28 19:22:45 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 28
2017-06-28 19:22:45 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 29
2017-06-28 19:22:45 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 30
2017-06-28 19:22:45 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 31
2017-06-28 19:22:54 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 32
2017-06-28 19:22:54 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 33
2017-06-28 19:22:54 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 34
2017-06-28 19:22:54 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 35
2017-06-28 19:22:54 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 36
2017-06-28 19:23:04 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 37
2017-06-28 19:23:04 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 38
2017-06-28 19:23:04 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 39
2017-06-28 19:23:04 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 40
2017-06-28 19:23:04 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 41
2017-06-28 19:23:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 42
2017-06-28 19:23:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 43
2017-06-28 19:23:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 44
2017-06-28 19:23:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 45
2017-06-28 19:23:13 INFO  KubernetesClusterSchedulerBackend:54 - Requesting a new executor, total executors is now 46

Can you try with the published images? Are your images built from HEAD?

These are built from the HEAD...I will check once, if there is some user/operator error on my part and report..

@varunkatta can you try with static allocation? This might be a bug with our images in general - this wouldn't look specific to dynamic allocation, we build the pod's arguments identically.

And also use kubectl describe pod to get the details of the executor pods. I double checked our Dockerfiles on HEAD and they seem sane. Here's the command:
CMD SPARK_CLASSPATH="${SPARK_HOME}/jars/*" && \
    if ! [ -z ${SPARK_MOUNTED_CLASSPATH}+x} ]; then SPARK_CLASSPATH="$SPARK_MOUNTED_CLASSPATH:$SPARK_CLASSPATH"; fi && \
    if ! [ -z ${SPARK_EXECUTOR_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_EXECUTOR_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi && \
    if ! [ -z ${SPARK_MOUNTED_FILES_DIR} ]; then cp -R "$SPARK_MOUNTED_FILES_DIR/." .; fi && \
    exec /sbin/tini -- ${JAVA_HOME}/bin/java -Dspark.executor.port=$SPARK_EXECUTOR_PORT -Xms$SPARK_EXECUTOR_MEMORY -Xmx$SPARK_EXECUTOR_MEMORY -cp $SPARK_CLASSPATH org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url $SPARK_DRIVER_URL --executor-id $SPARK_EXECUTOR_ID --cores $SPARK_EXECUTOR_CORES --app-id $SPARK_APPLICATION_ID --hostname $SPARK_EXECUTOR_POD_IP

So we're probably seeing something where SPARK_EXECUTOR_MEMORY isn't being set properly, but looking at KubernetesClusterSchedulerBackend I can't quite see how this would happen. Would be good to inspect the pod to get more insight.

on describe pod of executor, this is what I see. I will try digging more information, if possible.
.....
Volume Mounts:
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-2jnd3 (ro)
Environment Variables:
SPARK_EXECUTOR_PORT:	10000
SPARK_DRIVER_URL:		spark://CoarseGrainedScheduler@<IP_ADDRESS>:46339
SPARK_EXECUTOR_CORES:	1
SPARK_EXECUTOR_MEMORY:	2g
SPARK_APPLICATION_ID:	spark-5fefae5c99a04ca4b40249e078a653a1
SPARK_EXECUTOR_ID:	1
SPARK_MOUNTED_CLASSPATH:	/var/spark-data/spark-jars/*
SPARK_EXECUTOR_POD_IP:	 (v1:status.podIP)

K8s versions:
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6+e569a27", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"not a git tree", BuildDate:"2016-11-12T09:29:14Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:34:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

This issue is stale now.


Alternate approach to updating registry without adding @extra_features to Model

Closes: #826
What's Changed

Rather than requiring Models to have the decorator @extras_feature added in order to be added to the registry model_features, this prototype concentrates on updating registry without having to add the @extras_feature to model cause it would be smart enough to tell which model needs to be added to a particular feature.
This can be archived by adding some relevant data to lookup_confs
{
    "model": CustomField,
    "lookup_by": "field",
    "field_names": ["_custom_field_data"],
}

model: model feature to be included in the registry model_features
lookup_by: This describes how to find the model that satisfies the criteria; options include field or related model

field: During the process of iterating through all models, a particular model field name would be looked up, and if found, this is one of the models that would be returned.
related_model: While iterating through all models and their fields, we look for instances where a field has a connection to the related_model that has been specified.

field_names:  The field_names are the field names that will be searched for across all models IF lookup_by is field, and models that have all fields specified will be chosen.
related_model: If lookup_type is related_model, we would search for this related model in each model field, and if a model field has a relationship with it or has its related_model set to the specified, it would be chosen.
extras: Just in case we needed additional verifications to focus the search.

I made it my goal to update the registry because I thought that running this code at startup and storing the results there would be the best course of action. Additionally, this logic could be expanded to take into account almost all of the items on extra_features.
TODO

[ ] Explanation of Change(s)
[ ] Added change log fragment(s) (for more information see the documentation)
[ ] Attached Screenshots, Payload Example
[ ] Unit, Integration Tests
[ ] Documentation Updates (when adding/changing features)
[ ] Example Plugin Updates (when adding/changing features)
[ ] Outline Remaining Work, Constraints from Design

This PR/Issue does not include refactoring / cleaning up all models with @extras_features; however, a new issue could be opened to address this.

Add roles, statuses, tags, and changelogs to the 'update registry with new apps' and 'lookup confs.
Remove 'RoleModelsQuery,' 'TaggableClassesQuery,' and 'ChangeLoggedModelsQuery,' as all of these classes' functionality can now be obtained from 'FeatureQuery.'
Remove "custom fields," "relationships," and "statuses" from all '@extras_features'.

Still like this a lot. Needs documentation, model updates to remove the custom_fields and relationships extras_features, and verification that everything still works afterwards.

I manually checked that it functions and produces the same results as @extras_feature, but I had intended to refactor in a separate issue (my comment above). Do you believe I should just remove only relationship and custom_field from all @extras_fields in this PR and not create a new issue to address all?

Still like this a lot. Needs documentation, model updates to remove the custom_fields and relationships extras_features, and verification that everything still works afterwards.

I manually checked that it functions and produces the same results as @extras_feature, but I had intended to refactor in a separate issue (my comment above). Do you believe I should just remove only relationship and custom_field from all @extras_fields in this PR and not create a new issue to address all?

IMHO, removing just relationships and custom_fields from @extras_features usage is desirable for this PR, with a new issue to address the remaining features.

Needs documentation

Any ideas on where this documentation should be added.


Raspberry Pi5

I'm not sure if I can be of any use to you but I have a Raspberry Pi5 already in my hands as I was part of the test group for it. If you'd like any testing done on it i'm happy to help. I did try to get things running today from your github repo but i'm still a very newbie rust user so didn't get past the
'''
Error: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })
'''
running the examples which i believe is an error on how i built the library rather than the library itself

Thanks for the offer! I've added all the basics, but there is only so much I can do without a physical device, or access to technical documentation.
I'm not that worried about SPI, I2C, UART and PWM support as these are handled by linux devices. GPIO however, reads/writes to the registers directly through /dev/gpiomem, and with the addition of the new RP1, I don't know if anything has changed there.
The error you received likely means one of the relevant devices in the /dev/ directory isn't available, and either hasn't been configured in /boot/config.txt, or isn't available on the Pi 5. Which one is missing depends on the peripheral the example(s) were trying to access. I did just add a commit to allow the library to detect the pinctrl-bcm2712 driver, so that could've been the reason for that error as well.
Since you mentioned you're relatively new to Rust, I suggest running the examples one by one to figure out which ones work, and which ones return an error, so we can narrow down what's causing the problem. For instance, to test the GPIO status example, run cargo run --example gpio_status.

i'm not sure if /dev/gpiomem exists anymore as the new bookworm update to pios (public release due mid-october) has fully moved to libgpio

and this is what i get running gpio_status

update: just checked and yes no /dev/gpiomem BUT there is a /dev/rp1-gpiomem which i'm guessing is what we (ok you :) ) need

Hmm, I wonder if running bookworm on a Pi 4 or older will still get you a /dev/gpiomem instead of a /dev/rp1-gpiomem. The name change does make me wonder if the registers have changed significantly. Hopefully this'll become clear before the Pi 5 is released. I definitely don't want to just change the device name for the Pi 5 and hope for the best.
I think at this point the options are to wait for documentation and access to a Pi 5 at release, or consider libgpiod, assuming performance doesn't lag too far behind directly accessing the registers.

and this is what i get running gpio_status

Did you update your local rppal clone with the latest commit? Just curious if the file not found error was caused by the lack of pinctrl-bcm2712 detection or something else.

just did a git pull and get the same
ukscone@glooston:~/rppal $ git pull remote: Enumerating objects: 15, done. remote: Counting objects: 100% (15/15), done. remote: Compressing objects: 100% (2/2), done. remote: Total 10 (delta 8), reused 10 (delta 8), pack-reused 0 Unpacking objects: 100% (10/10), 2.18 KiB | 744.00 KiB/s, done. From https://github.com/golemparts/rppal    c475002..3a199de  master     -> origin/master Updating c475002..3a199de Fast-forward  src/gpio/ioctl.rs | 2 ++  src/gpio/mem.rs   | 6 +++---  2 files changed, 5 insertions(+), 3 deletions(-) ukscone@glooston:~/rppal $ cargo run --example gpio_status    Compiling rppal v0.14.1 (/home/ukscone/rppal)     Finished dev [unoptimized + debuginfo] target(s) in 1.25s      Running target/debug/examples/gpio_status Error: PermissionDenied("/dev/mem") ukscone@glooston:~/rppal $ sudo target/debug/examples/gpio_status Error: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" }) ukscone@glooston:~/rppal $

Thanks for all the testing so far. I think it's time to dive into https://github.com/raspberrypi/linux/pull/5618 and see what exactly has changed. Hopefully that'll give me a better picture of what I can do at this point.
My best guess is that I'm trying to look for the wrong driver name, but speculation will only get me so far 😁. I'll give that PR a read and will ping you when there's more to test. I appreciate the help!

@ukscone would you mind giving https://github.com/golemparts/gpiochip a run on your Pi 5 and paste the results? It should give you a list of all available gpiochip driver names and the number of lines, similar to this
meer@flex:~/gpiochip $ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `target/debug/gpiochip`
ChipInfo { name: "gpiochip0", label: "pinctrl-bcm2711", lines: 58 }
ChipInfo { name: "gpiochip1", label: "raspberrypi-exp-gpio", lines: 8 }

We should be seeing pinctrl-bcm2712 and/or pinctrl-rp1, but I'd like to confirm.

sorry for the delay. I overslept
ukscone@glooston:~ $ git clone https://github.com/golemparts/gpiochip.git Cloning into 'gpiochip'... remote: Enumerating objects: 11, done. remote: Counting objects: 100% (11/11), done. remote: Compressing objects: 100% (10/10), done. remote: Total 11 (delta 0), reused 11 (delta 0), pack-reused 0 Receiving objects: 100% (11/11), 7.41 KiB | 7.41 MiB/s, done. ukscone@glooston:~ $ cd gpiochip ukscone@glooston:~/gpiochip $ cargo run    Compiling libc v0.2.148    Compiling gpiochip v0.1.0 (/home/ukscone/gpiochip)     Finished dev [unoptimized + debuginfo] target(s) in 2.86s      Running target/debug/gpiochipChipInfo { name: "gpiochip0", label: "gpio-brcmstb@107d508500", lines: 32 } ChipInfo { name: "gpiochip1", label: "gpio-brcmstb@107d508520", lines: 4 } ChipInfo { name: "gpiochip2", label: "gpio-brcmstb@107d517c00", lines: 17 } ChipInfo { name: "gpiochip3", label: "gpio-brcmstb@107d517c20", lines: 6 } ChipInfo { name: "gpiochip4", label: "pinctrl-rp1", lines: 54 } ChipInfo { name: "gpiochip5", label: "fsm_leds", lines: 2 } ukscone@glooston:~/gpiochip $

sorry that's a mess recopying
ukscone@glooston:~ $ git clone https://github.com/golemparts/gpiochip.git
Cloning into 'gpiochip'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 11 (delta 0), reused 11 (delta 0), pack-reused 0
Receiving objects: 100% (11/11), 7.41 KiB | 7.41 MiB/s, done.
ukscone@glooston:~ $ cd gpiochip
ukscone@glooston:~/gpiochip $ cargo run
Compiling libc v0.2.148
Compiling gpiochip v0.1.0 (/home/ukscone/gpiochip)
Finished dev [unoptimized + debuginfo] target(s) in 2.86s
Running target/debug/gpiochip
ChipInfo { name: "gpiochip0", label: "gpio-brcmstb@107d508500", lines: 32 }
ChipInfo { name: "gpiochip1", label: "gpio-brcmstb@107d508520", lines: 4 }
ChipInfo { name: "gpiochip2", label: "gpio-brcmstb@107d517c00", lines: 17 }
ChipInfo { name: "gpiochip3", label: "gpio-brcmstb@107d517c20", lines: 6 }
ChipInfo { name: "gpiochip4", label: "pinctrl-rp1", lines: 54 }
ChipInfo { name: "gpiochip5", label: "fsm_leds", lines: 2 }
ukscone@glooston:~/gpiochip $

the above matches exactly what we get from libgpio's gpiodetect
ukscone@glooston:~/gpiochip $ gpiodetect
gpiochip0 [gpio-brcmstb@107d508500] (32 lines)
gpiochip1 [gpio-brcmstb@107d508520] (4 lines)
gpiochip2 [gpio-brcmstb@107d517c00] (17 lines)
gpiochip3 [gpio-brcmstb@107d517c20] (6 lines)
gpiochip4 [pinctrl-rp1] (54 lines)
gpiochip5 [fsm_leds] (2 lines)
ukscone@glooston:~/gpiochip $

Awesome. I'll have it grab pinctrl-rp1 when a Raspberry Pi 5 is detected. That should fix the File not found error message.
The next hurdle is /dev/rp1-gpiomem, although it would be nice to switch over to gpiochip completely (basically the same thing libgpiod does). I do recall looking into that before and finding some issues at the time (see https://github.com/golemparts/rppal/issues/23#issuecomment-468914674 if you're interested). However, that was 4 years ago and things may have improved. I'll have a look and see if it's ready for use now, as that would be the easiest way to support GPIO going forward.

Things seem to have improved a lot with the v2 API. Let's see if moving away from directly accessing gpiomem is an option now.

that's great. hopefully you'll be able to get everything Pi5 compatible by the time people start getting them in their paws.
i'v been slowly porting all my stuff that used python* & gpiozero to rust & rust+tauri and rppal is part of my path to a pythonfree life
*the python/pip/venv stuff on bookworm makes it a bit harder to quickly knock up a script

Great to hear rppal has been useful to you! I've only dabbled in python - my background lies in C++. When I started this project ~7 years ago (it actually started out as part of the blinkt crate) I was still relatively new to Rust, and I'm still as excited about the language as when I first heard about it.
I think we'll be okay with Pi5 support for the launch date with your help. I've reached out to Raspberry Pi to see if they have any leftover review units available (and for future device releases as well), but I haven't heard back from them yet. So for now we'll have to rely on your testing, which has been invaluable so far.

did a git pull just now just for the hell of it and tried gpio_status and that's looking much better :)
ukscone@glooston:~/rppal/target/debug/examples $ ./gpio_status
Error: PermissionDenied("/dev/mem")
ukscone@glooston:~/rppal/target/debug/examples $ sudo !!
sudo ./gpio_status
+------+-------+---+---------+---+-------+------+
| GPIO | Mode  | L |   Pin   | L | Mode  | GPIO |
+------+-------+---+----+----+---+-------+------+
|      | 3.3 V |   |  1 |  2 |   | 5 V   |      |
|    2 | IN    | 0 |  3 |  4 |   | 5 V   |      |
|    3 | IN    | 0 |  5 |  6 |   | GND   |      |
|    4 | IN    | 0 |  7 |  8 | 0 | IN    |   14 |
|      | GND   |   |  9 | 10 | 0 | IN    |   15 |
|   17 | IN    | 0 | 11 | 12 | 0 | IN    |   18 |
|   27 | IN    | 0 | 13 | 14 |   | GND   |      |
|   22 | IN    | 0 | 15 | 16 | 0 | IN    |   23 |
|      | 3.3 V |   | 17 | 18 | 0 | IN    |   24 |
|   10 | IN    | 0 | 19 | 20 |   | GND   |      |
|    9 | IN    | 0 | 21 | 22 | 0 | IN    |   25 |
|   11 | IN    | 0 | 23 | 24 | 0 | IN    |    8 |
|      | GND   |   | 25 | 26 | 0 | IN    |    7 |
|    0 | IN    | 0 | 27 | 28 | 0 | IN    |    1 |
|    5 | IN    | 0 | 29 | 30 |   | GND   |      |
|    6 | IN    | 0 | 31 | 32 | 0 | IN    |   12 |
|   13 | IN    | 0 | 33 | 34 |   | GND   |      |
|   19 | IN    | 0 | 35 | 36 | 0 | IN    |   16 |
|   26 | IN    | 0 | 37 | 38 | 0 | IN    |   20 |
|      | GND   |   | 39 | 40 | 0 | IN    |   21 |
+------+-------+---+----+----+---+-------+------+
ukscone@glooston:~/rppal/target/debug/examples $

Solved the file not found error at least 😁 . I wouldn't trust the GPIO data at the moment though, since the /dev/mem method still tries to access the registers directly using the old layout, which may have changed completely because of the RP1.

After checking out the gpiochip v2 API interface today, I've hit the same roadblocks as with v1, with the exception of pull-ups/pull-downs, which are supported now.
As far as I can tell, gpiochip isn't able to tell when a GPIO line has been set to any of the Alt functions. It shows those as Input instead. While that isn't a major issue per se, since a GPIO line configured for an alternative function technically isn't a GPIO line anymore, the problem lies with tools such as the gpio_status example, that need to be able to show the mode and pin level for each GPIO line.
When a GPIO_V2_GET_LINE_IOCTL and a subsequent GPIO_V2_LINE_GET_VALUES_IOCTL ioctl request is sent to get the pin level of such a line, gpiochip ends up changing the mode from whatever Alt function it used to have to Input. That means gpio_status simply wouldn't work anymore, because it would overwrite any configured Alt modes to Input simply by reading their levels. It's possible I missed something, so I'll be going over this in more detail tomorrow, but considering v1 had the same issue, I think this might not be solvable. I grabbed the gpiod commandline tools to compare, but they gave me the same results.
Additionally, it looks like gpiochip doesn't see pin mode changes that were made directly through the registers until a GPIO line has been accessed at least once, requiring 2 reads. I'm not sure if this issue is worth a workaround, since it only seems to affect changes that were made through /dev/gpiomem, but it's definitely a concern to think about.
It basically boils down to gpiochip only offering a subset of the register access RPPAL currently uses.
Assuming there are no easy fixes for the above, that leaves me with a few options.

Move to gpiochip, remove any support for Alt functions, and require a GPIO mode to be set before its level can be read, which means getting rid of Pin and only having InputPin, OutputPin and IoPin. Tools like gpio_status won't work anymore with RPPAL.
Wait for technical documentation and add support for /dev/rp1-gpiomem. All functionality will remain the same, but the library probably won't be Pi 5 compatible on the launch date.
Stick with gpiochip for now, remove the above-mentioned functionality, then switch back to /dev/gpiomem and /dev/rp1-gpiomem when technical documentation has been made available. I'm not a fan of removing and adding back functionality people might be relying on, but at least everything would work eventually, and RPPAL will be ready for the Pi 5. Users with older models can stick to the current release of RPPAL.

If I had a Pi 5 here I'd probably grab the pinctrl-rp1 source and work on direct register access support based on that, but it's simply too time consuming to test with our current situation.
Open to any other suggestions if you have some.

Just mulling over a 4th option, since I might be able to determine a GPIO line isn't configured as a GPIO anymore based on the name instead. For instance, GPIO 9, 10, 11 are set to Alt0 for SPI. gpiochip doesn't show the alt mode, but it does mention the appropriate functionality in the name field (SPI_MISO, SPI_MOSI, SPI_SCLK) as opposed to a GPIO line configured for GPIO having a GPIOxx name.
I could add some restrictions to what a Pin can read based on the name provided by gpiochip.

Looks like the names are hardcoded. After disabling SPI and rebooting, the names remain the same even though they're not configured for SPI anymore.

I'm not sure if it's any use but gpiozero v2 is compatible with the Pi5 so it might be worth looking in there to see if there is any info that'll give clues.
https://github.com/gpiozero/gpiozero/tree/pi5

Thanks for the link. While I'm not experienced with python, I gave it a quick look. It seems to be using lgpio under the hood to access gpiochip, which, based on the docs, has the same limitations as mentioned above - it only recognizes and works with input and output pins.

Hm... Option 5, switch to gpiochip, leave Pin as-is except for the ability to read the current pin level. That way, anyone reading the level will have to make a conscious decision to set a specific mode for the pin through into_input / into_output / into_io. No pin levels anymore for gpio_status, but it can still show input/output as reported by gpiochip, and the overall structure of the Gpio module remains mostly the same.

Btw, I've updated https://github.com/golemparts/gpiochip with the code I've been testing. If you look at src/main.rs, you'll see a commented out section that'll retrieve the GPIO pin levels. Uncommenting it causes any alt-configured pins to be changed to an inputs.

oh btw not sure if you realized that once pios (bookworm) is released libgpio and thus gpiochip (&gpiomem probably) will be the way for the gpio on the pi (all models) any other methods will go away. i think you are ok with the pi1 to pi4 but just wanted to make sure you were aware :)

Btw, I've updated https://github.com/golemparts/gpiochip with the code I've been testing. If you look at https://github.com/golemparts/gpiochip/blob/main/src/main.rs, you'll see a commented out section that'll retrieve the GPIO pin levels. Uncommenting it causes any alt-configured pins to be changed to inputs.

remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 9 (delta 4), reused 7 (delta 2), pack-reused 0
Unpacking objects: 100% (9/9), 5.33 KiB | 1.78 MiB/s, done.
From https://github.com/golemparts/gpiochip
0f7ec31..fcd6b6d  main       -> origin/main
Updating 0f7ec31..fcd6b6d
Fast-forward
src/ioctl.rs | 615 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------------
src/main.rs  |  48 ++++++++---
2 files changed, 476 insertions(+), 187 deletions(-)
ukscone@glooston:~/gpiochip $ cargo run
Compiling gpiochip v0.1.0 (/home/ukscone/gpiochip)
Finished dev [unoptimized + debuginfo] target(s) in 0.66s
Running target/debug/gpiochip
ChipInfo: "gpiochip4" "pinctrl-rp1" 54
0: ID_SD Consumer:[] Flags:[input]
1: ID_SC Consumer:[] Flags:[input]
2: PIN3 Consumer:[] Flags:[input]
3: PIN5 Consumer:[] Flags:[input]
4: PIN7 Consumer:[onewire@0] Flags:[used output open_drain]
5: PIN29 Consumer:[] Flags:[input]
6: PIN31 Consumer:[] Flags:[input]
7: PIN26 Consumer:[spi0 CS1] Flags:[used active_low output]
8: PIN24 Consumer:[spi0 CS0] Flags:[used active_low output]
9: PIN21 Consumer:[] Flags:[input]
10: PIN19 Consumer:[] Flags:[input]
11: PIN23 Consumer:[] Flags:[input]
12: PIN32 Consumer:[] Flags:[input]
13: PIN33 Consumer:[] Flags:[input]
14: PIN8 Consumer:[] Flags:[input]
15: PIN10 Consumer:[] Flags:[input]
16: PIN36 Consumer:[] Flags:[input]
17: PIN11 Consumer:[] Flags:[input]
18: PIN12 Consumer:[] Flags:[input]
19: PIN35 Consumer:[] Flags:[input]
20: PIN38 Consumer:[] Flags:[input]
21: PIN40 Consumer:[] Flags:[input]
22: PIN15 Consumer:[] Flags:[input]
23: PIN16 Consumer:[] Flags:[input]
24: PIN18 Consumer:[] Flags:[input]
25: PIN22 Consumer:[] Flags:[input]
26: PIN37 Consumer:[] Flags:[input]
27: PIN13 Consumer:[] Flags:[input]

oh btw not sure if you realized that once pios (bookworm) is released libgpio and thus gpiochip (&gpiomem probably) will be the way for the gpio on the pi (all models) any other methods will go away. i think you are ok with the pi1 to pi4 but just wanted to make sure you were aware :)

Yeah, thankfully I moved away from using sysfs years ago. I think I'll just have to switch to gpiochip for now on systems that have the v2 API available (another thing to check for. good thing bookworm is required for the Pi 5 at least), and reassess /dev/gpiomem and /dev/rp1-gpiomem at a later date. I'll have to remove some of RPPAL's functionality, but considering how widely gpiochip / libgpiod are used, at least we'll all be in the same boat. 😁

I've also come across https://github.com/raspberrypi/utils/tree/master/pinctrl after spotting Gadgetoid's fancy new CLI tool that uses its data. Unfortunately it requires sudo, so not really an option until that changes.

After sleeping on it, I think the best way forward is to make gpiochip the default interface used by the Gpio module, which means I'll have to remove the above-mentioned functionality. The existing gpiomem interface, including the removed functionality, will be put behind a feature flag, so older systems that don't have the v2 API or users that require the removed functionality are still able to use RPPAL the same way as before.
I'll let you know when the library is ready for some Pi 5 testing.

two documents have been released today that might be of use to you (disclaimer: not read them myself yet)
https://datasheets.raspberrypi.com/rp1/rp1-peripherals.pdf rp1-peripheral and https://datasheets.raspberrypi.com/bcm2711/bcm2711-peripherals.pdf  bcm2711 peripherals

Oh wow. I thought we weren't getting any RP1 documentation for weeks. That is going to be very helpful for adding rp1-gpiomem support. This certainly changes things. I should be able to add support for reading/writing to the registers directly based on this, which means gpiochip and its limitations won't be an issue anymore.
Would you mind checking if /dev/rp1-gpiomem has properly configured file permissions? I would expect its group to be set to gpio with read/write permissions.

ukscone@glooston:/dev $ ls -al /dev/rp1-gpiomem
crw-rw---- 1 root gpio 240, 0 Oct  6 10:41 /dev/rp1-gpiomem
ukscone@glooston:/dev $

Just to give you a quick update, I've had a busy week because of unrelated work stuff, but I should have some time over the next few days to get this up and running. I've made some initial internal changes to use a different gpiomem interface based on the Pi model, but I haven't implemented any specific functionality for the Pi 5 yet.
Based on what I found in the DTs and the documentation I think I have an accurate idea of the rp1-gpiomem layout, but misinterpretation and bugs are possible, so we should take things slow when testing this. Will ping you when it's time to run gpio_status 😁

i'll be out and about on Saturday but apart that that i'm at your beck and call :)

@ukscone I implemented all the needed functionality to work with rp1-gpiomem. The internal interface is a lot more involved compared to the old gpiomem so there's a higher chance for bugs, but we'll have to start testing to find any issues at this point since the code looks good to me.
While reading from the registers should be safe, there is a risk when changing modes / levels / bias. Values could be written to the wrong register, causing unexpected results. If you'd rather stick to only testing read operations, I completely understand.
When you have some time, you can update your local rppal repo and run cargo --run example gpio_status. Assuming you have some pins configured for I2C, SPI, as outputs, etc, you should be seeing the appropriate Alt modes or Input/Output and the pin levels in the output.
Should we run into any unsolvable issues, I'll probably ask you to dump the contents of rp1-gpiomem to a file so I can use that to do some debugging on my side. It's not perfect, but it beats having no test data at all.

heh you are not going to believe this. switched to bookworm release image rather than the beta image i had been using and the rp1-gpiomem has gone away :)
ukscone@glooston:~/rppal $ cargo run --example gpio_status
Updating crates.io index
Downloaded simple-signal v1.1.1
Downloaded lazy_static v1.4.0
Downloaded libc v0.2.149
Downloaded 3 crates (729.2 KB) in 1.00s
Compiling libc v0.2.149
Compiling lazy_static v1.4.0
Compiling rppal v0.14.1 (/home/ukscone/rppal)
Compiling simple-signal v1.1.1
Finished dev [unoptimized + debuginfo] target(s) in 6.45s
Running target/debug/examples/gpio_status
Error: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })
smh
we now have 5 gpiomems -- gpiomem0-4
lol it doesn't rain it pours :)

Well, on the bright side, at least that matches with the bcm2712-rpi.dtsi I've been basing some of this on. gpiomem0 should contain the same stuff. I'll make the change and we can see what roadblock we run into next. 😁

Should be good to go now.

the libgpiod tool gpioinfo shows this on the release image
ukscone@glooston:~/rppal $ gpioinfo
gpiochip0 - 32 lines:
line   0:          "-"       unused   input  active-high
line   1: "2712_BOOT_CS_N" "spi10 CS0" output active-low [used]
line   2: "2712_BOOT_MISO" unused input active-high
line   3: "2712_BOOT_MOSI" unused input active-high
line   4: "2712_BOOT_SCLK" unused input active-high
line   5:          "-"       unused   input  active-high
line   6:          "-"       unused   input  active-high
line   7:          "-"       unused   input  active-high
line   8:          "-"       unused   input  active-high
line   9:          "-"       unused   input  active-high
line  10:          "-"       unused   input  active-high
line  11:          "-"       unused   input  active-high
line  12:          "-"       unused   input  active-high
line  13:          "-"       unused   input  active-high
line  14:   "PCIE_SDA"       unused   input  active-high
line  15:   "PCIE_SCL"       unused   input  active-high
line  16:          "-"       unused   input  active-high
line  17:          "-"       unused   input  active-high
line  18:          "-"       unused   input  active-high
line  19:          "-"       unused   input  active-high
line  20:   "PWR_GPIO" "pwr_button"   input   active-low [used]
line  21: "2712_G21_FS" unused input active-high
line  22:          "-"       unused   input  active-high
line  23:          "-"       unused   input  active-high
line  24:     "BT_RTS"       unused   input  active-high
line  25:     "BT_CTS"       unused   input  active-high
line  26:     "BT_TXD"       unused   input  active-high
line  27:     "BT_RXD"       unused   input  active-high
line  28:      "WL_ON"  "wl_on_reg"  output  active-high [used]
line  29:      "BT_ON"   "shutdown"  output  active-high [used]
line  30: "WIFI_SDIO_CLK" unused input active-high
line  31: "WIFI_SDIO_CMD" unused input active-high
gpiochip1 - 4 lines:
line   0: "WIFI_SDIO_D0" unused input active-high
line   1: "WIFI_SDIO_D1" unused input active-high
line   2: "WIFI_SDIO_D2" unused input active-high
line   3: "WIFI_SDIO_D3" unused input active-high
gpiochip2 - 17 lines:
line   0:    "RP1_SDA"       unused   input  active-high
line   1:    "RP1_SCL"       unused   input  active-high
line   2:    "RP1_RUN" "RP1 RUN pin" output active-high [used]
line   3: "SD_IOVDD_SEL" "vdd-sd-io" output active-high [used]
line   4:  "SD_PWR_ON" "sd_vcc_reg"  output  active-high [used]
line   5:  "SD_CDET_N"       unused   input  active-high
line   6:   "SD_FLG_N"       unused   input  active-high
line   7:          "-"       unused   input  active-high
line   8:  "2712_WAKE"       unused   input  active-high
line   9: "2712_STAT_LED" "ACT" output active-low [used]
line  10:          "-"       unused   input  active-high
line  11:          "-"       unused   input  active-high
line  12:   "PMIC_INT"       unused   input  active-high
line  13: "UART_TX_FS"       unused   input  active-high
line  14: "UART_RX_FS"       unused   input  active-high
line  15:          "-"       unused   input  active-high
line  16:          "-"       unused   input  active-high
gpiochip3 - 6 lines:
line   0:  "HDMI0_SCL"       unused   input  active-high
line   1:  "HDMI0_SDA"       unused   input  active-high
line   2:  "HDMI1_SCL"       unused   input  active-high
line   3:  "HDMI1_SDA"       unused   input  active-high
line   4:   "PMIC_SCL"       unused   input  active-high
line   5:   "PMIC_SDA"       unused   input  active-high
gpiochip4 - 54 lines:
line   0:      "ID_SD"       unused   input  active-high
line   1:      "ID_SC"       unused   input  active-high
line   2:       "PIN3"       unused   input  active-high
line   3:       "PIN5"       unused   input  active-high
line   4:       "PIN7"       unused   input  active-high
line   5:      "PIN29"       unused   input  active-high
line   6:      "PIN31"       unused   input  active-high
line   7:      "PIN26"   "spi0 CS1"  output   active-low [used]
line   8:      "PIN24"   "spi0 CS0"  output   active-low [used]
line   9:      "PIN21"       unused   input  active-high
line  10:      "PIN19"       unused   input  active-high
line  11:      "PIN23"       unused   input  active-high
line  12:      "PIN32"       unused   input  active-high
line  13:      "PIN33"       unused   input  active-high
line  14:       "PIN8"       unused   input  active-high
line  15:      "PIN10"       unused   input  active-high
line  16:      "PIN36"       unused   input  active-high
line  17:      "PIN11"       unused   input  active-high
line  18:      "PIN12"       unused   input  active-high
line  19:      "PIN35"       unused   input  active-high
line  20:      "PIN38"       unused   input  active-high
line  21:      "PIN40"       unused   input  active-high
line  22:      "PIN15"       unused   input  active-high
line  23:      "PIN16"       unused   input  active-high
line  24:      "PIN18"       unused   input  active-high
line  25:      "PIN22"       unused   input  active-high
line  26:      "PIN37"       unused   input  active-high
line  27:      "PIN13"       unused   input  active-high
line  28: "PCIE_RP1_WAKE" unused input active-high
line  29:   "FAN_TACH"       unused   input  active-high
line  30:   "HOST_SDA"       unused   input  active-high
line  31:   "HOST_SCL"       unused   input  active-high
line  32:  "ETH_RST_N"  "phy-reset"  output   active-low [used]
line  33:          "-"       unused   input  active-high
line  34: "CD0_IO0_MICCLK" "cam0_reg" output active-high [used]
line  35: "CD0_IO0_MICDAT0" unused input active-high
line  36: "RP1_PCIE_CLKREQ_N" unused input active-high
line  37:          "-"       unused   input  active-high
line  38:    "CD0_SDA"       unused   input  active-high
line  39:    "CD0_SCL"       unused   input  active-high
line  40:    "CD1_SDA"       unused   input  active-high
line  41:    "CD1_SCL"       unused   input  active-high
line  42: "USB_VBUS_EN" unused output active-high
line  43:   "USB_OC_N"       unused   input  active-high
line  44: "RP1_STAT_LED" "PWR" output active-low [used]
line  45:    "FAN_PWM"       unused  output  active-high
line  46: "CD1_IO0_MICCLK" "cam1_reg" output active-high [used]
line  47:  "2712_WAKE"       unused   input  active-high
line  48: "CD1_IO1_MICDAT1" unused input active-high
line  49: "EN_MAX_USB_CUR" unused output active-high
line  50:          "-"       unused   input  active-high
line  51:          "-"       unused   input  active-high
line  52:          "-"       unused   input  active-high
line  53:          "-"       unused   input  active-high

Nice! GPIO 9, 10 and 11 marked as Alt0 is exactly as it should be when you have SPI enabled. Looks like it's reading the correct data.

it seems to be doing all the right things. i've asked in the pi forums what's up with the rp1-gpiomem vanishing and multiple gpiomems. it makes life easier in the end not to have to have two different gpiomems to have to check for but i don't recall it being mentioned anywhere and i thought the rp1 doc mentioned rp1-gpiomem

Considering read operations work as expected, that leaves write operations.
That will either require some coding (or using any of your existing code), or using the gpio_blinkled or gpio_blinkled_signals examples with an LED and resistor attached to GPIO 23 (physical pin 16) to see if changing modes and levels works. Another thing to double check would be to write some code to check the level on an input pin while it changes between high and low. It would also be helpful if you could test enabling a pull-up or pull-down and physically checking the voltage on the pin.
Let me know if you need any help with writing up some sample code.

i'll dig out some leds, jumper wires and breadboard

it seems to be doing all the right things. i've asked in the pi forums what's up with the rp1-gpiomem vanishing and multiple gpiomems.

Yeah the whole naming convention is a bit vague to me, but I can at least see the point in using a different name for gpiomem on older Pi models compared to the Pi 5, considering the registers are completely different. On my Pi 4 running Bookworm I'm still only seeing the old gpiomem without any suffix, so at least no changes are needed there. As long as they've settled on this scheme we should be okay.

gpio_blinkled & gpio_blinkled_signals are blinking an led :)
so gpio23 is working :)

Awesome! Thanks for the quick test results. I'm glad it's all going smoothly. That means both mode changes and level changes are working. That just leaves checking an input pin's level with the level getting changed between high and low by some external source, and checking the voltage on an input pin with pull-up/pull-down enabled, both of which will require some coding and external parts/multimeter (happy to help if needed).
It sounds like it's all working as expected though. After you get a chance to check the above-mentioned, that should be the entire GPIO module ready for the Pi 5! While I don't expect any major issues with the other peripherals, if you have any hardware lying around that would let you perform some tests on SPI/I2C/PWM/UART, it would be good to confirm they're still working before launch day.
BTW, the new GPIO registers also give us access to a few new toys (change output drive strength, input Schmitt trigger hysteresis, output slew rate limiter, debounce), but I'll wait until I have my hands on a Pi 5 as well before adding support for those, so something to look forward to for 0.16.0. 😁

i'll dig out all my bits and pieces tomorrow and start playing. i have a few i2c & spi displays & sure i have a few pwm thingies just a matter of finding them

dug out my ds3231 and the example i2c_ds3231.rs seems to work as expected

Great to hear! I wasn't expecting any problems, but it's always good to check.
It looks like rppal is ready for the Pi 5. I'll get 0.15.0 out the door within the next day or two. Thanks again for all the help!

great. i'm looking forward to the release as then i'll be able to try porting one of my last python programs left to rust (& tauri) Pity we don't seem to have a rust equivilent of the python module luma but we can't have everything :)

RPPAL v0.15.0 with Raspberry Pi support is now available!

@golemparts I'm getting the same error in most recent version :(
jezor@raspberry:~$ git clone https://github.com/golemparts/rppal.git
Cloning into 'rppal'...
remote: Enumerating objects: 5079, done.
remote: Counting objects: 100% (824/824), done.
remote: Compressing objects: 100% (120/120), done.
remote: Total 5079 (delta 746), reused 725 (delta 704), pack-reused 4255 (from 1)
Receiving objects: 100% (5079/5079), 1.51 MiB | 6.33 MiB/s, done.
Resolving deltas: 100% (3496/3496), done.

jezor@raspberry:~$ cd rppal

jezor@raspberry:~/rppal$ cargo run --example spi_25aa1024
    Updating crates.io index
     Locking 21 packages to latest compatible versions
      Adding embedded-hal v0.2.7 (latest: v1.0.0)
      Adding nb v0.1.3 (latest: v1.1.0)
   Compiling libc v0.2.158
   Compiling lazy_static v1.5.0
   Compiling rppal v0.19.0 (/home/jezor/rppal)
   Compiling simple-signal v1.1.1
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.30s
     Running `target/debug/examples/spi_25aa1024`
Error: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })

Thank you @golemparts, this worked!
jezor@raspberry:~/rppal$ cargo run --example spi_25aa1024
    Updating crates.io index
     Locking 21 packages to latest compatible versions
      Adding embedded-hal v0.2.7 (latest: v1.0.0)
      Adding nb v0.1.3 (latest: v1.1.0)
   Compiling libc v0.2.158
   Compiling lazy_static v1.5.0
   Compiling rppal v0.19.0 (/tmp/rppal)
   Compiling simple-signal v1.1.1
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 8.58s
     Running `target/debug/examples/spi_25aa1024`
Bytes read: [0, 0, 0, 0, 0]

I found this issue after I reported mine #154
Was the HW pwn tested and where there OS settings I might have missed that where not documented?


Add integration for BugZilla as a Remote Issue tracker

related https://trello.com/c/5rzcAYJD/15-create-model-for-other-issue-tracker-integration

Tried it out here : https://github.com/DhritiShikhar/ALM_issues/blob/master/alm_bugzilla.go


Issue with creating custom modifiers in ConText

I'm trying to create custom modifiers using ConTextRule (previously ConTextItem), however the documentation and other sources on the internet refer to module ConTextComponent which is not in the package anymore. I can see there is a new module ConTextModifier but it doesn't appear to be just a refactor as similar code doesn't work. Can someone help with this? Which packages do I need to create custom modifiers and is there any documentation/tutorial?
Thanks.

@Arooj-Hussain  Thank you for asking this question.  If you have found examples or documentation of older class names like ConTextComponent then indeed those class names and code are older and have been changed.
Have you consulted the 4 notebooks we have on ConText in our notebooks?  These are up to date.  I have checked them myself just now:
https://github.com/medspacy/medspacy/tree/master/notebooks/context

Thank you @burgersmoke for the reply. The notebooks will indeed be very helpful moving ahead.
Meanwhile I figured out that it has been changed to work the same way as TargetRule and TargetMatcher for other rules. Thanks.

OK, sounds great.  I'll close this for now, but please open a new issue or discussion as needed.


Seems slower than Ionic 1

I am trying out this app to see the component of Ionic 2 and how it perform. My biggest hope is to see it perform much faster than Ionic 1. The components are looking great. But it's performance seems to be much slower than Ionic 1. I am testing it on a low end Android 4.4 phone. The respond and everything is quite slow as compare to my app which is developed using Ionic 1.
Really hope the performance of Ionic 2 can be improved and perform better than Ionic 1.

I ran the app on my phone (Samsung Galaxy A5 2014) using "ionic run android". The splashscreen shows for around 15-18 seconds. It is shockingly slow. Once the app is finished loading, the performance isn't great either. It takes about a second for the tab to change. Surely this is not normal. Can anyone else share their experience on loading times and performance of this app?

@sunny061 Could you make sure you have the latest version of the project, a fresh copy of node_modules, and then try running ionic run android --prod and let me know what times you're seeing. Thanks!

I tried "ionic run android --prod". The splash screen loading time has decreased to about 6.5 - 7 seconds. It's a big improvement from the 15-18 seconds I was getting without the "--prod" flag. Will this be improved some more?

I tried "ionic run android --prod". The splash screen loading time has decreased to about 6.5 - 7 seconds. It's a big improvement from the 15-18 seconds I was getting without the "--prod" flag. Will this be improved some more?

Are you performing any storage read before setting rootPage?

@mohamedsharaf Don't think so. I am just running the app without any modifications and I haven't looked into the code so I don't know if the app does any storage read. Also, I am not running any other apps at the same time.

@sunny061 would you mind posting the specs of your device?

i have tested on samsung note 5
it takes around 4 or 5  seconds till splash screen disappear
is this normal for ionic2 or i have to work on improvement
note5
4GB RAM
Exynos 7420 Octa
http://www.gsmarena.com/samsung_galaxy_note5-7431.php
android 6

Hello all, thanks for using Ionic! I am going to close and lock this thread for now as this one was. We are constantly working on improving performance with Ionic including load time performance so look for more improvements to come in the near future around performance.

Also, just so everyone knows, you can find alot of good info on the issue i linked to above on some of our plans on how to improve load time performance.


tensorflow-gpu version

Hi,
very nice work indeed!
can you provide your tensorflow-gpu version? i encounter some problem when I try use ScanNet to run demo.
install
git clone https://github.com/jertubiana/ScanNet.git
cd ScanNet
conda create -n py_scannet python=3.6.12
conda activate py_scannet

i delete row tensorflow==1.14.0 in requirements.txt because i want use gpu.
then i install packages with pip install -r requirements.txt;pip install tensorflow-gpu.

   version in my environment

 Package                 Version  ----------------------- -----------  absl-py                 0.15.0  argon2-cffi             21.3.0  argon2-cffi-bindings    21.2.0  astunparse              1.6.3  async-generator         1.10  attrs                   22.1.0  backcall                0.2.0  biopython               1.78  bleach                  4.1.0  cached-property         1.5.2  cachetools              4.2.4  certifi                 2022.6.15  cffi                    1.15.1  charset-normalizer      2.0.12  clang                   5.0  cycler                  0.11.0  dataclasses             0.8  decorator               5.1.1  defusedxml              0.7.1  entrypoints             0.4  flatbuffers             1.12  gast                    0.4.0  google-auth             1.35.0  google-auth-oauthlib    0.4.6  google-pasta            0.2.0  grpcio                  1.47.0  h5py                    3.1.0  idna                    3.3  importlib-metadata      4.8.3  ipydatawidgets          4.3.1.post1  ipykernel               5.5.6  ipython                 7.16.3  ipython-genutils        0.2.0  ipywebrtc               0.5.0  ipywidgets              7.5.1  jedi                    0.17.2  Jinja2                  3.0.3  joblib                  1.1.0  jsonschema              3.2.0  jupyter-client          7.1.2  jupyter-core            4.9.2  jupyterlab-pygments     0.1.2  keras                   2.6.0  Keras-Applications      1.0.8  Keras-Preprocessing     1.1.2  kiwisolver              1.3.1  llvmlite                0.35.0  Markdown                3.3.7  MarkupSafe              2.0.1  matplotlib              3.3.3  mistune                 0.8.4  nbclient                0.5.9  nbconvert               6.0.7  nbformat                5.1.3  nest-asyncio            1.5.5  notebook                6.4.10  numba                   0.52.0  numpy                   1.19.5  oauthlib                3.2.0  opt-einsum              3.3.0  packaging               21.3  pandas                  1.1.5  pandocfilters           1.5.0  parso                   0.7.1  pexpect                 4.8.0  pickleshare             0.7.5  Pillow                  8.4.0  pip                     21.3.1  prometheus-client       0.14.1  prompt-toolkit          3.0.30  protobuf                3.19.4  ptyprocess              0.7.0  pyasn1                  0.4.8  pyasn1-modules          0.2.8  pycparser               2.21  Pygments                2.13.0  pyparsing               3.0.9  pyrsistent              0.18.0  python-dateutil         2.8.2  pythreejs               2.3.0  pytz                    2022.2.1  PyYAML                  6.0  pyzmq                   23.2.1  requests                2.27.1  requests-oauthlib       1.3.1  rsa                     4.9  scikit-learn            0.24.2  scipy                   1.5.4  Send2Trash              1.8.0  setuptools              58.0.4  six                     1.15.0  tensorboard             2.6.0  tensorboard-data-server 0.6.1  tensorboard-plugin-wit  1.8.1  tensorflow-estimator    2.6.0  tensorflow-gpu          2.6.2  termcolor               1.1.0  terminado               0.12.1  testpath                0.6.0  threadpoolctl           3.1.0  tornado                 6.1  traitlets               4.3.3  traittypes              0.2.1  typing-extensions       <IP_ADDRESS>  urllib3                 1.26.12  wcwidth                 0.2.5  webencodings            0.5.1  Werkzeug                2.0.3  wheel                   0.37.1  widgetsnbextension      3.5.2  wrapt                   1.12.1  zipp                    3.6.0

run
then i run python predict_bindingsites.py P38398 --noMSA, it raise exception
  File "predict_bindingsites.py", line 406, in predict_interface_residues
    model_obj = wrappers.load_model(model_folder + model, Lmax=Lmax)
  File "/data/wenyuhao/code/ScanNet/utilities/wrappers.py", line 80, in load_model
    model_builder, *model_args, **model_kwargs, backend=backend, **wrapper_builder_kwargs)
  File "/data/wenyuhao/code/ScanNet/utilities/wrappers.py", line 217, in __init__
    model_builder, *args, backend=backend, **kwargs)
  File "/data/wenyuhao/code/ScanNet/utilities/wrappers.py", line 95, in __init__
    self.model = self.model_builder(*self.args, **self.kwargs)
  File "/data/wenyuhao/code/ScanNet/network/scannet.py", line 351, in ScanNet
    embedded_attributes_aa = attribute_embedding(masked_attributes_aa,nembedding_aa,activation,name='embedded_attributes_aa')
  File "/data/wenyuhao/code/ScanNet/network/scannet.py", line 90, in attribute_embedding
    name=name)
  File "/data/wenyuhao/code/ScanNet/network/scannet.py", line 69, in addNonLinearity
    epsilon=1e-3, axis=-1, center=center, scale=scale, name=name + '_normalization')(input_layer) # Custom batch norm layer that takes into account the mask.
  File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/keras/engine/base_layer.py", line 977, in __call__
    input_list)
  File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/keras/engine/base_layer.py", line 1115, in _functional_construction_call
    inputs, input_masks, args, kwargs)
  File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/keras/engine/base_layer.py", line 848, in _keras_tensor_symbolic_call
    return self._infer_output_signature(inputs, args, kwargs, input_masks)
  File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/keras/engine/base_layer.py", line 888, in _infer_output_signature
    outputs = call_fn(inputs, *args, **kwargs)
  File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 695, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    /data/wenyuhao/code/ScanNet/network/embeddings.py:464 call  *
        del reduction_axes[self.axis]

    TypeError: list indices must be integers or slices, not ListWrapper

because of value axis of the object  MaskedBatchNormalization is ListWrapper in  /network/embeddings.py:464 .
maybe version of tensorflow-gpu is not matched.
it really make confused , run in centos7. Looking forward to.

Hi,
Thanks for your interest in our research! Sorry for the issue, the right tensorflow-gpu version is tensorflow-gpu==1.14.
Please let me know if that works for you.
Best regards,
Jérôme

On 30 Aug 2022, at 5:24, wenyuhao @.***> wrote:
Hi,
very nice work indeed!
can you provide your tensorflow-gpu version? i encounter some problem when I try use ScanNet to run demo.
install
git clone https://github.com/jertubiana/ScanNet.git
cd ScanNet
conda create -n py_scannet python=3.6.12
conda activate py_scannet
i delete row tensorflow==1.14.0 in requirements.txt because i want use gpu.
then i install packages with pip install -r requirements.txt;pip install tensorflow-gpu.
version in my environment
run
then i run python predict_bindingsites.py P38398 --noMSA, it raise exception
File "predict_bindingsites.py", line 406, in predict_interface_residues
model_obj = wrappers.load_model(model_folder + model, Lmax=Lmax)
File "/data/wenyuhao/code/ScanNet/utilities/wrappers.py", line 80, in load_model
model_builder, *model_args, **model_kwargs, backend=backend, **wrapper_builder_kwargs)
File "/data/wenyuhao/code/ScanNet/utilities/wrappers.py", line 217, in init
model_builder, *args, backend=backend, **kwargs)
File "/data/wenyuhao/code/ScanNet/utilities/wrappers.py", line 95, in init
self.model = self.model_builder(*self.args, **self.kwargs)
File "/data/wenyuhao/code/ScanNet/network/scannet.py", line 351, in ScanNet
embedded_attributes_aa = attribute_embedding(masked_attributes_aa,nembedding_aa,activation,name='embedded_attributes_aa')
File "/data/wenyuhao/code/ScanNet/network/scannet.py", line 90, in attribute_embedding
name=name)
File "/data/wenyuhao/code/ScanNet/network/scannet.py", line 69, in addNonLinearity
epsilon=1e-3, axis=-1, center=center, scale=scale, name=name + '_normalization')(input_layer) # Custom batch norm layer that takes into account the mask.
File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/keras/engine/base_layer.py", line 977, in call
input_list)
File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/keras/engine/base_layer.py", line 1115, in _functional_construction_call
inputs, input_masks, args, kwargs)
File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/keras/engine/base_layer.py", line 848, in _keras_tensor_symbolic_call
return self._infer_output_signature(inputs, args, kwargs, input_masks)
File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/keras/engine/base_layer.py", line 888, in _infer_output_signature
outputs = call_fn(inputs, *args, **kwargs)
File "/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 695, in wrapper
raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:
/data/wenyuhao/code/ScanNet/network/embeddings.py:464 call  *
    del reduction_axes[self.axis]

TypeError: list indices must be integers or slices, not ListWrapper

because of value the object MaskedBatchNormalization is ListWrapper in /network/embeddings.py:464 .
maybe version of tensorflow-gpu is not matched.
it really make confused , run in centos7. Looking forward to.
—
Reply to this email directly, view it on GitHub https://github.com/jertubiana/ScanNet/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHSPJR4CDTMYAI2GPE5AVLV3V5GLANCNFSM5776ORXQ.
You are receiving this because you are subscribed to this thread.

Thank you! it works for me after installing tensorflow-gpu==1.14!!! very nice work.
And i will get some numba warinings,such as
/data/wenyuhao/anaconda3/envs/py_scannet/lib/python3.6/site-packages/numba/typed/typedlist.py:66: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'item' of function 'impl_append.<locals>.impl'.

For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../anaconda3/envs/py_scannet/lib/python3.6/site-packages/numba/typed/listobject.py", line 597:

    def impl(l, item):
    ^

  l.append(item)

i think it also because of version of numba mismatch,can i see your full requenirment.txt in your environment?
Thank you.


iOS 13.2 breaks

Just reproducing the iOS SDK tutorial does not work on iOS 13.2. I'm using  Blockstack 1.0.1
Currently signed out so signing in.
MyApp(1094,0x10514d840) malloc: can't allocate region
*** mach_vm_map(size=6090375168) failed (error code=3)
MyApp(1094,0x10514d840) malloc: *** set a breakpoint in malloc_error_break to debug

Is this an objective-c app?

Closing this as I cannot reproduce it.