Building large modular JS / ScalaJS projects

Hi,
I'm trying for some time to find a solution to this problem and nobody seems to find it either.
What I want is to be able to build a large app that has at least 30 modules with ScalaJsn.
The app already exists, and the modules have been published on a private NPM repository, and are aggregated into one or different JS client apps (connected user/visitor/mobile/cordova/clipper bookmarklet...) by using Browserify or Webpack.
If I were to use ScalaJS I would migrate the existing modules one by one.
I am concerned by the build time, but also the bundle size.
However, not like others I'm perfectly fine with a 10Mb production bundle, even after full optimization (this is already near what we get with plain JS for a medium sized SPA, and it loads fast enough with gzip compression... but we also have something like 20 libraries used...)
As the project is not so small and modularized, I think it would not make sens to put all modules in a single Scala project (or SBT submodules) as it would probably take a long compilation time (I may be wrong?).
So I was thinking of being able to publish each module on NPM independently (wether it is a migrated ScalaJS module, or a legacy plain JS module) and glue these modules together with a plain JS project with Browserify / Webpack.
The problem is that as far as I know there's at least 100k of scalajs runtime per module that would be published on NPM, and aggregating 30+ modules with that overhead will lead to a lot of overhead. Not to mention all my modules would be using scalajs-react so the overhead would probably be much greater than a plain old HelloWorld
So I am wondering if there is any solution to this problem?
I guess optimizing the final output after Browserifycation will not reduce that overhead as GCC / Uglify will not be able to deduplicate ScalaJS Runtime things (right?)
I was suggesteds here that there could be something to be done by using .sjs files directly but don't really know how it could solve the problem.
Some useful links:
https://github.com/rockymadden/scala-node/issues/3
https://github.com/scala-js/scala-js/issues/1477
http://underscore.io/blog/posts/2015/06/10/scalajs-scaladays.html
What I wonder is if it would be possible, when publishing a ScalaJS project to NPM, to use something like dependency or peerDependency so that the external code is excluded from the project but has to be provided by the module client. Somehow I think this means that the ScalaJS runtime needs to be modularized and published on NPM so that other ScalaJS modules can use require.
I mean if 2 modules use the Scala collections, instead of packaging the Scala collections into the ScalaJS runtime, maybe I can publish my 2 modules with a npm dependency on scalajs-collections instead. You see what I mean? Don't know if it's possible at all...

This relates to https://github.com/scala-js/scala-js/issues/1703, and somewhat to https://github.com/scala-js/scala-js/issues/1751
Its currently being debated. I think pushing stuff to npm is out of question (mainly because we already have webjars. We are currently debating about how much of the module system should be done by SBT (as a plugin, or otherwise part of core) vs how much should be part of the framework, as well as defining what modules are in the first place (in the javascript sense)
I am still writing up what my own vision is, mainly so I have something concrete to discuss with @jducoeur, but for now, its still being discussed.

sorry @mdedetrich but I don't agree with you and wonder why using npm is out of question.
For me webjars is absolutly not what I'm looking for. We are building JS apps, that may integrate some ScalaJS components but the main language is still JS. I'm using Gulp / NPM with a quite complex build pipeline and won't do anything with ScalaJS if it forces me to use SBT or webjars.
Webjars solves an unexisting problem for people that did not want to invest time in learning the JS ecosystem. There are already mature solutions for JS dependencies and build, why not use them, just because you are not used to use these tools? It's not at all complicated to use a package.json file and type npm install on the terminal, why would I need something else?
In addition I need to use tools like Browserify or Webpack because I need to bundle my app. I produce 3 separate fils (appPolyfills.js, appLibs.js and app.js, the first 2 trying to hit the cache the most they can). I'm not loading my app with plain old list of script tags on the top of an HTML file. And as far as I know Browserify is probably not compatible with Webjar as it requires the NPM node_modules folder to be available with the dependencies. It is out of question to assign libraries to the global scope.
As explained I have something like 30 components. So if I want to use ScalaJS for one of them I have to switch my whole pipeline to SBT? And then what if I want to test 3 different solutions that compile to JS, would it make sens to use the language specific build tool everytime, forcing me to maintain 3 different build setups?
To me, not using NPM is crazy. It is the well-accepted js solution for dependency management and thus is the common denominator for all projects that need to use JS libraries.
Even if using a a Scala backend (like we do), you do not have to couple your frontend build to what you use on the frontend. We use Play2 backend, and our SPA is served by Play2, but still we use JS specific tools to build JS apps, instead of trying to reuse the existing SBT build.
I must admin I've already used SBT to do frontend stuff, like using webjars or compiling LESS files, minifying... The build is simpler since we use a single tool and it works great for simple usecases. but is absolutly not suitable for large frontend projects.

sorry @mdedetrich but I don't agree with you and wonder why using npm is out of question.

I don't think its out of the question, but its a lot of effort for something that

Segregates the ecosystem
Is something thats really hard to get "right", especially when we need to account for breaking API changes in Scala.js

To be clear, there is nothing stopping people from making an integration layer for npm in the form of a plugin if they want to, but I definitely wouldn't count on the core Scala.js team doing something like that.

For me webjars is absolutly not what I'm looking for. We are building JS apps, that may integrate some ScalaJS components but the main language is still JS. I'm using Gulp / NPM with a quite complex build pipeline and won't do anything with ScalaJS if it forces me to use SBT or webjars.

We also use npm/grunt for out frontend management, but we are going to be moving away from it (thankfully we are at a stage were this is a manageable proposition). So in this regard, I perfectly understand where you guys are coming from, but I would rather that efforts (at least if we are talking about the Scala.js team, and the community in general) be taken in improving the SBT build tool so it does the tasks that we would typically do with npm/grunt.
The reason why webjars are mentioned, is that if you are are using SBT, that is essentially the bower/npm, for SBT. Its the repository where compiled JS/CSS/HTML assets are placed that interfaces with maven

In addition I need to use tools like Browserify or Webpack because I need to bundle my app. I produce 3 separate fils (appPolyfills.js, appLibs.js and app.js, the first 2 trying to hit the cache the most they can). I'm not loading my app with plain old list of script tags on the top of an HTML file. And as far as I know Browserify is probably not compatible with Webjar as it requires the NPM node_modules folder to be available with the dependencies. It is out of question to assign libraries to the global scope.

Yup, and on my end, we are using require.js + other tasks to take care of the pipeline when it comes to asset management. So this leaves us with 2 directions

Implement functionality as plugins/core into SBT so it does what Browserify/Webpack does.
or
Implement grunt integration with Scala.js

The former is where I believe we should be putting (most) of our effort. I think realistically someone doing a light plugin for SBT that allows you to wrap @JSExported Scala code into an AMD module (and also wrapping the runtime) has merit as a stop gap solution (with grunt taking over the rest), but the former is definitely where we want to be.

As explained I have something like 30 components. So if I want to use ScalaJS for one of them I have to switch my whole pipeline to SBT? And then what if I want to test 3 different solutions that compile to JS, would it make sens to use the language specific build tool everytime, forcing me to maintain 3 different build setups?

Thats twisting what my statement was. My proposition is that the core team shouldn't spend a non trivial amount of time trying to create a plugin layer for an entirely different build system that is outside of the ecosystem. I didn't say that people should be forced to move their entire build systems, that clearly isn't realistic. For example, internally in our company we use wrappers for grunt and bower to interface with SBT, i.e.

https://github.com/mdedetrich/sbt-bower
https://github.com/rossbayer/grunt-sbt

These plugins are like 100 lines of code (not a lot) and just wrap around the npm/bower commands.

To me, not using NPM is crazy. It is the well-accepted js solution for dependency management and thus is the common denominator for all projects that need to use JS libraries.

Firstly lets be clear, NPM is the well accepted solution for Javascript source dependency management. SBT is the accepted solution for Scala source dependency management (which uses Maven behind the scenes). npm isn't a dependency management solution for Java/C/C++/(insert other language here), so expecting that SBT should inherently support a build tool for a completely different language is a bit farfetched. The fact that Scala.js generates Javascript means that it treats Javascript as a bytecode format, just like compiling Scala with scalac generates JVM ByteCode as a byteCode format. Scala.js definitely gives you more control over how things are outputted, because you actually need to run the Javascript, as well as integrate with other code, but claiming that SBT should support an package management system for an entire different language with an entire different ecosystem is a bit absurd
The other thing is, what you mentioned, isn't the well-accepted js solution, its one of them. Javascript in this sense is terrible, where they have competing module systems (AMD/Common.js/ new ECMA script modules), competing asset management systems (webpack, browserify, etc etc), competing DI systems (require.js vs frameworks providing their own), competing package management systems (node.js,bower), etc etc.
Within our company, we have used grunt.js, but I now regret it, however we didn't really have a choice. Stuff in Javascript land moves too fast in regards to how modules/DI/Lazy Loading + framework integration (and everything around it), moves. Our entire frontend build system, which uses grunt/bower/npm is already obsolete, doesn't compile with the latest version of node, and it would require as to rewrite our build system for it to get it to work with the latest version, and this all happened in just over a year.
I have been doing frontend development for 5 years now (along with backend development), and we definitely do not want to replicate what happens in JS land in this regard. What happened with JS is definitely understandable since they didn't establish a standard for modules, however its very chaotic in this regard, and I think we should avoid this if possible.

Even if using a a Scala backend (like we do), you do not have to couple your frontend build to what you use on the frontend. We use Play2 backend, and our SPA is served by Play2, but still we use JS specific tools to build JS apps, instead of trying to reuse the existing SBT build.

No you don't, we havn't and I suspect most companies haven't. But we are now in this territory were we have Scala compiling to Javascript, and just like you would expect to use Javascript build tools to deal with Javascript, we would expect Scala build tools (which currently is SBT) to use build and manage Scala code
For right now, you would have to make an SBT plugin that tries to create NPM modules out of the scala runtime, and also from @JSExported code. This isn't something that is easy to do, but I am not sure what you would expect. You have competing build systems, and you are trying to integrate them in a very tightly, coupled way. Since Scala.js is in Scala, you need SBT to build your Scala code. You would have this exact same problem with any JS backend from any other language if you want to integrate it with npm/grunt. Thats not even taking into account that npm is package management for node.js backend (not even getting to other backends, like vert.x and bower). Scala.js, I believe, in its design, is meant to be javascript client agnostic, so using NPM is preferring one javascript backend over others.

I must admin I've already used SBT to do frontend stuff, like using webjars or compiling LESS files, minifying... The build is simpler since we use a single tool and it works great for simple usecases. but is absolutly not suitable for large frontend projects.

Agreed, and many people (including you, and myself) are stock between a rock and a hard place in this regard, but I think claiming that SBT should provide support for grunt/npm just because it happens to be what javascript is drawing a really long bow. I would rather most of the effort go towards SBT/SBT plugins + frameworks to fix this.
Putting npm support in SBT is a really short sighted solution, which will cause a lot of pain in the long run.

@mdedetrich I diidn't say we should support NPM through SBT.
I'm perfectly fine to use ScalaJS tooling while still being in the ScalaJS world.
But my JS client side app is a composition of ScalaJS and plain old JS modules and libraries.
For example my client side app could be:

A main JS file entry point
3 Scala JS modules
20 NPM modules
Gulp / Browserify to produce a single JS file file

I don't have any problem using SBT for the ScalaJS things. I don't want to use SBT to do things with npm or Gulp or Browserify. But at some point I have to package my JS app and Browserify is a good tool for that.
So the idea is to use SBT to produce JS files, publish them on npm and then do like if they were regular JS modules. Once published, these modules won't ever have to be recompiled when my client side app changes.
The " big optimizations" of ScalaJS does not need to happen because optimizing is already in my JS pipeline when NODE_ENV=production. When fetched the node_modules are not optimized either and it's the choice of the client to perform the optimizations he wants to. (but maybe it's still required for ScalaJS as I wouldn't like to have modules of 20mb published on npm)
Doing all the stuff like Grunt in SBT. It is up to you. Wrapping your JS build in SBT does not solve at all the problem because at the end it's still the JS tooling that kicks in. I want to use Browserify or Webpack on a large project and being able to use advanced packaging configuration, hot code reloading and stuff like that.  These tools don't work with Webjars.
I don't care about using NPM or bower or whatever, but these tool are used my many, and many libraries have setup a package.json to express dependencies. Whatever you do with SBT, if I can't use the dependency graph already setup in these libs, or have to contact the autors to create or maintain webjars it is a no go: you have to integrate with existing tools first before reinventing your wheel or nobody will adopt...
NPM is no more source dependency management. Javascript is the browser VM so it's a very similar role comprared to Maven: you publish what everyone is able to understand, wether it is compiled Java or Scala for the JVM, or Javascript for the browser.

I don't know the JS tools like npm and Grunt terribly well, so I've mostly been keeping quiet in this discussion.  One note, though: the most crucial Scala.js optimization is dead code elimination, and you almost certainly do want that one in it -- otherwise, your application is apt to contain lots of code that is never being called.  And DCE has a lot of implications for the way the stack works: it's a primary reason why SJS thinks in terms of the application as a whole, why JSExports need to be stated, and so on...

@jducoeur isn't there a way to publish unoptimized code to NPM, and then do the Dead Code Elimitation at a later stage? Because clearly I want DCE to happen at the end of my mown packaging pipeline, but still I need it efficient...

@slorber No, you cannot publish unoptimized code to npm, because there is no such thing as unoptimized code, or, to be more precise, Scala.js does not optimize js code. It optimizes its .sjsir files. So if you publish .js files to npm, you have already linked and optimized one project at a time, and you cannot re-link after that.
GCC is an implementation detail in Scala.js. It is not trivially applied on the output of fastOptJS. There are other stuff going on. It is virtually impossible to apply correctly (and even less so reliably) GCC to Scala.js code if you do it yourself, outside Scala.js' tools.
If you know you'll have more Scala.js projects that will form your application, you should publish them as .jars, and use the Scala.js linker and optimizer at the latest stage. .jars contain .sjsir files (not .js files), and are therefore the unoptimized form that is meant to be published. If you link everything together at the last stage, you won't have duplicate runtime.

As the project is not so small and modularized, I think it would not make sens to put all modules in a single Scala project (or SBT submodules) as it would probably take a long compilation time (I may be wrong?).

You are mostly wrong. sbt uses incremental compilation, so that the modular separation between sbt subprojects matters little to the compilation time. Even so, you can separate in several sbt subprojects, that does not mean you can link everything together in one final subproject that depends on all the others. Scala.js' linker and optimizer is also incremental, so that the amount of code to be optimized in one pass does not matter much to the compilation time starting from the second run in one sbt session (although it does have an impact on the RAM used).

@jducoeur isn't there a way to publish unoptimized code to NPM, and then do the Dead Code Elimitation at a later stage? Because clearly I want DCE to happen at the end of my mown packaging pipeline, but still I need it efficient...

@sjrd explained it well, but more succinctly, Scala.js treats Javascript literally as a form of bytecode, which means only the final output (defined by @JSExport) is javascipt, before that is an intermediate formats (.sjsir)
There is a discussion which I linked earlier that is about creating facade's for code compiled with Scala.js, but thats only about splitting your emitted code in several parts so you can load them in stuff like CDN. Getting those parts to work with general JS code is a different ballgame.
Like mentioned earlier, its a lot of work, you would have to make a separate backend, and you would also have to adhere to an javascript compatible ABI. Although Scala.js has fantastic interoperability for Javascript, its not designed to be a transcompiler like Coffeescript or Typescript

@sjrd explained it well, but more succinctly, Scala.js treats Javascript literally as a form of bytecode, which means only the final output (defined by @JSExport) is javascipt, before that is an intermediate formats (.sjsir)

To be more precise, we treat javascript code as machine code, on which we can't do anything more, and can only feed to the "processor" (the js vm). Our bytecode is sjsir files.

Ok thanks I understand your point.
So if I understand correctly, the best way to produce a JS file is to have my modules as Scala jars, and compile them all at the same time and produce a single optimized output. Then I can still use this ScalaJS code as a library in another JS project. Thanks to incremental compilation, and the fast optimizer, I can still be productive in dev environment. Right?
So once I get that "machine code" I guess I can publish it to NPM (even if that does not bring me anything ^^)
Instead of having a single ScalaJS packaging pipeline for all modules, is it possible to have one pipeline per module, and publish the multiple "machine code" js files to NPM? I guess we can but it's not recommended as 2 ScalaJS modules will probably share a lot of scala js internals machine code and make my bundle big.
I'm asking all this because if we were to introduce ScalaJS to our project it would be very gradually. We have frontend developers with no prior experience with Scala and that are bored to death because of compilation time when they have to work with Play2 scala templates. So we need to be able to test ScalaJS without requiring these developers to have to recompile ScalaJS everytime they change their git branch or things like that, and using NPM seems the right solution to solve this problem.

Yes, you can have one pipeline per module. And yes, some code will be duplicated in several modules.

Thanks, and what is the approximate amount of duplicated machine code if I have something like 10 independant ScalaJS modules? Would it be something like 10* 150k? Would repassing the code iin GCC / UglifyJS reduce that size? Or would Gzip compress that duplicated code efficiently?

what is the approximate amount of duplicated machine code if I have something like 10 independant ScalaJS modules? Would it be something like 10* 150k?

It will deep depend a lot on how much shared code is actually used by the different modules.

Would repassing the code iin GCC / UglifyJS reduce that size?

No.

Or would Gzip compress that duplicated code efficiently?

No.

ok thanks @sjrd
I ask asking for an estimate don't hesitate to give something that is very wrong :) I would use on each module reactjs-scala, immutable collections, tuples, case classes, options, map, flatmap, futures...
Just want to know if this is realistic to split for a production usecase or if I should rather go with a single ScalaJS compilation pipeline and only use jars.

What would the feasibility be of a mode for the linker to receive N input modules and generate N+1 output JS files, while still doing dead code elimination?
The idea would be that if you have K javascript "binaries" (N>=K), the output would be one JS file containing all 3rdparty code, then one blob for each of your N modules with dead code elimination deciding what is eliminated from each module.
This would handle the case of incrementally loading portions of a javascript app that is split across multiple pages.

@slorber I think you may find that if you consider compiling two JS blobs with two toolchains, most things will work themselves out. Use scala.js with sbt to compile all your modules, think of it as babel + webpack/browserify + uglify. Use npm et al for your js stuff. Then you just need to make sure that "shared libraries" like jquery et al are available to both.
Personally I'm using the -jsdeps file from scala.js for this since I am stuck with require on the .js side, but if you're using a full toolchain you should be able to construct an equivalent file using js tools quite easily.

I believe the problems and solutions raised in this discussion are addressed by a combination of:

Emitting CommonJS modules (see also #2175)
scalajs-bundler
Possibly #2681 if the output needs to be decomposed for lazy-loading reasons

great thanks :)


Add cabal-hoogle to nix infra

Description
Credits to @amesgen.
With this, inside a shell you can do cabal-hoogle generate && cabal-hoogle run -- server --local and have a hoogle server running

I'll leave it to Alex to approve this one, but should we mention in the README that now people can use this tool to fire up a hoogle server?

I mentioned it now in the CONTRIBUTING.md file.


CI-388 - addition of requestValidationMode to web.config to handle da…

CI-388 - addition of requestValidationMode to web.config to handle dangerous path URL's the same way as das-console (i.e. send to 404 page)

Removed invalid character attribute from httpRuntime as it wasn't required


Exporting to video rendering

Is this supported? https://www.youtube.com/watch?v=kw07wyP3qm4

@RichardLindhout
I haven't tested it specifically, but as long as it is a function that the photoshop javascript API can support, this API can basically be implemented


How do i handle a dynamic list of items?

7.7.0
I have a usecase where the user can create a list of pricing options for an item.
Each of these items contains:
Type - A user created name for the pricing type (String)
Cost - The actual value (String -> double)
This list could vary from 0 to a hypothetical maximum of 10.
This is all contained within a single field "pricing"
An example result:
[
	{
		"type": "General"
		"cost": 400,
	},
	{
		"type": "Early bird",
		"cost": 200,
	}
]

A similar json can also be provided to it via the initial value, in the case where an existing pricing structure is being edited.
How do I implement this optimally?
I've attempted to create a FormBuilderField with a nested form (and a listview within it), but interactions cause it to bug out.
Ive read a response about dynamic fields and using register but Im not sure how to use it in this context.
Any help would be appreciated!

Additional information.
This field is within a nested form ("requirements") within the main form.

This is a question relative to your code work, I can't help too much


GEODE-5806: Adding example for durable messaging in client subscriptions

Thanks for the feedback @davebarnes97!

"Events for durable queries and subscriptions are saved in queue" -> "save in a queue"

This  one is actually just right out of our manual so I think the same fix needs to go there.
https://geode.apache.org/docs/guide/16/developing/events/implementing_durable_client_server_messaging.html

"A simple cache listener is used to print output to terminal create events are received" -> "output to the terminal as create events are received"

Do you think it isn't worth mentioning that a cache listener is used to print to terminal?  I suppose they can see that in the code, but technically cache listeners are a different topic and I didn't want users to be lost if they didn't know about cache listeners.

"Run the example to demonstrate continues queries" -> "to demonstrate durable messaging"

This one is just a copy paste typo.  Good catch, thanks!

"saved in queue" -> "save in a queue"
This one is actually just right out of our manual so I think the same fix needs to go there.

Yes, thanks for the pointer.

"A simple cache listener is used to print output to terminal create events are received" -> "output to the terminal as create events are received"

Do you think it isn't worth mentioning that a cache listener is used to print to terminal? I suppose they can see that in the code, but technically cache listeners are a different topic and I didn't want users to be lost if they didn't know about cache listeners.

Not my intention - unclear comment on my part. I'm suggesting the following:

Leave the 'cache listener' part in place
Replace  "output to terminal create events are received" with "output to the terminal as create events are received"

Thanks for clarifying @davebarnes97!  Appreciate the feedback.


I added a new class pollingWorker in a new thread that prints values of parameters to the terminal at rates specified in the config file

new file created "pollingWorker.py".
serverConfigExample.txt
The example config file demonstrates how to use the worker. Adding the pollingRates field will create a new Qtimer for each parameter with the specified interval in the config file (in seconds). Timers will only be created for existing parameters when the server is started. Adding new parameters in the parameter manager will not create new timers. The worker runs in a new Qthread "pollingThread" created in apps.py. The thread is stopped when the server is stopped in core.py.

I have added a few more cosmetic changes that I found when I was looking over the code again.
I still think there should be a way of avoiding that import in the middle of the code. Let me see if I can figure some workaround


[SDPA-4527] Moved search related requests to search page

JIRA issue: https://digital-engagement.atlassian.net/browse/SDPA-4527
Changed

Moved search related site domains requests to search page. Which was a blocking request for all pages.

Screenshots

@tim-yao looks good to me.
Interested to see performance comparison (: Do we have third party app to track website's performance?

We have Google Analytics to track the page speed only. For lighthouse audit, we have a simple audit in Circle CI and has a plan to improve it.


[Translation zh-tw] _book.yaml

Check out a review request at GitLocalize

Reviewed by @limhenry translated by @kennith

We have signed Corporate CLA but it will takes time to accept a few days according FAQ for Google CLA. https://cla.developers.google.com/about

Sadly, we can't localize _book.yaml or toc.yaml files.

Due to the way DevSite works, localizing some files will cause some pretty serious headaches:

_book.yaml, _toc.yaml or _upper_tabs-*.yaml
_project.yaml

But, we can do _footer.yaml!
Sorry! :(

@plete Oh. I got it. We will lock these files to avoid translating in GitLocalize.

@sotayamashita how would that be possible from our side? Can we manipulate translatable/non-translatable directories/files?

@agektmr

how would that be possible from our side?
Can we manipulate translatable/non-translatable directories/files?

We will implement a feature to ignore some file on GitLocalize. After implementing the feature, we will configure these file manually instead of you.


Automated cherry pick of #183

Cherry pick of #183 on release-1.0.
#183: BUGFIX: token file should not be decoded

Codecov Report

Merging #189 into release-1.0 will increase coverage by 0.1%.
The diff coverage is 100%.

@@              Coverage Diff               @@
##           release-1.0     #189     +/-   ##
==============================================
+ Coverage        94.46%   94.56%   +0.1%
==============================================
  Files                9        9
  Lines              668      681     +13
==============================================
+ Hits               631      644     +13
  Misses              37       37

Impacted Files
Coverage Δ

kubernetes/config/kube_config.py
96.47% <100%> (+0.1%)
:arrow_up:

kubernetes/config/kube_config_test.py
92.22% <100%> (+0.23%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3517bed...2c80352. Read the comment docs.

original PR has LGTM. the diff is exactly the same.

This is a backward incompatible change (people may fixed their token file to make it work). Reverting PR.


Grey Rectangles Appearing in background of PDF

Attach (recommended) or Link to PDF file here:
Day Trade Spy Case Study(v1).pdf
Configuration:

Web browser and its version: Chrome 68.0.3440.106 (Official Build) (64-bit)
Operating system and its version: Windows 10
PDF.js version:
Is a browser extension: No

Steps to reproduce the problem:

Open pdf with PDFJS
Observe gray rectangles in background

What is the expected behavior? (add screenshot)

What went wrong? (add screenshot)

Link to a viewer (if hosted on a site other than mozilla.github.io/pdf.js or as Firefox/Chrome extension):

Duplicate of issue #269.


update to Healpix 3.70

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found here.

I'm not sure why the Linux build is failing, could anyone test locally?

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I have tried to debug this to the best of my abilities. The segmentation fault seems to come from within libm of all places. The issue appears to be the same that is described in this StackOverflow question, even though the cause is different: The stack trace for this PR looks identical to that one, and I have also run with LD_DEBUG to verify that fma is the last symbol lookup before the crash.
2020-11-09T16:49:45.3024294Z      20479:	symbol=fma;  lookup in file=.libs/alm2map_cxx [0]
2020-11-09T16:49:45.3025055Z      20479:	symbol=fma;  lookup in file=/home/conda/feedstock_root/build_artifacts/healpix_cxx_1604940406704/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/libhealpix_cxx.so.2 [0]
2020-11-09T16:49:45.3026077Z      20479:	symbol=fma;  lookup in file=/home/conda/feedstock_root/build_artifacts/healpix_cxx_1604940406704/_build_env/lib/libstdc++.so.6 [0]
2020-11-09T16:49:45.3026584Z      20479:	symbol=fma;  lookup in file=/lib64/libm.so.6 [0]
2020-11-09T16:49:45.3027724Z      20479:	binding file /home/conda/feedstock_root/build_artifacts/healpix_cxx_1604940406704/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/libsharp.so.1 [0] to /lib64/libm.so.6 [0]: normal symbol `fma' [GLIBC_2.2.5]
2020-11-09T16:49:45.3029047Z /home/conda/feedstock_root/build_artifacts/healpix_cxx_1604940406704/work/conda_build.sh: line 23: 20479 Segmentation fault      (core dumped) LD_DEBUG=all .libs/alm2map_cxx nlmax=128 infile=test.alm 'outfile=!test.fits' nside=64 polarisation=true

In addition, valgrind also identifies fma as the last named symbol before the crash.
Not sure where to go from here. Feel free to close, I suppose.

@conda-forge/healpix_cxx I have also tried to work around the issue by moving to CentOS 7 with glibc 2.17. I'm not sure how to finish the configuration though, i.e. adding a correct conda_build_config.yaml file. Help?

Moving to CentOS 7 and glibc 2.17 seems to have resolved this bizarre issue at last.

So I'm fine with requiring centos7. However, I'd be nice to understand why.


Aliexpress source for low-profile nylocks?

I'd like to do this mod, but I cannot find AXIC1052 anywhere. Amazon delivery here is hilariously expensive (more than the item in question), and the only nylock nuts I can find are about 4mm tall.

Sorry, I don't know of any sources. But no worries, in practice you just need a single low-profile nylock at the center back, you need to make sure it clears the Y-axis motor mount. It's no biggie, really - just a few minutes of sanding down a regular nylock on some sandpaper should do the trick.
If anyone stumbles upon a source for these nuts, feel free to update the docs with a link via a PR.


Add status_code instead of status and query to logs

What
Changed the format of error logs and catching rust exceptions during e=query.
How to review
Run query:
http://localhost:28800/categories?query=Are%20there%20any%20national%20statistics%20that%20show%20fishermen%20have%20higher%20divorce%20rate%20than%20the%20average%20shore%20based%20workerAre%20there%20any%20national%20statistics%20that%20show%20fishermen%20have%20higher%20divorce%20rate%20than%20the%20average%20shore%20based%20workeAre%20there%20any%20national%20statistics%20that%20show%20fishermen%20have%20higher

Who can review
Linden.

This does not actually trigger an exception when tested, but is a good comparison.


Change theme from light?

I'm trying to use a dialog confirm box.  My nuxt/vuetify app uses dark theme, but I'm not sure why the dialog box comes up with the light theme.
                const res = await this.$dialog.confirm({                     title: 'Warning',                     text: 'Are you sure you want to remove SQL Group ' + sqlGroup.SQL_ID                 });
The div element for the card is:
<div class="v-card v-sheet v-sheet--tile theme--light">
which is not what I want, I'd want theme--dark.
How do I achieve this?
Thanks.

Hello, @sm373373. Thank you for issue. I've begun to fix it

fixed in v0.3.1 version


add Joomla! chart

My install failed as follows:
RunContainerError: "GenerateRunContainerOptions: secrets \"dusty-antelope-mariadb\" not found"

 $  kubectl get secret
NAME                    TYPE                                  DATA      AGE
default-token-h3678     kubernetes.io/service-account-token   3         17d
dusty-antelope-joomla   Opaque                                2         3m

Looks like the secret is not referenced properly the pod is looking for dusty-antelope-mariadb while dusty-antelope-joomla is being created.

Looks like previous comment was because i didnt do the dep update before running the chart. Rerunning now.


[Feature Request] Will Newtonsoft.Json support 'System.Numerics.BigInteger' in future?

We have a game. In the game, players can get credits more than that UInt64 can save.

Emmm... So? Is Newtonsoft.Json based on Bson rules?

I have an idea. If I use decimal and limit the integer to a certain range(-10^26 ~ 10^26), is the integer accurate in that range?


python module: get item from id ?

Can the python module retrieve a catalog item from its id ?
That's very easy to do with the client, but from the python module I cannot find a way.
For now the only way I found to programmatically get a specific VDS given its ID and path is to start at the root et move throught the path, folder by folder. And this has a limitation given most of my folders have spaces in them, which must be converted to underscores:
"path": ["BDF", "VDS Management"]

In [45]: catalog["BDF"]["VDS Management"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-45-b5d3437247dc> in <module>
----> 1 catalog["BDF"]["VDS Management"]

KeyError: 'VDS Management'

In [46]: catalog["BDF"]["VDS_Management"]
Out[46]: {"entityType": "folder", "id": "652074a1-7c74-4f25-bc82-e64c32d0cdbb", "path": ["BDF", "VDS Management"], "tag": "vjbOHwWjwZo=", "accessControlList": {"users": null, "groups": null, "version": "1"}}

Thank you for your advice.

Hey @insatomcat
The 'Simple' client has https://github.com/rymurr/dremio_client/blob/master/dremio_client/dremio_simple_client.py#L89 which will fetch by catalog id. Nothing in the object based client though. You might be able to leverage that to get the path and apply that to the catalog. Or there may be an easy way to add the that method to the catalog

Thanks, I solved my problem using the simple client which was perfect for this usecase.
However I looked into adding the method to the catalog, see #62 which creates an object from an arbitrary cid or path.

fixed in 0.12.1


Doc is outdated.

3.0 support is added but docs says not supported.
Also there is no mention of Linux support. - it should be mentioned here (what is supported what is not supported) to avoid confusion.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 098ffe3a-2cbd-1038-30f2-dd2f528984a3
Version Independent ID: 65da3e70-2ac2-8f6e-51a5-e759bcc8253f
Content: Monitor Azure app services performance
Content Source: articles/azure-monitor/app/azure-web-apps.md
Service: azure-monitor
Sub-service: application-insights
GitHub Login: @mrbullwinkle
Microsoft Alias: mbullwin

@cijothomas Thank you for bringing this to our attention!  I've assigned this to the author so they can update the documentation accordingly.


[BUG]: Audit Log via O365 Manamagent API - different seruity screen as in documentation

Describe the bug
If I setup the Custom Connector I have other fields to fill out as discribed in the documentation.
Component (please tell us which flow or app you are experiencing issues with):

Solution Center of Excellence Audit Logs
Version 1.32
Flow with Azure App ID registration

To Reproduce
Steps to reproduce the behavior:
Follow the steps in:

API-Permission in Azure is granted.
And I have saved the Secret ClientKey, the Client Key and the Tenant ID.
But in the Security Tab in the custom connector setup I don't have these fields:
• Login URL
• Tenant ID
• Resource URL
But i have these fields:
• Authorization URL
• Token URL
• Refresh URL

How must I fill out these fields?

Are you installing via option 1 or option 2?
If option 1, please note:
(Step 5) Select Create Connector. You don't need to change the Security and Definition information.
If option 2, please note:
(Step 6) Select Edit at the bottom of the OAuth 2.0 area to edit the authentication parameters.

I'm using option 2 and If i select "Edit" I can modify the fields in my screenshot.
But my attached screenshot looks different than the one in the documentation.
As mentioned, I have these 3 fields in my screenshot:
• Authorization URL
• Token URL
• Refresh URL
But in the documentation, these fields are listed to fill out:
• Login URL
• Tenant ID
• Resource URL

Oh I see now, you seem to have gotten the incorrect identity provider, which is why you are seeing the incorrect input options. Can you please try the steps again?

Perfect. Thanks. This not really fixed the problem. But because of this hint, I re-installed the solution and now it works.
Thank you very much!


Process inline JS with SystemJS

For reference: https://github.com/ModuleLoader/es6-module-loader/issues/95

Inline js is now processed by SystemJS when the script tag type is module.  For example:
  <script type="module">
    import 'polymer/polymer.html!html';
    import marked from 'marked';

    Polymer({
      is: 'dom-element-dep',
      properties: {
        greeting: {
          type: String,
          value: '*I\'m a DOM element. This is my local DOM!*'
        }
      },
      ready: function() {
        var markdown = marked(this.greeting);
        this.$.content.innerHTML = markdown;
      }
    });
  </script>

However, relative urls use the root document base url.

Full element in the tests: https://github.com/Hypercubed/systemjs-plugin-html/blob/master/test/import-with-dependency/dom-element.html

Currently only works for direct imports.  Nested imports do not have script type="module" tags processed.

@guybedford brought up a good point here:  https://github.com/ModuleLoader/es6-module-loader/issues/95#issuecomment-121970049
What to do with <script type="module" src="filename.js"></script> tags.

Any news on SystemJS.import for html or other plugin dependent imports ?


Correct some small errors in tutorial examples

Addresses: https://github.com/angular/angular.io/issues/2231
And also a small issue where an <h1> tag is being included in a template where it was previously deleted.
At https://angular.io/docs/ts/latest/tutorial/toh-pt5.html:
Near the top, user sees:

Since the user has moved the <h1> tag to another template, we should not see it here, below:

Thanks in advance for your consideration of these tweaks.

Looks good to me, but that is a big new file just for the template.

Makes sense, I can cut that down to just the essentials.

Thanks for taking this on. I appreciate the effort and your care.
I don't think we should add an entire file just to delete the <h1> . The maintenance burden is not worth it. You can see the problems already. For example, your version still has the title property which shouldn't be there. And all those irrelevant #docregion tags.
I think the world will survive an extra <h1>, especially if we tell folks to delete it.
Your other change to ToH-6 is important.
I incorporated these thoughts into #2264. Closing yours. Thanks.


locator pallet

Is this pallet is working?

Should the sddrafts folder be published to github?
It appears to be a forgotten secret. I updated the readme and added the missing items.
If I am reading this pallet correctly, the locators need to be in the locators.gdb prior to a run. This is problematic for a few reasons. We should be able to purge the staging data and have things rebuild properly.
A corrupt locator has no way to recover without intervention.

Failed to execute. Parameters are not valid. ERROR 000732: Input Address Locator: Dataset staging\locators.gdb\AddressPoints_AddressSystem does not exist or is not supported Failed to execute (RebuildAddressLocator).

Keeping a gdb of valid locators with the project in github could give us the opportunity to generate the locators.gdb by arcpy.copy_management. We could update the data, (delete if exists) copy the valid locator into staging, rebuild, and finally publish.
Thoughts?

I don't think that we can assume that everything in staging is reproducible. There are some datasets that are static that are maintained there. Although I agree that this would be ideal.
I wonder if the copy method would work reliably. I've had varied success copying address locators.

There are some datasets that are static that are maintained there.

This was not be the intention of the staging area as staging should not become another MapData mess. We should be able to add and remove pallets, delete the staging area and only have what is necessary be there. There needs to be another solution to the static problem.

We should be able to add and remove pallets, delete the staging area and only have what is necessary be there.

That would break some production services if we removed multiple pallets that participated in updating the same fgdb.
I can see where you are coming from. I'd be open to other solutions for static data.

The use case would more so be that we retire an application than run a subset of necessary pallets.
Will you open another ticket with the static data you are aware of and how it's being managed in staging?

So, the best practice for creating locators, for more performance etc, is to store them on the file system like a toolbox - outside of a geodatabase. We should consider this change for an upcoming 10.5 update and for the performance and possible stability issue was currently have.


Support for evaluating changes to IAM policy for Pub/Sub topics and subscriptions

Updates to the stackdriver parser to support pubsub topic and subscription iam policy changes

Closing again to see if re-creating removes the dockercloud CI check that wont complete


Error: #<Bosh::Clouds::CloudError: Invalid data disk name (version 1) -- in disk attach -- in Azure

But when I manually attach .....no problem..and happening on Azure.
command:
bosh -e bosh-my-env -d my-deployment-name attach-disk blueprint_z1/06457717-5e63-4d1a-94d0-671fb59cdd4e  my-disk-25917-2017112412334
output:
Error: CPI error 'Bosh::Clouds::CloudError' with message 'attach_disk - Failed to create the managed disk for my-disk-25917-20171124123341. Error: #<Bosh::Clouds::CloudError: Invalid data disk name (version 1) my-disk-25917-20171124123341
/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/helpers.rb:163:in cloud_error' /var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/disk_id.rb:133:in parse_storage_account_v1'
/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/disk_id.rb:101:in storage_account_name' /var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/cloud.rb:384:in block in attach_disk'
/var/vcap/packages/bosh_azure_cpi/vendor/bundle/ruby/2.4.0/gems/bosh_common-1.3262.24.0/lib/common/thread_formatter.rb:49:in with_thread_name' /var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/cloud.rb:371:in attach_disk'
/var/vcap/packages/bosh_azure_cpi/vendor/bundle/ruby/2.4.0/gems/bosh_cpi-2.4.1/lib/bosh/cpi/cli.rb:82:in public_send' /var/vcap/packages/bosh_azure_cpi/vendor/bundle/ruby/2.4.0/gems/bosh_cpi-2.4.1/lib/bosh/cpi/cli.rb:82:in run'
/var/vcap/packages/bosh_azure_cpi/bin/azure_cpi:39:in <top (required)>' /var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:74:in load'
/var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:74:in kernel_load' /var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:27:in run'
/var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:362:in exec' /var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run'
/var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command' /var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor.rb:387:in dispatch'
/var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:22:in dispatch' /var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/base.rb:466:in start'
/var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:13:in start' /var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/exe/bundle:30:in block in <top (required)>'
/var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/friendly_errors.rb:121:in with_friendly_errors' /var/vcap/packages/ruby_azure_cpi/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/exe/bundle:22:in <top (required)>'
/var/vcap/packages/ruby_azure_cpi/bin/bundle:23:in load' /var/vcap/packages/ruby_azure_cpi/bin/bundle:23:in '' in 'attach_disk' CPI method

@sauravmndl what i suspect is that cpi expects to have disks with particular names. may be @AbelHu has an idea how to support what you are trying to do? (i assume its some disks that you have to recover)

Azure has naming rules for disk name. Please see the rules here.

@AbelHu  and @cppforlife Thank you very much for your help so far. I have given a acceptable name () in attach command, but giving me following error now
DirectorJobRunner: CPI error 'Bosh::Clouds::CloudError' with message 'Unknown disk caching xxxxxxxxxx' in 'attach_disk' CPI method /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/cloud/external_cpi.rb:195:inhandle_error'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/cloud/external_cpi.rb:99:in invoke_cpi_method' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/cloud/external_cpi.rb:61:in attach_disk'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/disk_manager.rb:76:in attach_disk' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/attach_disk.rb:78:in handle_new_disk'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/attach_disk.rb:31:in block in perform' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:136:in _transaction'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:110:in block in transaction' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/connecting.rb:251:in block in synchronize'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/connection_pool/threaded.rb:105:in hold' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/connecting.rb:251:in synchronize'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:99:in transaction' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/transactor.rb:5:in block in retryable_transaction'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/retryable.rb:28:in block in retryer' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/retryable.rb:26:in loop'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/retryable.rb:26:in retryer' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/common.rb:118:in retryable'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/transactor.rb:4:in retryable_transaction' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/attach_disk.rb:29:in perform'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:96:in perform_job' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:32:in block in run'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/thread_formatter.rb:49:in with_thread_name' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:32:in run'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/base_job.rb:10:in perform' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:36:in block in perform'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:85:in block (3 levels) in run' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in block in create_with_logging_context' D, [2017-11-27 12:17:42 #5561] [task:820] DEBUG -- DirectorJobRunner: (0.000084s) SELECT NULL D, [2017-11-27 12:17:42 #5561] [task:820] DEBUG -- DirectorJobRunner: (0.000194s) SELECT * FROM "tasks" WHERE ("id" = 820) LIMIT 1 D, [2017-11-27 12:17:42 #5561] [task:820] DEBUG -- DirectorJobRunner: (0.000122s) SELECT NULL D, [2017-11-27 12:17:42 #5561] [task:820] DEBUG -- DirectorJobRunner: (0.000053s) BEGIN

@sauravmndl It seems like that you used an invalid caching xxxxxxxxxx. Please see the code here.
valid_caching = ['None', 'ReadOnly', 'ReadWrite']

Hi @AbelHu , thanks for the help. Now I have found the correct command. Bosh cli is accepting the command but throwing following error saying volume not found. But it is actually present in Azure. Following is the error detail.
[2017-11-27 18:33:11 #13053] [task:869] ERROR -- DirectorJobRunner: Action Failed get_task: Task b5b3835f-b379-44c0-6f25-9ad693e5be0d result: Persistent disk with volume id 'disk_name:bosh-disk-data-0a8f47bc-e934-4a7f-90c5-ec775d5b71ae;caching:None;resource_group_name:rg-sample-azure-dev1' could not be found /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/agent_client.rb:301:in handle_method'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/agent_client.rb:356:in handle_message_with_retry' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/agent_client.rb:58:in method_missing'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/agent_client.rb:412:in get_task_status' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/agent_client.rb:206:in wait_for_task'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/agent_client.rb:376:in send_message' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/agent_client.rb:106:in mount_disk'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/disk_manager.rb:169:in mount_disk' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/disk_manager.rb:78:in attach_disk'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/attach_disk.rb:78:in handle_new_disk' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/attach_disk.rb:31:in block in perform'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:136:in _transaction' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:110:in block in transaction'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/connecting.rb:251:in block in synchronize' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/connection_pool/threaded.rb:105:in hold'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/connecting.rb:251:in synchronize' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:99:in transaction'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/transactor.rb:5:in block in retryable_transaction' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/retryable.rb:28:in block in retryer'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/retryable.rb:26:in loop' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/retryable.rb:26:in retryer'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/common.rb:118:in retryable' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/transactor.rb:4:in retryable_transaction'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/attach_disk.rb:29:in perform' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:96:in perform_job'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:32:in block in run' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-0.0.0/lib/common/thread_formatter.rb:49:in with_thread_name'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:32:in run' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/base_job.rb:10:in perform'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:36:in block in perform' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:85:in block (3 levels) in run'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in block in create_with_logging_context'
D, [2017-11-27 18:33:11 #13053] [task:869] DEBUG -- DirectorJobRunner: (0.000094s) SELECT NULL
D, [2017-11-27 18:33:11 #13053] [task:869] DEBUG -- DirectorJobRunner: (0.000237s) SELECT * FROM "tasks" WHERE ("id" = 869) LIMIT 1
D, [2017-11-27 18:33:11 #13053] [task:869] DEBUG -- DirectorJobRunner: (0.000133s) SELECT NULL
D, [2017-11-27 18:33:11 #13053] [task:869] DEBUG -- DirectorJobRunner: (0.000063s) BEGIN
D, [2017-11-27 18:33:11 #13053] [task:869] DEBUG -- DirectorJobRunner: (0.000414s) UPDATE "tasks" SET "state" = 'error', "timestamp" = '2017-11-27 18:33:11.981151+0000', "description" = 'attach disk ''disk_name:bosh-disk-data-0a8f47bc-e934-4a7f-90c5-ec775d5b71ae;caching:None;resource_group_name:rg-sample-azure-dev1'' to ''blueprint_z1/c6df4272-3efc-44ee-a186-ce8101fb12db''', "result" = 'Action Failed get_task: Task b5b3835f-b379-44c0-6f25-9ad693e5be0d result: Persistent disk with volume id...', "output" = '/var/vcap/store/director/tasks/869', "checkpoint_time" = '2017-11-27 18:32:47.015496+0000', "type" = 'attach_disk', "username" = 'admin', "deployment_name" = 'service-fabrik-1047-a8ff8ce0-9577-4506-be04-3483ddd2f2f2', "started_at" = '2017-11-27 18:30:46.968654+0000', "event_output" = '{"time":1511807591,"error":{"code":450001,"message":"Action Failed get_task: Task b5b3835f-b379-44c0-6f25-9ad693e5be0d result: Persistent disk with volume id ''disk_name:bosh-disk-data-0a8f47bc-e934-4a7f-90c5-ec775d5b71ae;caching:None;resource_group_name:rg-sf-azure-dev1'' could not be found"}}
', "result_output" = '', "context_id" = '' WHERE ("id" = 869)
D, [2017-11-27 18:33:12 #13053] [task:869] DEBUG -- DirectorJobRunner: (0.025650s) COMMIT
I, [2017-11-27 18:33:12 #13053] []  INFO -- DirectorJobRunner: Task took 2 minutes 25.041717989999995 seconds to process.
Task 869 error
Capturing task '869' output:
Expected task '869' to succeed but state is 'error'
Exit code 1`

@sauravmndl From the disk name disk_name:bosh-disk-data-0a8f47bc-e934-4a7f-90c5-ec775d5b71ae;caching:None;resource_group_name:rg-sample-azure-dev1 in your latest logs, it shows that you are using managed disks in V2 format. If so, you should be able to find the managed disk bosh-disk-data-0a8f47bc-e934-4a7f-90c5-ec775d5b71ae under the resource group rg-sample-azure-dev1 in Azure portal. Please confirm it at first.

@AbelHu  I am confirming again that the disk is present in the resource group.

@sauravmndl It seems like that Azure CPI has attached this disk to the VM (I think you can double confirm this in Azure portal) and the error was thrown by bosh-agent. I suggest you to login to the VM, check attached disks by lsblk and check the disks' information in /var/vcap/bosh/settings.json.

@AbelHu
I can see the disks section of /var/vcap/bosh/setings.json contains following  but sdd is not included in lsblk command
"disks": {                                                                                "raw_ephemeral": null,                                                                                "persistent": {                                                                                "caching:None;disk_name:bosh-disk-data-0a8f47bc-e934-4a7f-90c5-ec775d5b71ae;resource_group_name:rg-sample-azure-dev1": {           "path": "/dev/sdd",                                                                                "lun": "1",                                                                                "host_device_id": "{f8b3781b-1e82-4818-a1c3-63d806ec15bb}"                                                                 }                                                                                },                                                                                "ephemeral": {                                                                                "path": "/dev/sdc",                                                                                "lun": "0",                                                                                "host_device_id": "{f8b3781b-1e82-4818-a1c3-63d806ec15bb}"                                                                 },                                                                                "system": "/dev/sda"                                                                                },

@sauravmndl From /var/vcap/bosh/setings.json, it seems like that Azure CPI did have attached the data disk successfully.

Did you check whether the disk is attached to the VM in Azure portal?
Could you see sdd by sudo fdisk -l inside the VM?

If both are yes, maybe bosh-agent failed to format the new disk. You should be able to find some errors in bosh-agent's logs /var/vcap/bosh/log/* inside the VM.

@AbelHu  Did you check whether the disk is attached to the VM in Azure portal? ===> Not attached
Could you see sdd by sudo fdisk -l inside the VM? ==> sdd is not listed
I have observed the behaviour: The disk gets attached for some moment and doesn't get mounted. Then detached. Then shows the error --> volume not found.

I see the issue is to some extent similar to https://github.com/cloudfoundry-incubator/bosh-azure-cpi-release/issues/227
The vm type I am using is: D2S_V3
And bosh stemcell is : bosh-azure-hyperv-ubuntu-trusty-go_agent  3468

@sauravmndl It seems like to be an infrastructure issue. I suggest you to file a support ticket with the log of the task to attach the disk in Azure portal.

@AbelHu  Thanks for  the help above. Will create a support request.

Reopening it as seems to be bosh issue. Creating a ticket also and will update here.

Same as https://github.com/cloudfoundry/bosh/issues/1857

Closing as duplicate of resolved #1857.


Kernel 5.10

This kernel requires fewer patches with better upstream support, and has better supports KMS. More information here: https://www.raspberrypi.org/forums/viewtopic.php?t=288234
Additionally, we've had users request support for this kernel: https://github.com/balena-os/balena-raspberrypi/issues/601
This is a WIP branch to add support for kernel 5.10 (derived from upstream meta-raspberrypi) to balena-raspberrypi.

Can one of the admins verify this patch?

@resin-jenkins add to whitelist

There are a few issues with this PR as it stands, and I wanted to get some feedback on them now that it builds.
First, the kernel doesn't boot. U-boot runs and loads the kernel, but the boot stops at Starting kernel..., there's no LED status light indicating that the kernel is running, and it never comes on the network. I thought it might have been a firmware incompatibility issue, so I updated to the version from meta-raspberrypi to no effect.
U-Boot 2020.07 (Jul 18 2020 - 15:34:49 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In:    serial
Out:   erial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
No working controllers founderror -110
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
516 bytes rea in 11 ms (44.9 KiB/s)
## Executing script at 02400000
Scanning mmc usb devices 0 1 2
24 bytes read in 14 ms (1000 Bytes/s)
Found esin image on mmc 0
Loading resinOS_uEnv.txt from mmc device 0 partition 1
Failed to load 'resinOS_uEnv.txt'
Loading extrauEnv.txt from mmc device 0 partition 1
0 bytes read in 12 ms (0 Bytes/s)
Import extra_uEnv.txt in environment
Loading botcount.env from mmc device 0 partition 1
Failed to load 'bootcount.env'
No bootcount.env file. Setting bootcount=0 in evironment
12759288 bytes read in 569 ms (21.4 MiB/s)
Uncompressed size: 32451072 = 0x1EF2A00
Moving Image from 0x80000 to 0x20000, end=2230000
## Flattened Device Tree blob at 2eff3b00
   Booting using the fdt blob at 0x2eff3b00
   Using Device Tree n place at 000000002eff3b00, end 000000002f002fa8

Starting kernel ...

Additionally, the build fails on kernel-module-headers with: gen_mod_headers: Missing kernel configuration. Specifying BALENA_DISABLE_KERNEL_HEADERS=1 works around this, but I'd like to understand what's missing here.

We'll probably also want to drop the commit making kernel 5.10 the default before merging this, and then I'm not sure how or at what point we want to distribute this kernel.

@jakogut thanks for working on this. I am the one that requested this :)
Are there any blockers to progress with the PR or just waiting for some feedback from the maintainers?

@balena-ci rebase

@jakogut You're seeing some weird behaviour because you-re missing inherit kernel-resin in your kernel recipe.

I pushed some suggested changes in a new commit. Feel free to use it, or drop it, I won't be offended either way :D
Changes:

copied linux-raspberrypi.inc and linux-raspberrypi_5.10.bb from meta-raspberrypi master without modifications
created linux-raspberrypi_5.10.bbappend similar 5.4 but with your updated patch list
copied vc4graphics.cfg and initramfs-image-bundle.cfg files from meta-raspberrypi master

I just noticed last compile failed (cp: cannot stat 'arch/arm/kernel/module.lds': No such file or directory) so there's still some work to do and it's obviously untested. Just throwing this out there in case it helps unblock you!

Thanks for the assist, @klutchell! This certainly helps.

@balena-ci rebase

@balena-ci rebase

I tried this on a CM4 (MACHINE=raspberrypicm4-ioboard) and it's not booting, just stops after u-boot
U-Boot 2021.04-rc3 (Mar 03 2021 - 15:10:34 +0000)

DRAM:  2 GiB
RPI Compute Module 4 (0xb03140)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In:    serial
Err:   serial
Net:   Could not get PHY for mdio@e14: addr 1
No ethernet found.

PCIe BRCM: link down
starting USB...
No working controllers foun
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
516 bytes read in 10 ms (49.8 KiB/s)
## Excuting script at 02400000
Scanning mmc usb devices 0 1 2
24 bytes read in 8 ms (2.9 KiB/s)
Found resin image on mmc 0
Loading resinOS_uEnvtxt from mmc device 0 partition 1
Failed to load 'resinOS_uEnv.txt'
Loading extra_uEnv.txt from mmc device 0 partition 1
0 bytes read i 11 ms (0 Bytes/s)
Import extra_uEnv.txt in environment
Loading bootcount.env from mmc device 0 partition 1
Failed to load 'bootount.env'
No bootcount.env file. Setting bootcount=0 in environment
14001486 bytes read in 625 ms (21.4 MiB/s)
Uncompressed size: 3528108 = 0x21A5A00
Moving Image from 0x80000 to 0x200000, end=24f0000
## Flattened Device Tree blob at 2eff6f00
   Booting using th fdt blob at 0x2eff6f00
   Using Device Tree in place at 000000002eff6f00, end 000000002f002fb6

Starting kernel ...

I expect the booting issue is a firmware issue, on other images I've updated the firmware files and it fixes the boot issue.  However, what concerns me is the failure to load the ethernet
Net:   Could not get PHY for mdio@e14: addr 1
Any ideas on where to start?  I've built other meta-raspberrypi images for the CM4 using the same u-boot version and don't experience the issue.  I'm having a difficult time figuring out where it's coming from.
Thanks.

@jakogut could you please take a look to @smithandrewc comment above? Thanks!

Copying in a working bcm2711-rpi-cm4.dtb fixed the ethernet/mdio problem.
Copying in working firmware files (fixup4cd.dat, fixup4.dat, fixup4x.dat, start4cd.elf, start4.elf, start4x.elf) fixes the booting problem.

@smithandrewc Awesome, glad to hear you got it working! I'll look at updating this PR with those changes.

@balena-ci rebase

I recompiled this and I'm still get the same error Net: Could not get PHY for mdio@e14: addr 1 unless I overwrite the bcm2711-rpi-cm4.dtb file.  Comparing the two, the only difference I see are all the phandle are missing and
__symbols__ {
...
};

By copying over the __symbols__ section from the working one to the one compiled here, it boots up properly.  Is the device-tree compiler step being changed in one of the balena recipes?  My "working" dtb comes from a yocto compiled image from meta-raspberrypi so upstream seems to be correct.

@alexgg I'm seeing a weird build failure with kernel-headers-test exiting with status 2.
ERROR: kernel-headers-test-1.0-r0 do_compile: Execution of '/home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/temp/run.do_compile.308038' failed with exit code 2:
The command '/bin/sh -c ARCH=${kernel_arch} CROSS_COMPILE=${cross_compile_prefix}  make -C /usr/src/app/kernel_source/*/build M=/usr/src/app/example_module_headers_src && echo "compiled hello module with header sources"' returned a non-zero code: 2
WARNING: /home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/temp/run.do_compile.308038:1 exit 2 from 'IMAGE_ID=$(DOCKER_API_VERSION=1.22 docker build --build-arg kernel_arch=arm64 --build-arg cross_compile_prefix=aarch64-linux-gnu- /home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/kernel-headers-test-1.0/work)'

ERROR: Logfile of failure stored in: /home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/temp/log.do_compile.308038
Log data follows:
| DEBUG: Executing shell function do_compile
| The command '/bin/sh -c ARCH=${kernel_arch} CROSS_COMPILE=${cross_compile_prefix}  make -C /usr/src/app/kernel_source/*/build M=/usr/src/app/example_module_headers_src && echo "compiled hello module with header sources"' returned a non-zero code: 2
| WARNING: /home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/temp/run.do_compile.308038:1 exit 2 from 'IMAGE_ID=$(DOCKER_API_VERSION=1.22 docker build --build-arg kernel_arch=arm64 --build-arg cross_compile_prefix=aarch64-linux-gnu- /home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/kernel-headers-test-1.0/work)'
| ERROR: Execution of '/home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/temp/run.do_compile.308038' failed with exit code 2:
| The command '/bin/sh -c ARCH=${kernel_arch} CROSS_COMPILE=${cross_compile_prefix}  make -C /usr/src/app/kernel_source/*/build M=/usr/src/app/example_module_headers_src && echo "compiled hello module with header sources"' returned a non-zero code: 2
| WARNING: /home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/temp/run.do_compile.308038:1 exit 2 from 'IMAGE_ID=$(DOCKER_API_VERSION=1.22 docker build --build-arg kernel_arch=arm64 --build-arg cross_compile_prefix=aarch64-linux-gnu- /home/build/build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/kernel-headers-test-1.0/work)'
|
ERROR: Task (/home/build/build/../layers/meta-balena/meta-balena-common/recipes-kernel/linux/kernel-headers-test.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 4222 tasks of which 3774 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/build/build/../layers/meta-balena/meta-balena-common/recipes-kernel/linux/kernel-headers-test.bb:do_compile
Summary: There were 5 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
[000000550][LOG]Build for raspberrypi4-64 failed. Check failed log in build/tmp/log/cooker/raspberrypi4-64 .
[000000550][LOG]If build for raspberrypi4-64 succeeded, final image should have been generated here:
[000000550][LOG]   build/tmp/deploy/images/raspberrypi4-64/balena-image-raspberrypi4-64.balenaos-img
[000000550][LOG]Done.

I tested out this image build outside of Yocto to see if we could get a more descriptive error message:
┌─[✗]─[13:17:57]─[joseph@wash]
└──> balena-raspberrypi $ >> docker build --build-arg kernel_arch=arm64 --build-arg cross_compile_prefix=aarch64-linux-gnu- build/tmp/work/aarch64-poky-linux/kernel-headers-test/1.0-r0/kernel-headers-test-1.0/work/
Sending build context to Docker daemon  25.61MB
Step 1/14 : FROM balenalib/intel-nuc-debian:stretch-20190717
 ---> 6d11514990a5
Step 2/14 : RUN apt-get update && apt-get install -y curl wget build-essential libelf-dev bc flex libssl-dev bison gcc-arm-linux-gnueabi gcc-aarch64-linux-gnu python-minimal
 ---> Using cache
 ---> 3398de266875
Step 3/14 : ADD example_module /usr/src/app/example_module_headers_src/
 ---> Using cache
 ---> 870c80bc34fe
Step 4/14 : ADD example_module /usr/src/app/example_module_headers_built/
 ---> Using cache
 ---> d2aceabf498d
Step 5/14 : ADD kernel_source.tar.gz /usr/src/app/kernel_source
 ---> 9042e734d73f
Step 6/14 : ADD kernel_modules_headers.tar.gz /usr/src/app/
 ---> c6e76266a818
Step 7/14 : WORKDIR /usr/src/app
 ---> Running in eb5b27468d71
Removing intermediate container eb5b27468d71
 ---> bdf2c11b779e
Step 8/14 : RUN if [ -f /lib64/ld-linux-x86-64.so.2 ]; then 	if [ ! -f /lib/ld-linux-x86-64.so.2 ]; then 		ln -s /lib64/ld-linux-x86-64.so.2  /lib/ld-linux-x86-64.so.2 ; 	fi;     fi;
 ---> Running in 01c677c29905
Removing intermediate container 01c677c29905
 ---> f8e5ab9bcc66
Step 9/14 : ARG kernel_arch
 ---> Running in 23f64c5d6df5
Removing intermediate container 23f64c5d6df5
 ---> 60807e8e04f3
Step 10/14 : ARG cross_compile_prefix
 ---> Running in cb59b4aac77c
Removing intermediate container cb59b4aac77c
 ---> ebb38a58d451
Step 11/14 : RUN ARCH=${kernel_arch} CROSS_COMPILE=${cross_compile_prefix} make -C /usr/src/app/kernel_source/*/build modules_prepare
 ---> Running in c51c8ffffdb3
make: Entering directory '/usr/src/app/kernel_source/5.10.31-v8/build'
  SYNC    include/config/auto.conf.cmd
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
  HOSTCC  scripts/dtc/dtc.o
  HOSTCC  scripts/dtc/flattree.o
  HOSTCC  scripts/dtc/fstree.o
  HOSTCC  scripts/dtc/data.o
  HOSTCC  scripts/dtc/livetree.o
  HOSTCC  scripts/dtc/treesource.o
  HOSTCC  scripts/dtc/srcpos.o
  HOSTCC  scripts/dtc/checks.o
  HOSTCC  scripts/dtc/util.o
  LEX     scripts/dtc/dtc-lexer.lex.c
  YACC    scripts/dtc/dtc-parser.tab.[ch]
  HOSTCC  scripts/dtc/dtc-lexer.lex.o
  HOSTCC  scripts/dtc/dtc-parser.tab.o
  HOSTLD  scripts/dtc/dtc
  HOSTCC  scripts/genksyms/genksyms.o
  YACC    scripts/genksyms/parse.tab.[ch]
  HOSTCC  scripts/genksyms/parse.tab.o
  LEX     scripts/genksyms/lex.lex.c
  HOSTCC  scripts/genksyms/lex.lex.o
  HOSTLD  scripts/genksyms/genksyms
  HOSTCC  scripts/kallsyms
  HOSTCC  scripts/recordmcount
  HOSTCC  scripts/sorttable
  HOSTCC  scripts/asn1_compiler
  HOSTCC  scripts/extract-cert
  WRAP    arch/arm64/include/generated/uapi/asm/kvm_para.h
  WRAP    arch/arm64/include/generated/uapi/asm/errno.h
  WRAP    arch/arm64/include/generated/uapi/asm/ioctl.h
  WRAP    arch/arm64/include/generated/uapi/asm/ioctls.h
  WRAP    arch/arm64/include/generated/uapi/asm/ipcbuf.h
  WRAP    arch/arm64/include/generated/uapi/asm/msgbuf.h
  WRAP    arch/arm64/include/generated/uapi/asm/poll.h
  WRAP    arch/arm64/include/generated/uapi/asm/resource.h
  WRAP    arch/arm64/include/generated/uapi/asm/sembuf.h
  WRAP    arch/arm64/include/generated/uapi/asm/shmbuf.h
  WRAP    arch/arm64/include/generated/uapi/asm/siginfo.h
  WRAP    arch/arm64/include/generated/uapi/asm/socket.h
  WRAP    arch/arm64/include/generated/uapi/asm/sockios.h
  WRAP    arch/arm64/include/generated/uapi/asm/stat.h
  WRAP    arch/arm64/include/generated/uapi/asm/swab.h
  WRAP    arch/arm64/include/generated/uapi/asm/termbits.h
  WRAP    arch/arm64/include/generated/uapi/asm/termios.h
  WRAP    arch/arm64/include/generated/uapi/asm/types.h
  WRAP    arch/arm64/include/generated/asm/early_ioremap.h
  WRAP    arch/arm64/include/generated/asm/mcs_spinlock.h
  WRAP    arch/arm64/include/generated/asm/qrwlock.h
  WRAP    arch/arm64/include/generated/asm/qspinlock.h
  WRAP    arch/arm64/include/generated/asm/set_memory.h
  WRAP    arch/arm64/include/generated/asm/user.h
  WRAP    arch/arm64/include/generated/asm/bugs.h
  WRAP    arch/arm64/include/generated/asm/delay.h
  WRAP    arch/arm64/include/generated/asm/div64.h
  WRAP    arch/arm64/include/generated/asm/dma-mapping.h
  WRAP    arch/arm64/include/generated/asm/dma.h
  WRAP    arch/arm64/include/generated/asm/emergency-restart.h
  WRAP    arch/arm64/include/generated/asm/hw_irq.h
  WRAP    arch/arm64/include/generated/asm/irq_regs.h
  WRAP    arch/arm64/include/generated/asm/kdebug.h
  WRAP    arch/arm64/include/generated/asm/kmap_types.h
  WRAP    arch/arm64/include/generated/asm/local.h
  WRAP    arch/arm64/include/generated/asm/local64.h
  WRAP    arch/arm64/include/generated/asm/mm-arch-hooks.h
  WRAP    arch/arm64/include/generated/asm/mmiowb.h
  WRAP    arch/arm64/include/generated/asm/msi.h
  WRAP    arch/arm64/include/generated/asm/rwonce.h
  WRAP    arch/arm64/include/generated/asm/serial.h
  WRAP    arch/arm64/include/generated/asm/switch_to.h
  WRAP    arch/arm64/include/generated/asm/trace_clock.h
  WRAP    arch/arm64/include/generated/asm/unaligned.h
  WRAP    arch/arm64/include/generated/asm/vga.h
  CC      scripts/mod/empty.o
  HOSTCC  scripts/mod/mk_elfconfig
  MKELF   scripts/mod/elfconfig.h
  HOSTCC  scripts/mod/modpost.o
  CC      scripts/mod/devicetable-offsets.s
  UPD     scripts/mod/devicetable-offsets.h
  HOSTCC  scripts/mod/file2alias.o
  HOSTCC  scripts/mod/sumversion.o
  HOSTLD  scripts/mod/modpost
scripts/Makefile.build:414: warning: overriding recipe for target 'modules.order'
Makefile:1424: warning: ignoring old recipe for target 'modules.order'
  CC      arch/arm64/kernel/vdso/vgettimeofday.o
  AS      arch/arm64/kernel/vdso/note.o
  AS      arch/arm64/kernel/vdso/sigreturn.o
  LD      arch/arm64/kernel/vdso/vdso.so.dbg
  VDSOSYM include/generated/vdso-offsets.h
  LDS     scripts/module.lds
make: Leaving directory '/usr/src/app/kernel_source/5.10.31-v8/build'
Removing intermediate container c51c8ffffdb3
 ---> 402b9c2a1e1c
Step 12/14 : RUN ARCH=${kernel_arch} CROSS_COMPILE=${cross_compile_prefix} KERNEL_TREE_PATH=/usr/src/app/kernel_source/*/build/ MODULE_PATH=/usr/src/app/example_module_headers_src  make -C /usr/src/app/example_module_headers_src
 ---> Running in a0b9813a14ce
make: Entering directory '/usr/src/app/example_module_headers_src'
make -C /usr/src/app/kernel_source/*/build/ M=/usr/src/app/example_module_headers_src modules
make[1]: Entering directory '/usr/src/app/kernel_source/5.10.31-v8/build'
  CC [M]  /usr/src/app/example_module_headers_src/hello.o
  MODPOST /usr/src/app/example_module_headers_src/Module.symvers
WARNING: modpost: missing MODULE_LICENSE() in /usr/src/app/example_module_headers_src/hello.o
ERROR: modpost: "_mcount" [/usr/src/app/example_module_headers_src/hello.ko] undefined!
scripts/Makefile.modpost:111: recipe for target '/usr/src/app/example_module_headers_src/Module.symvers' failed
make[2]: *** [/usr/src/app/example_module_headers_src/Module.symvers] Error 1
make[2]: *** Deleting file '/usr/src/app/example_module_headers_src/Module.symvers'
Makefile:1728: recipe for target 'modules' failed
make[1]: Leaving directory '/usr/src/app/kernel_source/5.10.31-v8/build'
make[1]: *** [modules] Error 2
Makefile:10: recipe for target 'hello.ko' failed
make: Leaving directory '/usr/src/app/example_module_headers_src'
make: *** [hello.ko] Error 2
The command '/bin/sh -c ARCH=${kernel_arch} CROSS_COMPILE=${cross_compile_prefix} KERNEL_TREE_PATH=/usr/src/app/kernel_source/*/build/ MODULE_PATH=/usr/src/app/example_module_headers_src  make -C /usr/src/app/example_module_headers_src' returned a non-zero code: 2

The interesting line is:
ERROR: modpost: "_mcount" [/usr/src/app/example_module_headers_src/hello.ko] undefined!
This looks like a compiler incompatibility to me, so I bumped the kernel to 5.10 for balena-generic as well, and there's no such error. However, bumping the base image in meta-balena-common/recipes-kernel/linux/files/Dockerfile from balenalib/intel-nuc-debian:stretch-20190717 to balenalib/intel-nuc-debian:buster-20210225 does resolve the issue. However, as commented in that file, we're using an older Debian release because some devices with kernel v3.10 don't build with GCC 8 from Buster.

Yep, I changed the image-balena.bbclass to get it to compile
diff --git a/meta-balena-common/classes/image-balena.bbclass b/meta-balena-common/classes/image-balena.bbclass
index 9f081830..919e6d13 100644
--- a/meta-balena-common/classes/image-balena.bbclass
+++ b/meta-balena-common/classes/image-balena.bbclass
@@ -6,7 +6,8 @@ inherit image_types_balena

 # When building a Balena OS image, we also generate the kernel modules headers
 # and ship them in the deploy directory for out-of-tree kernel modules build
-DEPENDS += "coreutils-native jq-native ${@bb.utils.contains('BALENA_DISABLE_KERNEL_HEADERS', '1', '', 'kernel-modules-headers kernel-devsrc kernel-headers-test', d)}"
+#DEPENDS += "coreutils-native jq-native ${@bb.utils.contains('BALENA_DISABLE_KERNEL_HEADERS', '1', '', 'kernel-modules-headers kernel-devsrc kernel-headers-test', d)}"
+DEPENDS += "coreutils-native jq-native ${@bb.utils.contains('BALENA_DISABLE_KERNEL_HEADERS', '1', '', 'kernel-modules-headers kernel-devsrc', d)}"

Probably not an option for you, just how I got around it for my work.

Thanks for the confirmation, @smithandrewc. You shouldn't need to comment out that line, instead, you should be able to disable building kernel headers with the BALENA_DISABLE_KERNEL_HEADERS=1 config, which can be passed to barys with the -a flag.
I've cherry picked the last commit from https://github.com/balena-os/balena-raspberrypi/pull/668. The first commit removing the 2xmcp2517fd.dtbo overlay was fixed by applying a patch that was being ignored. I'll also apply the change adding the PCIe overlay.

@jakogut see https://www.linuxjournal.com/content/simplifying-function-tracing-modern-gcc. As Debian Stretch is using gcc 4.6 I reckon we could change -pg for -mfentry when building the module.

Superseded by https://github.com/balena-os/balena-raspberrypi/pull/684


change gremlin-server version for vagrant to 3.0.2, added composer.lock

@chrismichaels84 can you just double check the sanity of the vagrant changes? I can't really test vagrant in my env yet (will try to get it up and running)

Or @smolinari if you get a chance to test vagrant on this.

@smolinari does this work in the end? can I merge this branch or do changes need to be made?

@PommeVerte - yes. It is working fine. I think the issue I had yesterday afternoon was caused by a flaky internet connection and somehow screwed up the Gremlin install. This morning all worked as it should on the Gremlin/Neo4j side.
Scott

ok awesome. adding this then :)

Theoretically, if Chris accepts my changes (plus some needed cleanup work), then we'd also have ODB up to 2.1.6 too.
Scott


QB: Incorrect URL is created when using recipient-country with transaction output

QB Selections:
Recipient Country = Nigeria
Output Format = Each Financial Transaction
The resulting URL is: https://iati.cloud/search/transaction?q=recipient_country_code:(NG)&wt=xslt&tr=transaction-csv.xsl&rows=5000000
The resulting URL should be: https://iati.cloud/search/transaction?q=transaction_recipient_country_code:(NG)&wt=json&rows=5000000
This should work in the same way that it does for the Sector (as Sector can also be reported at either Transaction OR Activity level).
When using the Sector filter, the resulting URL changes depending on which Output Format you select.
When you select Each Activity: https://iati.cloud/search/activity?q=sector_code:("11110")&wt=json&rows=5000000
When you select Each Transaction: https://iati.cloud/search/transaction?q=transaction_sector_code:("11110")&wt=json&rows=5000000

Tested - the query URL is changing depending on the chosen row-output now, as expected :)


Select peripheral component values for LMR14050 buck regulator

Requirements:

V_in = ~20V
V_out = 5.1V
I_out = 3A

Data sheet: https://www.ti.com/lit/ds/symlink/lmr14050.pdf
TODO

Feedback resistor network

[x] R_FBT = 100k digikey
[x] R_FBB = 17.4k digikey

[x] R_T = 84.5k digikey
[x] C_in digikey x2 in series (Also add a generic 100nF decoupling capacitor with appropriate voltage range close the the chip power pin.)
[x] C_out digikey x3 in series
[x] C_boot = 0.1uF digikey
[x] C_ss = 22 nF digikey
[x] L = 8.2uH digikey
[x] D digikey

IO connectors

[x] V_out to raspi digikey
[x] V_in to Laptop Charger digikey

Protection circuitry
Reverse polarity protection using a P-MOSFET

[x] P-FET for reverse polarity protection digikey
[x] 10V zener digikey
A generic 100k 0603 resistor

Notes
By using feedback resistor values given in the data sheet example, we get a bit over 5V output at 5A. It is recommended
to use 1% tolerance or better and temperature coefficient of 100 ppm or lower divider resistors.
The switching frequency should not matter at the moment, so we can also use the values for 300kHz from the example application. This can be changed in the future if EMC issues arise.
Soft start time does not seem important for this application, using example values.

The design has been finished and documented to this issue.


obscure OCR after fine tuning?

Dear reader, I'm trying to improve the deu.traineddata using the Bembo font.

That looks a bit like rot13 but not quite ;) I suspect the unicharset got corrupted when combining the traineddata files.
Since you aren't actually using tesstrain but tesseract's training tools directly, you might want to ask on the tesseract mailing list.

Is font "fine tuning" possible with tesstrain.sh and nothing else?

Fine tuning uses lstmtraining. And that is used by the tesstrain.sh or the replacement tesstrain.py scripts (both part of the tesseract repository), and it is also used by the Makefile of the tesstrain repository. All of them support fine tuning. And of course you can write your own wrappers.
I usually use the tesstrain Makefile for fine tuning.

Output looks similar to that reported in Tesseract issue https://github.com/tesseract-ocr/tesseract/issues/1603

tesseract-ocr/tesseract#1603 looks like real OCR issues, whereas the output by @jbarth-ubhd has a 1:1 mapping to the correct prediction. I.e. 8 is consistently i etc. This could probably fixed with a tr(1) call, so I think the character set got jumbled in some weird way.

That could be fixed by replacing the unicharset component in the model file. It looks like somewhere in Jochens workflow a wrong unicharset was used.

Thanks for the hint @stweil. Following should fix it (haven't tested it though)
lstmtraining --stop_training \
  --continue_from ~/tesstutorial/bembo_from_full/bembo_checkpoint \
  --traineddata tessdata/best/deu.traineddata \
  --model_output ~/tesstutorial/bembo_from_full/deu.traineddata

--traineddata should be same as used in lstmtraining command. The unicharset is used from that.

helps. PS: Did copy the commands from https://tesseract-ocr.github.io/tessdoc/TrainingTesseract-4.00 with this modification: impact -> bembo, eng -> deu.

Documentation will need to be updated.

@jbarth-ubhd
You can create single line png and gt.txt files from lstmf files using unpack feature (see https://github.com/tesseract-ocr/tesseract/issues/2669#issuecomment-569782835) from @stweil's repo. Those files can then be used as input for tesstrain.


added finding nemo movie quote

Added Finding Nemo 2003 movie quote.
removed trailing empty spaces.

Thanks for merging my pull request! I will delete it.


install failed

Bug Description
ons last.
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.6.8 TARGET_ARCH=x86_64 sh -
durning
[root@k8s-master01 istio-1.6.8]# kubectl apply -f samples/addons
serviceaccount/grafana created
configmap/grafana created
service/grafana created
deployment.apps/grafana created
configmap/istio-grafana-dashboards created
configmap/istio-services-grafana-dashboards created
deployment.apps/jaeger created
service/tracing created
service/zipkin created
clusterrole.rbac.authorization.k8s.io/kiali-operator created
serviceaccount/kiali-operator created
clusterrolebinding.rbac.authorization.k8s.io/kiali-operator created
deployment.apps/kiali-operator created
serviceaccount/prometheus created
configmap/prometheus created
service/prometheus created
deployment.apps/prometheus created
unable to recognize "samples/addons/kiali.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "samples/addons/kiali.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "samples/addons/kiali.yaml": no matches for kind "Kiali" in version "kiali.io/v1alpha1"
unable to recognize "samples/addons/prometheus.yaml": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "samples/addons/prometheus.yaml": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
[root@k8s-master01 istio-1.6.8]# kubectl rollout status deployment/kiali -n istio-system
Error from server (NotFound): deployments.apps "kiali" not found

Version
[root@k8s-master01 istio-1.6.8]# istioctl version
client version: 1.12.2
control plane version: 1.12.2
data plane version: 1.12.2 (9 proxies)
[root@k8s-master01 istio-1.6.8]# kubectl version --short
Client Version: v1.22.0
Server Version: v1.22.0

Additional Information
[root@k8s-master01 istio-1.6.8]# istioctl bug-report
Target cluster context: kubernetes-admin@kubernetes
Running with the following config:
istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: {  }
exclude: { Namespaces: kube-system,kube-public,kube-node-lease,local-path-storage }
end-time: 2022-02-20 00:12:41.362377288 +0800 CST
Cluster endpoint: https://apiserver.cluster.local:6443
CLI version:
version.BuildInfo{Version:"1.12.2", GitRevision:"af0d66fd0aa363e9a7b0164f3a94ba36252fe60f", GolangVersion:"go1.17.5", BuildStatus:"Clean", GitTag:"1.12.2"}
The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
{
Component: "pilot",
Info:      version.BuildInfo{Version:"1.12.2", GitRevision:"af0d66fd0aa363e9a7b0164f3a94ba36252fe60f", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.12.2"},
},
}
The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.12.2}
Fetching proxy logs for the following containers:
dapr-system//dapr-placement-server-0/dapr-placement-server
dapr-system/dapr-dashboard/dapr-dashboard-8664d5c45f-sxdlf/dapr-dashboard
dapr-system/dapr-operator/dapr-operator-58b9d5fd59-tcbmj/dapr-operator
dapr-system/dapr-sentry/dapr-sentry-858fddc4f7-t2g28/dapr-sentry
dapr-system/dapr-sidecar-injector/dapr-sidecar-injector-7497b7945-59xvz/dapr-sidecar-injector
default/details-v1/details-v1-79c697d759-6fqkc/details
default/details-v1/details-v1-79c697d759-6fqkc/istio-proxy
default/productpage-v1/productpage-v1-65576bb7bf-8h5w5/istio-proxy
default/productpage-v1/productpage-v1-65576bb7bf-8h5w5/productpage
default/prometheus/prometheus-65f4d4f5bd-xhshv/istio-proxy
default/prometheus/prometheus-65f4d4f5bd-xhshv/prometheus-server
default/prometheus/prometheus-65f4d4f5bd-xhshv/prometheus-server-configmap-reload
default/ratings-v1/ratings-v1-7d99676f7f-stmv9/istio-proxy
default/ratings-v1/ratings-v1-7d99676f7f-stmv9/ratings
default/reviews-v1/reviews-v1-987d495c-zqjwq/istio-proxy
default/reviews-v1/reviews-v1-987d495c-zqjwq/reviews
default/reviews-v2/reviews-v2-6c5bf657cf-p5ns5/istio-proxy
default/reviews-v2/reviews-v2-6c5bf657cf-p5ns5/reviews
default/reviews-v3/reviews-v3-5f7b9f4f77-hr56j/istio-proxy
default/reviews-v3/reviews-v3-5f7b9f4f77-hr56j/reviews
istio-system/grafana/grafana-b7f8b6585-wwkcr/grafana
istio-system/istio-egressgateway/istio-egressgateway-c9cbbd99f-28qh6/istio-proxy
istio-system/istio-ingressgateway/istio-ingressgateway-7c8bc47b49-9hnsh/istio-proxy
istio-system/istiod/istiod-765596f7ff-6nrrz/discovery
istio-system/jaeger/jaeger-7bdfbf5977-trz9w/jaeger
istio-system/kiali-operator/kiali-operator-55c7bb84bb-rzh8v/operator
Fetching Istio control plane information from cluster.
Fetching Istio control plane information from cluster.
Running istio analyze on all namespaces and report as below:
Analysis Report:
Info [IST0102] (Namespace dapr-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace dapr-system istio-injection=enabled' to enable it, or 'kubectl label namespace dapr-system istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0118] (Service dapr-system/dapr-api) Port name  (port: 80, targetPort: 6500) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service dapr-system/dapr-dashboard) Port name  (port: 8080, targetPort: 8080) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service dapr-system/dapr-placement-server) Port name api (port: 50005, targetPort: 50005) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service dapr-system/dapr-placement-server) Port name raft-node (port: 8201, targetPort: 8201) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service dapr-system/dapr-sentry) Port name  (port: 80, targetPort: 50001) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service dapr-system/dapr-webhook) Port name  (port: 443, targetPort: 19443) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service istio-system/grafana) Port name service (port: 3000, targetPort: 3000) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service istio-system/kiali-operator-metrics) Port name cr-metrics (port: 8686, targetPort: 8686) doesn't follow the naming convention of Istio port.
Creating an archive at /root/istio-1.6.8/bug-report.tar.gz.
Cleaning up temporary files in /tmp/bug-report.
Done.

Why are you setting ISTIO_VERSION=1.6.8? That would install 1.6 Istio which is many years old and won't run on modern Kubernetes deployment

I operated according to the command of the version specified on the official website. After changing to the new version, I confirmed that it was good to use.However, I also recommend that official documentation be updated with the latest version.
https://istio.io/latest/docs/setup/getting-started/

Oh it's an example. We should fix it to always use the latest version though

thanks


Cot 597

Jira link (if applicable)
https://tools.hmcts.net/jira/browse/COT-597
Change description

This is superseded by #1660