initial support for floating-point numbers

Support reading floating-point numbers, printing them and num->str
support.  Internally, the C float is stored in the high 32-bits of an
any variable.
Added core functions integer? and float?.
Still missing support of:

reading exponential notation (5.6E10, -7.8E-9)
comparison (=, <, <=, >, >=)
arithmetic (+, -, *, /)

NOTE: I wouldn't merge it as-is because floats are quite useless without the arithmetic and comparison. I'm not sure which approach should be taken:

Should + work both for int, float or mixed?
Should (=? 123 123.0) be true?
Should we introduce new functions float=? , float+ , etc?
Should we add int->float and round ceil floor ?

Re approach:
I think everyone will get it wrong all the time if + does not work for both ints and floats, so it should work for both. It should be optimized for using ints since that is the common case. If it cannot be done without losing performance for int-only operations, it might be a good idea to also have int+ etc. (By the way, integer? is a rather long name, Bone generally keeps names short, so int? would be better.)
int->float can then be trivially done in the prelude as | x (+ x 0.0), but I'm not even sure if this is ever needed; I can't remember ever needing it. round ceil floor on the other hand are pretty important. Maybe round should have the alias float->int?
Exponential notation can be done with strtod (from stdlib.h), so you don't have to write the code yourself. If you would like to do it yourself, I would suggest implementing it as a reader macro.

Re code:
bprintf("#{?}"); should be an error instead, I think - if this ever happens, this is an internal error of the interpreter, so it seems unusual if it just prints a value and continues. Even just calling abort() would probably be fine.
Everything else looks good.

How do you plan to implement / (division)? Will it always return a float? Return an int if all arguments are ints? Return an int whenever the result actually is an integer?

Hmmm. I think as follows:

/ always returns float
int/ accepts two int arguments and returns int (equivalent to Python3's // operator)
mod accepts two int arguments and returns int (modulo)

What do you think?

I think that I've been thinking about these kinds of questions for too long already. :)
The behaviour of mod is out of question, it should be as you describe.
Using / and int/ sounds okay in principle (except that int/ should also accept more than 2 arguments, because... why wouldn't it? The code for this even exists already). But I'm not entirely sure. Personally I would expect / to return an int if called with int arguments. This is also the behaviour in C, C++, Ruby, Emacs Lisp and Python2. It would also be consistent with the other subs, e.g. + returns an int if all its arguments are ints.
Python3 and Perl5 behave as you describe, so that seems to be considered acceptable, too.

I added commits adding arithmetic and comparison and round, floor, ceil, trunc (and float->int, int->float). I think this is what we discussed so far. I went with the C/C++/Ruby style for division - if you divide two (or more) ints you get integer division.
Should I refactor all the number-related tests from tests/base.bn to tests/numbers.bn ?

It's fine to keep the tests for numeric subs in tests/base.bn.
The code seems fine for now (I'll look in more detail into it later - I really need to get some sleep first). But it has quite a bit of potential for optimization - the calls to is_list_of_ints make us visit each list twice, which is not necessary. You don't need to change this, we can always optimize these things later, and I guess there are places with larger optimization potential in the evaluator anyway.

by the way, do you think this is ready for merging now?

I think it is ready for merging.
I still don't have parsing for exponent format (1.3E9, -4.5E-5). Can be added in the future. If we want to use strtof(3) we'll need some changes in chars2num.
Re: optimization: as far as I understand, for any argument you'll need to (a) check whether it is int or float, and then (b) add to the total integer value or to the total float value. You can do it in one loop but it'll still be 2N operations. I think I don't understand your optimization suggestion.

Okay, great.
Yes, that can wait.
The expensive part is not checking and adding, but following the cdr-pointers. (That's why working with linked lists is generally considered slow.)

Good point about the linked list traversal. So is this better (only one foreach)?
DEFSUB(fullplus) {
  int64_t ires = 0;
  float fres = 0.0;
  bool all_ints = true;
  foreach(n, args[0])
    switch (get_num_type(x)) {
    case t_num_int:
      if(all_ints)
        ires += any2int(n);
      else
        fres += any2int(n);
      break;
    case t_num_float:
      if(all_ints)
        fres = ires + any2float(n);
      else
        fres += any2float(n);
      all_ints = false;
      break;
    default:
      abort();
    }
  last_value = all_ints ? int2any(ires) : float2any(fres);
}

Basically it sums up ints in ires until a float is encountered. Then it switches over to fres and sums all the remaining as floats.

You can even avoid the conditional branch on the all_ints variable (though branch prediction might safe your day, so it might not make much of a difference). The following is untested code, but should illustrate the idea.
DEFSUB(fullplus) {
  int64_t ires = 0;  // as long as we encounter only ints, we operate in "int mode"
  foreach_cons(c, args[0]) {
    any n = far(c);
    switch(get_num_type(n)) {
    case t_num_int:
      ires += any2int(n);
      break;
    case t_num_float: {  // switching to "float mode"
      float fres = any2float(n)+(float)ires;
      foreach(x, fdr(c))
        switch(get_num_type(x)) {
        case t_num_int:
          fres += (float)any2int(x);
          break;
        case t_num_float:
          fres += any2float(x);
          break;
        default: abort();
        }
      last_value = float2any(fres);
      return;   // end of code for "float mode"
    }
    default: abort();
    }
  }
  last_value = int2any(ires);
}

I like it. The "float mode" can be even shorter with my anynum2float helper. I'll change the code (and solve the merge conflicts).

Great, thanks.

OK I changed the full arithmetic operations (still local).
Things get nastier in the logical operators, this is what I have:
any full_num_gtp_floats(float first, any xs) {
  float f = first;
  foreach(x, xs) {
    float g = anynum2float(x);
    if(f <= g)
      return BFALSE;
    f = g;
  }
  return BTRUE;
}

DEFSUB(full_num_gtp) {
  last_value = BTRUE;
  if(is_nil(args[0]))
    return;
  any first_n = far(args[0]);
  switch (get_num_type(first_n)) {
  case t_num_int: {
    int64_t n = any2int(first_n);
    foreach_cons(c, fdr(args[0])) {
      any x = far(c);
      switch (get_num_type(x)) {
      case t_num_int: {
        int64_t m = any2int(x);
        if(n <= m) {
          last_value = BFALSE;
          return;
        }
        n = m;
        break;
      }
      case t_num_float:
        last_value = full_num_gtp_floats(n, c);
        break;
      default:
        abort();
      }
    }
    break;
  }
  case t_num_float: {
    last_value = full_num_gtp_floats(any2float(first_n), fdr(args[0]));
    break;
  }
  default:
    abort();
  }
}

The same thing should duplicated for >,<,>=,<=. We can create a preprocessor macro for these, but that's ain't a beauty too...
Alternatively, we can choose to ignore the optimization on 0, 1 and >2 arguments and just implement it in prelude.bn:
(defsub (gtp . xs)
  ">? for 0, 1 or >2 arguments"
  (with loop (lambda (xs)
               (if (single? xs)
                 #t
                 (if (>? (car xs) (cadr xs))
                   (loop (cdr xs))))))
    (if (nil? xs)
      #t
      (loop xs))))

(of course the name should be _full>? and it should call _fast>? internally.)

The pragmatic answer is that not all operations have to optimized equally. It's not uncommon to do something like (apply + some-list), so it makes sense to optimize +. I think it's pretty rare to use >? with long lists. Sometimes it's convenient to use it with 3 arguments to check if some value is in a certain range, but 3 elements is not a long list. :) So if it gets to nasty, we don't have to do the fastest possible solution.
I agree that solving it with a preprocessor macro is not the best thing to do, because the uglyness makes people (including me) skip over the code most of the time instead of trying to understand it. Only code that with absolute certainty has no bugs is allowed to make people skip over it. ;)
Now choose whatever you think is most reasonable. :)
Stylistically, I don't like the first argument of full_num_gtp_floats, because that only makes sense in the very specific context the function is called here, it does not make much sense when trying to understand the function in isolation. I would recommend to avoid that argument and just do last_value = n + full_num_gtp_floats(c); in the caller.
The docstring ">? for 0, 1 or >2 arguments" is actually incorrect: It's also for 2 arguments, e.g. I might do (apply >? xs) where xs happens to be (1 2). The 2-argument optimization is only used when >? appears in the first element of an evaluated list because that's the only place the macro expansion is looking at.

I chose to implement _full=, _full> etc in bone. Cut a lot of C code, probably lost some performance but these are rarely used as we said before.

Great, I will merge this now. Just some minor things:

is_list_of_ints() is unused now and can be removed. Kind of my fault because I disabled compiler warnings for unused functions.
You omitted the check for invalid number types in the float mode of full_plus and friends; doesn't matter currently because it can't happen anyway, but if we later add e.g. bignums or rationals and forget to handle them in one of these places, they would be silently ignored. (But fortunatly we add testcases for everything. ;-))
I have to try it myself, but I think full_minus can be greatly simplified by writing it in a similar way as full_div.


Text not visible inside Table

Current Behavior
Text in Table not visible on https://docs.layer5.io/cloud/identity/users/default-permissions/

Expected Behavior
Should be visible
Screenshots/Logs

Environment

Host OS: Mac Linux Windows
Platform: Docker or Kubernetes
Meshery Server Version: stable-v
Meshery Client Version: stable-v

Contributor Guides and Resources

🛠 Meshery Build & Release Strategy
📚 Instructions for contributing to documentation

Meshery documentation site and source

🎨 Wireframes and designs for Layer5 site in Figma (open invite)
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack

Thank you! Help wanted. 👍

@Aviral0702 Sure, go ahead!

@Ashparshp Thanks! Starting to work on it.

Are you still working on this issue?

@Aviral0702 ready to let another contributor take a crack at this one or will you have this completed no later than today?


[Request] Force browser window into foreground when searching google

[x] I am on the latest Cerebro.app version

[x] I have searched the issues of this repo and believe that this is not a duplicate

OS version and name: elementary OS 5.0 Juno (64bit)

Cerebro.app version: cerebro-0.3.2-x86_64.AppImage

Browser version: Chromium 71.0.3578.80 Built on Ubuntu

Browser stays in background 🚫
REPRO

Open cerebro window
search google for "anyting"
Hit enter
minimize browser
Open cerebro window
search google for "anyting"
Hit enter

EXPECTED
Browser pops up on second google search
ACTUAL
Browser stays in background

I'm closing this issue as outdated. This has been fixed in newer versions (at least on Windows).
Take a look at the latest version and, if the problem persists, feel free to open a new issue.


[RFC] corrected example #1078

Issue #1078

You're right, we forgot to update the doc. Thanks for the fix!


Support cleanup ambient node level rules based on controller status

Please provide a description of this PR:
Refer to daemonset controller's status to determine whether we're in uninstall or upgrade/restart stage. Some background discussion is in #44208
With this change,

while istio-cni restarting or upgrading, no node level rule will be cleaned up, traffic will keep being redirected to ztunnel.
while uninstall istio or uninstall istio-cni daemonset, node level rules will be cleaned up.

Sorry I saw you mentioned this on the other PR and I should have commented before you spent time on it. My bad.
I don't feel very comfortable with this level of coupling. A container shouldn't be aware of its deployment method (even its own Pod - especially the higher level controller).
I would not be surprised if we end up with multiple daemonsets for ztunnel for upgrades. Or don't use daemonset, etc.

Thanks for your comments, @howardjohn.  It's ok for me, never mind.
My original idea is come from k8s finalizers, however, its typical usage is required another controller/operator to execute the finalizer routines,  it looks like a bit complex but clear role boundary.  So I come to the idea to let container be aware of its supervisor controller's status.
And any detail istio-cni or ztunnel upgrade plan?


Tree agglomeration methods

Create TreeSE equivalents for speedyseq::tip_glom and speedyseq::tree_glom, to agglomerate tree leaves that have a small (user-defined) distance in the tree. This could be a useful dimension reduction method.
See https://rdrr.io/github/mikemc/speedyseq/man/tree_glom.html
Not top priority.

Naming could follow the one suggested in #392
For instance,

tip_glom -> mergeFeaturesBySimilarity
tree_glom -> mergeFeaturesByTree

tree_glom() is similar to tip_glom(), except that it uses the tree directly, rather than as the basis for hierarchical clustering -> to assess whether both are needed.
Note that the TreeSummarizedExperiment package sometimes refers to the finest level features as (tree) leaves. Consider the naming on the same go.

From those 3 suggestions, "mergeFeaturesByCluster" is already supported in a level that we want to support this "standardization", i.e., creating wrapper for this might not give extra value.
tse <- addCluster(tse, ...)
tse <- agglomerateByVariable(tse, by = "rows", f = "cluster")

The others, no immediate need now I assume.


Document Docker

Add documentation and an example of a docker image for a Nit application

+1

+1


Can't disable devup with .env.local

Can't disable with local config:
cat .env.local
DEVUP_ENABLED=false

But devup is still enabled. But this disables devup:
export DEVUP_ENABLED=false

fixed: https://github.com/sergio-fry/devup/commit/f87c42498ab773808f198215ca111ece5eedeba8


Add Machogu Details to Team section

Added Machogu's details and links to Team section

Could you change the image dimensions to 600x600

Let us know if you need any assistance.

Hi @AbedM15, were you able to look at the comments above?

Hi @AbedM15, I have seen the changes, perhaps you could remove changes not related to this PR? Like these files

Also, you can look into fixing merge conflicts since your PR has one.

Funny photo! The PR looks good to me!


NetworkError when creating silences.

Hey thanks for the great project!
When i try to create silences in Firefox 74.0 (64-bit) i get "NetworkError when attempting to fetch resource.".
I expect to see the silence being created.
In Chrome Version 63.0.3239.108 (Official Build) (64-bit) i get "Failed to fetch"
Can someone please help ?

Browsers typically don’t return much detail in fetch errors.
Developer console in the browser should have more detailed logs of the error.

Sure, here i get the following:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at ‘https://URL/api/v2/silences’. (Reason: Credential is not supported if the CORS header ‘Access-Control-Allow-Origin’ is ‘*’).
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://URL/api/v2/silences. (Reason: CORS request did not succeed).

You need to disable CORS credentials, see
https://github.com/prymitive/karma/blob/v0.56/docs/CONFIGURATION.md#alertmanagers

Thanks a lot, that solved the issue


Scalability issues

See the related PR in tableschema-py https://github.com/frictionlessdata/tableschema-py/pull/254
I have scalability issues when trying to validate a datapackage which contains one resource with 397201 lines containing foreign keys.
I needed to split my 397k lines resource into many files to organize those by source.
I finally chose to use share schema and group notion to split into many resources.
Trying to validate all those lines brought many scalability issues in both tableschema and datapackage libraries.
First issue is about memory management.
Checking relations in a resource, makes the object not only to read the related resource but to hold the data in memory as it is kept as an object attribute.
This has the consequences to make the memory grow when checking relations of a large amount of resource.
I don't know why the relations data are kept in the object so I proposed in this PR a new method to clean the relations data drop_relations.
Once the memory issue solved I had a performance one. Checking relations of my +1000 resources hodling 397k lines of data took 98m49.895s.
That's very long.
So I thought about two optimizations.
The first one is to avoid loading the relations for each resources which belongs to a group.
To enable this optimization I propose to add a check_relations method into the Group object.
Thus we load relations data once and then use this data in memory to validate all the resource belonging to that group.
Than a second optimization has been proposed into tableschema-py  by the PR tableschema-py https://github.com/frictionlessdata/tableschema-py/pull/254.
The idea is to pre-index the relations data by the values of the foreign keys. This index is called foreign_keys_values. This index is then used to test if the row reference some of the existing value (simple hash map lookup).
To speed up, I also propose to expose the get_foreign_keys_values() method so that the Group object can pre-compute the index only once before using it to validate all the resource.
Using those optimizations made the validation process to drop from 98m49.895s to 1m3.609s.
I've just realized that this PR miss updates on the documentation.
Let's see if the principles are good by reviewers before updating the doc..
@roll what do you think ?

Oh yeah of course since this PR is based on an update of the tableschema-py dependency, CI failes.
This PR can be valid before the tablechema one is accepted and deployed...

@roll I just removed my try catch to let exceptions flow upstream. Maybe there is a policy to raise a datapackage exceptions and not tableschema ones ?


Documentation

Provide basic documentation on the intent, installation, and utilization of the package. Document the fields and data structures produced by histomics stream.

+1 for Sphinx / ReadTheDoc's

@thewtex Sphinx offers quite a bit of functionality and I am unsure how to prioritize.  Can you recommend a project similar enough to HistomicsStream in scope that is already using Sphinx and would make a good template?

Hi @Leengit , there are many examples, prehaps xarray-datatree. ReadTheDocs simplifies deployment. They say they have 80,000 open source project examples, too :-).


generate new soundcloud API key ?

I received this email from soundcloud, this morning:
Hello,We are reaching out today as you are the registered owner of a SoundCloud application that has been inactive for over 3 months. We’re writing to inform you that as of June 8, 2023, we are revoking access to your application in order to make necessary changes to our platform. As a result you will be unable to access SoundCloud’s API and any apps that use our APIs will no longer work.We understand that this may be disappointing; however, we need to do this for two reasons:Recently, we haven’t been able to provide developers with the level of support that you deserve. That includes being able to help you when things go wrong and supporting you when you build amazing things.More pressingly, we haven’t been able to sustain the level of oversight needed to make sure bad actors aren’t abusing our API. As such, change is necessary in order to protect the artists and fans that use SoundCloud’s services.Rest assured, this is not the final state of our API program - just a temporary reset. We’re working hard to reopen the doors as soon as possible so that you can use our API in the future.We’ll be releasing updates over the coming months, so please keep an eye on our Developer Blog and Twitter for updates to come.Amid these changes, we value your feedback. If you have notes or ideas for our developer platform, please feel free to leave comments on the form.
Hello,We are reaching out today as you are the registered owner of a SoundCloud application that has been inactive for over 3 months. We’re writing to inform you that as of June 8, 2023, we are revoking access to your application in order to make necessary changes to our platform. As a result you will be unable to access SoundCloud’s API and any apps that use our APIs will no longer work.We understand that this may be disappointing; however, we need to do this for two reasons:Recently, we haven’t been able to provide developers with the level of support that you deserve. That includes being able to help you when things go wrong and supporting you when you build amazing things.More pressingly, we haven’t been able to sustain the level of oversight needed to make sure bad actors aren’t abusing our API. As such, change is necessary in order to protect the artists and fans that use SoundCloud’s services.Rest assured, this is not the final state of our API program - just a temporary reset. We’re working hard to reopen the doors as soon as possible so that you can use our API in the future.We’ll be releasing updates over the coming months, so please keep an eye on our Developer Blog and Twitter for updates to come.Amid these changes, we value your feedback. If you have notes or ideas for our developer platform, please feel free to leave comments on the form.SOUNDCLOUD
Hello,
We are reaching out today as you are the registered owner of a SoundCloud application that has been inactive for over 3 months. We’re writing to inform you that as of June 8, 2023, we are revoking access to your application in order to make necessary changes to our platform. As a result you will be unable to access SoundCloud’s API and any apps that use our APIs will no longer work.
We understand that this may be disappointing; however, we need to do this for two reasons:
Recently, we haven’t been able to provide developers with the level of support that you deserve. That includes being able to help you when things go wrong and supporting you when you build amazing things.
More pressingly, we haven’t been able to sustain the level of oversight needed to make sure bad actors aren’t abusing our API. As such, change is necessary in order to protect the artists and fans that use SoundCloud’s services.
Rest assured, this is not the final state of our API program - just a temporary reset. We’re working hard to reopen the doors as soon as possible so that you can use our API in the future.
We’ll be releasing updates over the coming months, so please keep an eye on our Developer Blog and Twitter for updates to come.
Amid these changes, we value your feedback. If you have notes or ideas for our developer platform, please feel free to leave comments on the form.

The soundcloud developer account associated to the recipient email address only has 1 application:

client id: 94f7290349b7801c04969260c4433fed

Current situation:

On Openwhyd, Soundcloud tracks: can no longer be added, nor played. (e.g. Harissa - No Service)
I have no longer access to any Soundcloud API key, and it's no longer possible to ask for a new one
On Playemjs tests (make test-web), a few soundcloud tracks can (supposedly) play

Cf related openwhyd issues:

Study impact of Soundcloud API changes · Issue #465 · openwhyd/openwhyd
repair soundcloud integration · Issue #614 · openwhyd/openwhyd
Can't add Soundcloud tracks using bookmarklet · Issue #433 · openwhyd/openwhyd

=> Proposed plan:

get rid of logic that requires a Soundcloud API key / a client id
try to make as many features work as possible: playback, recognizing a track by URL, fetching metadata...


ci(concurrency)!: run a single CI workflow as required

Previous behavior:
Multiple Mainnet full syncs were able to run on the main branch at the
same time, and pushing multiple commits to the same branch would run
multiple CI workflows, when only the run from last commit was relevant
Expected behavior:
Ensure that only a single CI workflow runs at the same time in PRs.
The latest commit should cancel any previous running workflows from the
same PR.
Solution:
Use GitHub actions concurrency feature https://docs.github.com/en/actions/using-jobs/using-concurrency
Fixes https://github.com/ZcashFoundation/zebra/issues/4977
Fixes https://github.com/ZcashFoundation/zebra/issues/4857
Fixes https://github.com/ZcashFoundation/zebra/issues/1785
Review
@teor2345 can review this
Reviewer Checklist

[x] Re-run a workflow while running or push an empty commit
[ ] Tests for Expected Behaviour
[ ] Tests for Errors

I'm not sure if this PR solves ticket #4857

As a single PR could have several commits running (using GCP instances) at the same time + the main branch also having several commits running a full sync, we were having more than expected concurrent jobs in a busy week.
This will considerable reduce the concurrency + https://github.com/ZcashFoundation/zebra/issues/4976
And we can also increase the quota a 25% more.

Ensure that only a single CI workflow runs at the same time in PRs.
The latest commit should cancel any previous running workflows from the
same PR.

@gustavovalverde can you update the CI and CD docs with a short mention of the concurrency limits & cancellation behaviour?

@gustavovalverde should I mention in the changelog that this PR is a breaking change? I don't think it's a breaking change for users, that's why I'm not sure about it.

It's just for developers, not for users. It can be kept internal.


ConnectionString Implementation

Initial implementation for the Connection String URI parser. User-specified options are not validated, currently.

lgtm on my end. @alabid ?

Hey @steveklabnik, still interested in helping us review PRs for general rustyness?
This PR is an implementation of our (currently internal) spec for parsing connection strings, like "mongodb://localhost:27017" so we can determine which server to connect to.

LGTM. Good stuff.

@vkarpov15 yes, I am :) I will check the diff out.

Comments left! Overall, this code seems great, just a few things :)

Thanks for the feedback, @steveklabnik! I'll have a new PR out to get these changes in.


Assignment 2

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)
Added code, answers and comments
What did you learn from the changes you have made?
Learnt how to spot + and - associations and how to fit a multivariable linear regression model onto a dataset
Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?
no
Were there any challenges? If so, what issue(s) did you face? How did you overcome it?
yes, by reading the error messages and by looking up the internet
How were these changes tested?
by running the code in vs code editor
A reference to a related issue in your repository (if applicable)
none
Checklist

[x] I can confirm that my changes are working as intended

You have unfortunately made a pull request to the wrong repo. You shouldn't be making a pull request to UofT-DSI/applying_statistical_concepts, but rather your own forked applying_statistical_concepts repo. Please read the instructions on your onboarding Assignment Submission Guide more carefully.


Barcode ML kit not working on specific android device

Describe your issue. If applicable, add screenshots to help explain your problem.
In some android devices barcode scan is not working! (For now samsung galaxy tab a7 Android 12 and tab s9+ Android 13) There is no specific error just return empty barcode list. But there is specific log on devices which not working.
I/tflite  (23965): Initialized TensorFlow Lite runtime.
E/libc    (23965): Access denied finding property "ro.mediatek.platform"
I/tflite  (23965): Created TensorFlow Lite XNNPACK delegate for CPU.

This logs are not showing on working devices.
Here my camera controller how initialized
 var cameraController = CameraController(
    ultraWideCamera ?? backCameras.firstOrNull ?? cameras.first,
    ResolutionPreset.veryHigh,
    enableAudio: false,
    imageFormatGroup: Platform.isIOS ? ImageFormatGroup.bgra8888 : ImageFormatGroup.nv21,
  );

Steps to reproduce.
Run on specific devices ( now we only know Tab A7 & Tab s9)
What is the expected result?
Normal barcode scanning behaviout
Did you try our example app?
Yes
Is it reproducible in the example app?
Yes
Reproducible in which OS?
Android
Flutter/Dart Version?
[✓] Flutter (Channel stable, 3.22.1, on macOS 14.4 23E214 darwin-arm64, locale en-US)
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 15.4)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2023.3)
[✓] VS Code (version 1.89.1)
Plugin Version?
google_mlkit_barcode_scanning: ^0.12.0
camera: ^0.11.0

After I added this nv21 converting function it worked! So somehow in some android devices even you give NV21 image format to controller it is not returning nv21, or it is not proper nv21. But in somewhere we should get format exception.
Here my convert function
extension Nv21Converter on CameraImage {
  Uint8List getNv21Uint8List() {
    var width = this.width;
    var height = this.height;

    var yPlane = planes[0];
    var uPlane = planes[1];
    var vPlane = planes[2];

    var yBuffer = yPlane.bytes;
    var uBuffer = uPlane.bytes;
    var vBuffer = vPlane.bytes;

    var numPixels = (width * height * 1.5).toInt();
    var nv21 = List<int>.filled(numPixels, 0);

    // Full size Y channel and quarter size U+V channels.
    int idY = 0;
    int idUV = width * height;
    var uvWidth = width ~/ 2;
    var uvHeight = height ~/ 2;
    // Copy Y & UV channel.
    // NV21 format is expected to have YYYYVU packaging.
    // The U/V planes are guaranteed to have the same row stride and pixel stride.
    // getRowStride analogue??
    var uvRowStride = uPlane.bytesPerRow;
    // getPixelStride analogue
    var uvPixelStride = uPlane.bytesPerPixel ?? 0;
    var yRowStride = yPlane.bytesPerRow;
    var yPixelStride = yPlane.bytesPerPixel ?? 0;

    for (int y = 0; y < height; ++y) {
      var uvOffset = y * uvRowStride;
      var yOffset = y * yRowStride;

      for (int x = 0; x < width; ++x) {
        nv21[idY++] = yBuffer[yOffset + x * yPixelStride];

        if (y < uvHeight && x < uvWidth) {
          var bufferIndex = uvOffset + (x * uvPixelStride);
          //V channel
          nv21[idUV++] = vBuffer[bufferIndex];
          //V channel
          nv21[idUV++] = uBuffer[bufferIndex];
        }
      }
    }
    return Uint8List.fromList(nv21);
  }
}

I can contribute on this issue , can you please assign it to me @flutter-ml

@SuryaAbyss: Assigned to you. Thanks.

where did you used it because i m still getting error
InputImage? _inputImageFromCameraImage(CameraImage image) {
// get image rotation
// it is used in android to convert the InputImage from Dart to Java
// rotation is not used in iOS to convert the InputImage from Dart to Obj-C
// in both platforms rotation and camera.lensDirection can be used to compensate x and y coordinates on a canvas
final camera = widget.camera[0];
final sensorOrientation = camera.sensorOrientation;
InputImageRotation? rotation;
if (Platform.isIOS) {
rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
} else if (Platform.isAndroid) {
var rotationCompensation =
_orientations[_controller!.value.deviceOrientation];
if (rotationCompensation == null) return null;
if (camera.lensDirection == CameraLensDirection.front) {
// front-facing
rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
} else {
// back-facing
rotationCompensation =
(sensorOrientation - rotationCompensation + 360) % 360;
}
rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
}
if (rotation == null) return null;
// get image format
final format = InputImageFormatValue.fromRawValue(image.format.raw);
// validate format depending on platform
// only supported formats:
// * nv21 for Android
// * bgra8888 for iOS
if (format == null ||
    // (Platform.isAndroid && format != InputImageFormat.nv21) ||
    (Platform.isIOS && format != InputImageFormat.bgra8888)) return null;

// since format is constraint to nv21 or bgra8888, both only have one plane
// if (image.planes.length != 1) return null;
final plane = image.planes.first;
final nv21Image = image.getNv21Uint8List();

// compose InputImage using bytes
return InputImage.fromBytes(
  bytes: Platform.isAndroid ? nv21Image : plane.bytes,
  metadata: InputImageMetadata(
    size: Size(image.width.toDouble(), image.height.toDouble()),
    rotation: rotation, // used only in Android
    format: format, // used only in iOS
    bytesPerRow: plane.bytesPerRow, // used only in iOS
  ),
);

}

Before convert input image.

Ah, I see! The issue was with the format being InputImageFormat.yuv_420_888, which was passed to the metadata and caused the error.
Although it is reported for the ios I don't why is causing issue in the android also

where did you put the code for the nv21 conversion? @gauravRNDev

Here is a more complete example using the code from @cagrialta in case anyone is still struggling with where to put it and how to make it work. The following code works on both Android and iOS.
extension Nv21Converter on CameraImage {
  // Code from @cagrialta
}

const _orientations = {
  DeviceOrientation.portraitUp: 0,
  DeviceOrientation.landscapeLeft: 90,
  DeviceOrientation.portraitDown: 180,
  DeviceOrientation.landscapeRight: 270,
};

InputImage? cameraImageToInputImage(
    CameraImage image,
    CameraDescription camera,
    DeviceOrientation deviceOrientation,
  ) {
    final format = InputImageFormatValue.fromRawValue(image.format.raw);
    if (format == null) {
      return null;
    }
    final plane = image.planes.firstOrNull;
    if (plane == null) {
      return null;
    }

    final sensorOrientation = camera.sensorOrientation;

    final InputImageRotation? rotation;
    if (Platform.isIOS) {
      rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
    } else if (Platform.isAndroid) {
      var rotationCompensation = _orientations[deviceOrientation];
      if (rotationCompensation == null) {
        return null;
      }
      if (camera.lensDirection == CameraLensDirection.front) {
        // front-facing
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
      } else {
        // back-facing
        rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360;
      }
      rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
    } else {
      rotation = null;
    }

    if (rotation == null) {
      return null;
    }

    final Uint8List bytes;
    if (Platform.isAndroid) {
      bytes = image.getNv21Uint8List();
    } else {
      final allBytes = WriteBuffer();
      for (final plane in image.planes) {
        allBytes.putUint8List(plane.bytes);
      }
      bytes = allBytes.done().buffer.asUint8List();
    }

    return InputImage.fromBytes(
      bytes: bytes,
      metadata: InputImageMetadata(
        size: Size(image.width.toDouble(), image.height.toDouble()),
        rotation: rotation, // used only in Android
        format: Platform.isAndroid ? InputImageFormat.nv21 : format, // Hardcode format for Android
        bytesPerRow: plane.bytesPerRow, // used only in iOS
      ),
    );
  }

Then simply call it wherever you have the input image like this
final inputImage = cameraImageToInputImage(image, camera, cameraController.value.deviceOrientation);
if (inputImage == null) {
  return;
}
final barcodes = await _barcodeScanner.processImage(inputImage);

Don't forget to hardcode the image format for Android otherwise it uses yuv420 (at least on my device) which doesn't work and causes the PlatformException even if you put bytes that are formatted as nv21.

Here is a more complete example using the code from @cagrialta in case anyone is still struggling with where to put it and how to make it work. The following code works on both Android and iOS.
extension Nv21Converter on CameraImage {
  // Code from @cagrialta
}

const _orientations = {
  DeviceOrientation.portraitUp: 0,
  DeviceOrientation.landscapeLeft: 90,
  DeviceOrientation.portraitDown: 180,
  DeviceOrientation.landscapeRight: 270,
};

InputImage? cameraImageToInputImage(
    CameraImage image,
    CameraDescription camera,
    DeviceOrientation deviceOrientation,
  ) {
    final format = InputImageFormatValue.fromRawValue(image.format.raw);
    if (format == null) {
      return null;
    }
    final plane = image.planes.firstOrNull;
    if (plane == null) {
      return null;
    }

    final sensorOrientation = camera.sensorOrientation;

    final InputImageRotation? rotation;
    if (Platform.isIOS) {
      rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
    } else if (Platform.isAndroid) {
      var rotationCompensation = _orientations[deviceOrientation];
      if (rotationCompensation == null) {
        return null;
      }
      if (camera.lensDirection == CameraLensDirection.front) {
        // front-facing
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
      } else {
        // back-facing
        rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360;
      }
      rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
    } else {
      rotation = null;
    }

    if (rotation == null) {
      return null;
    }

    final Uint8List bytes;
    if (Platform.isAndroid) {
      bytes = image.getNv21Uint8List();
    } else {
      final allBytes = WriteBuffer();
      for (final plane in image.planes) {
        allBytes.putUint8List(plane.bytes);
      }
      bytes = allBytes.done().buffer.asUint8List();
    }

    return InputImage.fromBytes(
      bytes: bytes,
      metadata: InputImageMetadata(
        size: Size(image.width.toDouble(), image.height.toDouble()),
        rotation: rotation, // used only in Android
        format: Platform.isAndroid ? InputImageFormat.nv21 : format, // Hardcode format for Android
        bytesPerRow: plane.bytesPerRow, // used only in iOS
      ),
    );
  }

Then simply call it wherever you have the input image like this
final inputImage = cameraImageToInputImage(image, camera, cameraController.value.deviceOrientation);
if (inputImage == null) {
  return;
}
final barcodes = await _barcodeScanner.processImage(inputImage);

Don't forget to hardcode the image format for Android otherwise it uses yuv420 (at least on my device) which doesn't work and causes the PlatformException even if you put bytes that are formatted as nv21.

This solution is quite good but still have issues when testing on Android - google_mlkit_object_detection: ^0.14.0
--> The solutions works with camera: ^0.11.0 but for some reason the rotation of the camera is switched by 90 degrees. I believe is an issue with the camera package. The solution was to downgrade to camera: ^0.10.6 but then trying your code on Android device is again returning an error below. I already tested with iPhone has no issue.
It looks like using Camera 0.10.6 - image.planes has only one item in the array instead of 3.
CameraImage? img;
doObjectDetectionOnFrame() async {
InputImage? frameImg;
frameImg = cameraImageToInputImage(
    img!, cameras[0], controller!.value.deviceOrientation);

// getInputImage();
if (frameImg != null) {
  List<DetectedObject> objects =
      await objectDetector.processImage(frameImg);
  // print("len= ${objects.length}");
  setState(() {
    _scanResults = objects;
  });

  // print('length results: ' + objects.length.toString());
}
//   else {
//   print('InputImage is null');
// }
isBusy = false;

}
E/flutter (17521): #0      _Array.[] (dart:core-patch/array.dart)
E/flutter (17521): #1      Nv21Converter.getNv21Uint8List (package:GLOBAL_FISH_360/MFDevPackage/ObjectDetection/ObjectStreamFeed.dart:374:24)
E/flutter (17521): #2      cameraImageToInputImage (package:GLOBAL_FISH_360/MFDevPackage/ObjectDetection/ObjectStreamFeed.dart:469:19)
E/flutter (17521): #3      _ObjectDetectionFeedState.doObjectDetectionOnFrame (package:GLOBAL_FISH_360/MFDevPackage/ObjectDetection/ObjectStreamFeed.dart:133:16)
E/flutter (17521): #4      _ObjectDetectionFeedState.initializeCamera. (package:GLOBAL_FISH_360/MFDevPackage/ObjectDetection/ObjectStreamFeed.dart:64:11)

Here is a more complete example using the code from @cagrialta in case anyone is still struggling with where to put it and how to make it work. The following code works on both Android and iOS.
extension Nv21Converter on CameraImage {
  // Code from @cagrialta
}

const _orientations = {
  DeviceOrientation.portraitUp: 0,
  DeviceOrientation.landscapeLeft: 90,
  DeviceOrientation.portraitDown: 180,
  DeviceOrientation.landscapeRight: 270,
};

InputImage? cameraImageToInputImage(
    CameraImage image,
    CameraDescription camera,
    DeviceOrientation deviceOrientation,
  ) {
    final format = InputImageFormatValue.fromRawValue(image.format.raw);
    if (format == null) {
      return null;
    }
    final plane = image.planes.firstOrNull;
    if (plane == null) {
      return null;
    }

    final sensorOrientation = camera.sensorOrientation;

    final InputImageRotation? rotation;
    if (Platform.isIOS) {
      rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
    } else if (Platform.isAndroid) {
      var rotationCompensation = _orientations[deviceOrientation];
      if (rotationCompensation == null) {
        return null;
      }
      if (camera.lensDirection == CameraLensDirection.front) {
        // front-facing
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
      } else {
        // back-facing
        rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360;
      }
      rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
    } else {
      rotation = null;
    }

    if (rotation == null) {
      return null;
    }

    final Uint8List bytes;
    if (Platform.isAndroid) {
      bytes = image.getNv21Uint8List();
    } else {
      final allBytes = WriteBuffer();
      for (final plane in image.planes) {
        allBytes.putUint8List(plane.bytes);
      }
      bytes = allBytes.done().buffer.asUint8List();
    }

    return InputImage.fromBytes(
      bytes: bytes,
      metadata: InputImageMetadata(
        size: Size(image.width.toDouble(), image.height.toDouble()),
        rotation: rotation, // used only in Android
        format: Platform.isAndroid ? InputImageFormat.nv21 : format, // Hardcode format for Android
        bytesPerRow: plane.bytesPerRow, // used only in iOS
      ),
    );
  }

Then simply call it wherever you have the input image like this
final inputImage = cameraImageToInputImage(image, camera, cameraController.value.deviceOrientation);
if (inputImage == null) {
  return;
}
final barcodes = await _barcodeScanner.processImage(inputImage);

Don't forget to hardcode the image format for Android otherwise it uses yuv420 (at least on my device) which doesn't work and causes the PlatformException even if you put bytes that are formatted as nv21.

This solution is quite good but still have issues when testing on Android - google_mlkit_object_detection: ^0.14.0 --> The solutions works with camera: ^0.11.0 but for some reason the rotation of the camera is switched by 90 degrees. I believe is an issue with the camera package. The solution was to downgrade to camera: ^0.10.6 but then trying your code on Android device is again returning an error below. I already tested with iPhone has no issue.
It looks like using Camera 0.10.6 - image.planes has only one item in the array instead of 3.
CameraImage? img; doObjectDetectionOnFrame() async { InputImage? frameImg;
frameImg = cameraImageToInputImage(
    img!, cameras[0], controller!.value.deviceOrientation);

// getInputImage();
if (frameImg != null) {
  List<DetectedObject> objects =
      await objectDetector.processImage(frameImg);
  // print("len= ${objects.length}");
  setState(() {
    _scanResults = objects;
  });

  // print('length results: ' + objects.length.toString());
}
//   else {
//   print('InputImage is null');
// }
isBusy = false;

}
E/flutter (17521): #0 _Array.[] (dart:core-patch/array.dart) E/flutter (17521): #1 Nv21Converter.getNv21Uint8List E/flutter (17521): #2 cameraImageToInputImage E/flutter (17521): #3 _ObjectDetectionFeedState.doObjectDetectionOnFrame E/flutter (17521): #4 _ObjectDetectionFeedState.initializeCamera.

I fixed it with initiating the controller with yup420 instead of nv21.
controller = CameraController(
  cameras.first,
  ResolutionPreset.high,
  imageFormatGroup: Platform.isAndroid
      ? ImageFormatGroup.yuv420
      : ImageFormatGroup.bgra8888,
);

@simeonangelov94 Thanks! Actually I run into the same issue right now when tested on different physical device. I applied your suggestions and it works great.


Open an official Docker issue on the swarm bug

Defined a clear, repeatable use case to make obvious the issue we found on swam multi nodes configuration in order to get official Docker update or feedback.

Good news: the bug was ours, after a fix on the security group, I don't notice any issues anymore on a full Swarm cluster (3 managers + 5 nodes). No need to open an issue on the Docker repo.
I've applied the changes on https://github.com/appcelerator/amp-swarm-deploy


cloudstack reconcile cp

Issue #, if available:
Implement ReconcileControlPlane#1014
Description of changes:
Testing (if applicable):
unit test
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm


'sails' is not recognized as an internal or external command

OS: Windows 10
Node version:
node -v
v10.16.2
npm -v
6.10.3
Sails version (sails): 1.2.3

https://github.com/balderdashy/sails/issues/1196 - not solved

@SGQuar try navigating to your E: directory and then write a sails command

Maybe your path is broken somehow did you try?
E:\> where sails

Try the following

Close the terminal and open it again
try to start sails from D: prompt

Your global npm path not added windows  path. add in path, reopen the terminal


Error compiling Readium in Android Studio

From the IDPF Forum, jojoth, http://idpf.org/forum/topic-2509
I'm import readium_SDK_Test_Android in android studio follow https://github.com/readium/SDKLauncher-Android
So I have error when run::
Error:Execution failed for task ':ePub3Library:compileReleaseNdk'.

com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command '/Users/pong/Library/Android/ndk-r10e/ndk-build'' finished with non-zero exit value 2
Some body have any solution

Thank for advance,
sukat
PS. I'm not good abut writing.

Related issue:
https://github.com/readium/SDKLauncher-Android/issues/61
Note that @JCCR managed to convert the Eclipse project to Gradle / Android Studio, but I think that the native readium-sdk lib (including the JNI bits) was built via the command line (ndk-compile.sh) which takes care of all the peripheral setup / prepare tasks (such as copying the headers across, etc.).

Thank you  daniel and ekwright now  I make them to jar file follow by http://wudev.digitaltorque.com/2014/03/building-readium-for-x86-android.html
But  anywhere have sample or tutorial about how to use it.
Thank you for advance

@jojoth I successfully use these feature branches:

@jojoth Hello, I successfully compil readium sdk and android launcher app.
we need

download latest android studio
from studio install Android SDK Tools, Android SDK Platforms-tools and Android SDK Build-tools. Install SDK Platform >= 4.1.2
download NDK r10e
download readium-sdk, readium-shared-js and SDKLauncher-Android. branch develop

compil epub3library

open studio, make a new project
Application name : EPub3Library
Company domain : android.sdk.readium.org
Package name : org.readium.sdk.android
Minium SDK : API 16 Android 4.1
Add no activity.
File > New > New module and choose Android Library
Application name : Library
Module name : ePub3-Library
Company domain : android.sdk.readium.org
Package name : org.readium.sdk.android
Minium SDK : API 16 Android 4.1
Add no activity
delete app module, File > Project Structure. studio no delete folder app, delete it
delete [PROJECT_DIR]/app
delete [PROJECT_DIR]/[LIB_DIR]/src/androidTest
delete [PROJECT_DIR]/[LIB_DIR]/src/main/res/*
File > Projet structure, SDK Location, Android NDK location put ndk path
in file [PROJECT_DIR]/[LIB_DIR]/src/main/AndroidManifest.xml delete section application
in file [LIB_DIR]/build.grable, add in android, defaultConfig

ndk {
moduleName "libepub3"
}
in  android add
    sourceSets.main {
        jni.srcDirs = [] // disable automatic ndk-build call
    }

    task makeHeaders(type: Exec, description: 'Copy headers source') {
        commandLine './MakeHeaders.sh', 'Android'
    }

    task ndkBuild(type: Exec, dependsOn: 'makeHeaders',
            description: 'Compile JNI source via NDK') {
        commandLine 'ndk-build',
                '-C',
                file('src/main').absolutePath,
                'V=1',
                'NDK_APPLICATION_MK=Application.mk',
                'NDK_PROJECT_PATH=build',
                'APP_BUILD_SCRIPT=Android.mk',
                'NDK_DEBUG=1'
    }

    task ndkToJar(type: Zip, dependsOn: 'ndkBuild',
            description: 'Create a JAR of the native libs') {
        destinationDir new File(buildDir, 'libs')
        baseName 'epub3-library'
        extension 'jar'
        from fileTree(dir: 'src/main/build/libs', include: '**/*')
        into 'lib/'
    }

    tasks.withType(JavaCompile) {
        compileTask -> compileTask.dependsOn ndkToJar
    }

in dependencies modifie libs path
    dependencies {
        compile fileTree(dir: 'build/libs', include: ['*.jar'])
        compile 'com.android.support:appcompat-v7:22.2.1'
    }

copy files

copy readium-sdk/Platform/Android/src/org > [PROJECT_DIR]/[LIB_DIR]/src/main/java

copy readium-sdk/Platform/Android/jni > [PROJECT_DIR]/[LIB_DIR]/src/main

copy readium-sdk/ePub3 > [PROJECT_DIR]/[LIB_DIR]

copy readium-sdk/MakeHeaders.sh > [PROJECT_DIR]/[LIB_DIR]
replace Platform/"$1"/MakeHeaders.sh by
[[ -d "src/main/include" ]] || {
src/main/MakeHeaders.sh
}

don't forget add right executable

copy readium-sdk/Platform/Android/Application.mk  > [PROJECT_DIR]/[LIB_DIR]/src/main
copy readium-sdk/Platform/Android/prefix.h > [PROJECT_DIR]/[LIB_DIR]/src/main

Build > Make Project

Now for SDKLauncher

In studio create a new project
Application name : SDKLauncher
Company domain : android.sdk.readium.org
Package name : org.readium.sdk.android
Minium SDK : API 16 Android 4.1
Add no activity
delete [PROJECT_DIR]/app/src/androidTest
delete [PROJECT_DIR]/app/src/main/res/*
copy files

copy SDKLauncher-Android/SDKLauncher-Android/src > [PROJECT_DIR]/app/src/main/java
copy SDKLauncher-Android/SDKLauncher-Android/res > [PROJECT_DIR]/app/src/main/res
copy SDKLauncher-Android/SDKLauncher-Android/AndroidManifest.xml > [PROJECT_DIR]/app/src/main
copy SDKLauncher-Android/SDKLauncher-Android/libs > [PROJECT_DIR]/app
copy SDKLauncher-Android/SDKLauncher-Android/assets > [PROJECT_DIR]/app/src/main
copy readium-shared-js/build-output/css > [PROJECT_DIR]/app/src/main/assets/readium-shared-js/builds/css
copy readium-shared-js/build-output/_multiple-bundles > [PROJECT_DIR]/app/src/main/assets/readium-shared-js/builds/multiple-bundle
copy readium-shared-js/build-output/_single-bundle > [PROJECT_DIR]/app/src/main/assets/readium-shared-js/builds/single-bundle

configure grabble, add path to epub3library

open [PROJECT_DIR]/setting.grable and add
include ':app'
include ':Epub3Library'
project(':Epub3Library').projectDir =
new File(settingsDir, '../Epub3Library/ePub3-Library')

configure module app

in file  [PROJECT_DIR]/app/build.grable, modify dependencies and add configurations
configurations {
all*.exclude group: 'com.android.support', module: 'support-v4'
}
dependencies {
compile project(path: ':epub3library')
compile fileTree(dir: 'libs', include: ['*.jar'])
}

Build > Make Project

@TienHai that's an awesome step-by-step guide, thanks!
I wonder how this compares with Montano's Gradle setup:
https://github.com/readium/readium-sdk/pull/198/files#diff-a168633d10f4ca9d28f1fae36331b4c4
(scroll down to see the other Gradle files)

Hello @TienHai,thanks for your step-by-step guide.But I've met some problems when I am compiling compil epub3library.Can you help me to fix it?

@Lylicfuture can you try running MakeHeaders.sh directly from your Bash / Cygwin shell? It is located in readium-sdk/Platform/Android/ https://github.com/readium/readium-sdk/blob/develop/Platform/Android/MakeHeaders.sh

Hi @Lylicfuture
The scritp "MakeHeaders.sh" copy the header files from ePub3/ThirdParty in "src/main/includes".
Exist 2 files "MakeHeaders.sh"

readium-sdk/MakeHeaders.sh
readium-sdk/Platform/Android/MakeHeaders.sh

Copy the first in > [PROJECT_DIR]/[LIB_DIR] and replace Platform/"$1"/MakeHeaders.sh by
[[ -d "src/main/include" ]] || {
src/main/MakeHeaders.sh
}
for the second, I see, I forget in tuto we have to copy readium-sdk/Platform/Android/MakeHeaders.sh > [PROJECT_DIR]/[LIB_DIR]/src/main (I have update the tuto, thansk) .

@danielweck I am afraid I can not running MakeHeaders.sh directly from my Cygwin shell.The feedback I get from Cygwin is that
.It seems the interpreter is wrong.

Have you tried (re)installing zsh?

thanks @danielweck，now I can run MakeHeaders.sh directly from cygwin.But when I buile the project in Andrid studio the same question exist.

@Lylicfuture
Did you retry running MakeHeaders.sh directly?

Hello @TienHai.I have tried it several times ,it works well.

@Lylicfuture
Did you try running ePub3-Library/MakeHeaders.sh directly?
or ePub3-Library/src/main/MakeHeaders.sh?

@TienHai I have tried them both.they both work well

@Lylicfuture
Can we show your build.grable file in ePub3-Library directory?

@TienHai .OK no problem.
here I post below:
import com.android.build.gradle.internal.dsl.ProductFlavor
apply plugin: 'com.android.library'
android {
compileSdkVersion 23
buildToolsVersion "23.0.1"
final def config = defaultConfig {
    minSdkVersion 16
    targetSdkVersion 23
    versionCode 1
    versionName "1.0"
    ndk {
        moduleName "libepub3"
    }
}
defaultConfig {
    minSdkVersion 16
    targetSdkVersion 23
    versionCode 1
    versionName "1.0"
    ndk {
        moduleName "libepub3"
    }
}
buildTypes {
    release {
        minifyEnabled false
        proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
    }
}

sourceSets.main {
    jni.srcDirs = [] // disable automatic ndk-build call
}

task makeHeaders(type:Exec,description:'Copy headers source') {
    commandLine './MakeHeaders.sh', 'Android'
}

task ndkBuild(type: Exec, dependsOn: 'makeHeaders',
        description: 'Compile JNI source via NDK') {
    commandLine 'ndk-build',
            '-C',
            file('src/main').absolutePath,
            'V=1',
            'NDK_APPLICATION_MK=Application.mk',
            'NDK_PROJECT_PATH=build',
            'APP_BUILD_SCRIPT=Android.mk',
            'NDK_DEBUG=1'
}

task ndkToJar(type: Zip, dependsOn: 'ndkBuild',
        description: 'Create a JAR of the native libs') {
    destinationDir new File(buildDir, 'libs')
    baseName 'epub3-library'
    extension 'jar'
    from fileTree(dir: 'src/main/build/libs', include: '**/*')
    into 'lib/'
}

tasks.withType(JavaCompile) {
    compileTask -> compileTask.dependsOn ndkToJar
}

}
dependencies {
compile fileTree(dir: 'build/libs', include: ['*.jar'])
compile 'com.android.support:appcompat-v7:23.+'
}

@Lylicfuture
All look fine,
try compil with option "--stacktrace"
In studio : "File" > "Settings" > "Build, Execution, Deployment" > "Compiler"
add "--stacktrace" in Comande-line Options.

@TienHai here is the result of stacktrace.

@Lylicfuture
Are the file ePub3-Library/MakeHeaders.sh  is executable ?
and the file ePub3-Library/src/main/MakeHeaders.sh is executable ?
try in you build.grable:
task makeHeaders(type:Exec,description:'Copy headers source') {
commandLine '/bin/bash','MakeHeaders.sh', 'Android'
}
replace '/bin/bash' by your path to bash

@TienHai thanks.Now the task makeheaders can pass.but task ndkBuild have problems too.Maybe I should use my ndk-build path to replace the "ndk-build"  too.I am trying now

@Lylicfuture
Great, maybe you can modify your path to add ndk path and sdk tool path ?
export PATH=$PATH:/home/tien/Devel/Android/Ndk:/home/tien/Devel/Android/Sdk/tools

@TienHai,I have a question is your tuto made for windows or linux?It seems I can not run ndk-build through command line directly.

@Lylicfuture
For linux, you have to translate ndk-build command to windows command.

Thanks @TienHai  for your patience for a rookie like me，now I can compile them successfully.I have spend ten days learn how to use linux and I have learnt much.When I am working under the guide of your tuto,I have find some errors and I have some suggestions for your tuto.
1、I think you should add a tip of installing zsh;
2、In
task makeHeaders(type: Exec, description: 'Copy headers source') {
commandLine './MakeHeaders.sh', 'Android'
}
I think you can add a choice:commandLine 'sh','MakeHeaders.sh', 'Android'.the command line may be different for different systems,for my system,I can not run a file by './',but command 'sh' works well.
3、In your second project there is a little problem in the step.
include ':app'
include ':Epub3Library'
project(':Epub3Library').projectDir =
new File(settingsDir, '../Epub3Library/ePub3-Library')
we need to change 'Epub3Library' to 'EPub3Library',otherwise the project can't find the project we need.
That's all.
Hope my advise can help you to improve your tuto and help more people.
Now I am trying to deploy the launcher in my phone.I have met a problem here is the messages:
Error:Execution failed for task ':app:dexDebug'.

com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command '/usr/lib/jvm/jdk1.8.0_65/bin/java'' finished with non-zero exit value 2

have you ever met it before?

Hello @Lylicfuture thanks for using this tuto.
I add install zsh shell, and I look how change the task "makeHeaders".
For error: Execution failed for task ':app:dexDebug'.
add "multiDexEnabled" option in build.grable file like this.
defaultConfig {
multiDexEnabled true
}

thanks @TienHai I have add multiDexEnabled option in my build.gradle(Module:app) but when I try to deploy it in my phone a new problem exit:
Error:Execution failed for task ':app:packageAllDebugClassesForMultiDex'.

java.util.zip.ZipException: duplicate entry: org/readium/sdk/android/SpineItem.class

thanks @TienHai I have add multiDexEnabled option in my build.gradle(Module:app) but when I try to deploy it in my phone a new problem exit:
Error:Execution failed for task ':app:packageAllDebugClassesForMultiDex'.

java.util.zip.ZipException: duplicate entry: org/readium/sdk/android/SpineItem.class

Goodmorning @Lylicfuture
Can you try change library build.grable file like :
dependencies {
compile fileTree(dir: 'build/libs', include: ['*.jar'])
compile ('com.android.support:appcompat-v7:22.2.1') {
exclude module: 'support-v4'
}
}

@TienHai after I change the build.gradle in Epub3Library.the studio show me a error:
Error:(61, 0) Gradle DSL method not found: 'com.android.support:appcompat-v7:23.1.0()'
Possible causes:The project 'SDKLauncher' may be using a version of Gradle that does not contain the method.

@TienHai I also have build to android studio using your steps and did some of the tweaks to the ./MakeHeader.sh issue and it compiles and works like other in the emulator but i am unable to find the contents of the epub when i open it. Only i get is the metadata and the spine lists. When i open the spine it just is empty. Can you help me in this?
My Log:
https://gist.github.com/teddywest32/e71f80f276859fb58dab

Hi @teddywest32
We're so pleased our tutorial have been useful to you.
Now, I think you can't open your file because is not a EPUB but a PDF. try with real ePub file.
You can try with ePub from readium-test-files.
Hi @Lylicfuture , can you upgrade Gradle ?
Thanks

@TienHai
I have tested using the Test files provided and the i am getting the same issues.
Would that be is because i am missing some files?
My New Log using childrens-literature-20120722.epub
https://gist.github.com/teddywest32/5950ee568c48401d6f8f

@teddywest32 it looks like your build is missing the readium-shared-js JS file (single RequireJS bundle that contains the lightweight Almond AMD loader). @TienHai is that included in your build method?
11-02 13:00:12.270    4545-4678/? D/WebViewActivity﹕ -------- shouldInterceptRequest: file:///android_asset/readium-shared-js/builds/single-bundle/readium-shared-js_all.js
11-02 13:00:12.270    4545-4678/? D/WebViewActivity﹕ file:///android_asset/readium-shared-js/builds/single-bundle/readium-shared-js_all.js => builds/single-bundle/readium-shared-js_all.js
11-02 13:00:12.326    4545-4678/? E/WebViewActivity﹕ NO MANIFEST ITEM ... file:///android_asset/readium-shared-js/builds/single-bundle/readium-shared-js_all.js
11-02 13:00:12.427    4545-4678/? D/chromium﹕ Unknown chromium error: -6
11-02 13:00:12.548    4545-4545/? I/Choreographer﹕ Skipped 41 frames!  The application may be doing too much work on its main thread.
11-02 13:00:12.607    4545-4545/? D/WebViewActivity﹕ onLoadResource: file:///android_asset/readium-shared-js/builds/single-bundle/readium-shared-js_all.js
11-02 13:00:13.137    4545-4678/? E/cutils-trace﹕ Error opening trace file: No such file or directory (2)
11-02 13:00:13.167    4545-4545/? E/Web Console﹕ Uncaught ReferenceError: $ is not defined at file:///android_asset/readium-shared-js/host_app_feedback.js:30
11-02 13:00:13.167    4545-4545/? D/WebViewActivity﹕ onPageFinished: file:///android_asset/readium-shared-js/reader.html

Hi @rkwright , @teddywest32
in step 4 "copy files" for build SDKLauncher

copy files

...
copy SDKLauncher-Android/SDKLauncher-Android/assets > [PROJECT_DIR]/app/src/main
copy readium-shared-js/build-output/css > [PROJECT_DIR]/app/src/main/assets/readium-shared-js/builds/css
copy readium-shared-js/build-output/_multiple-bundles > [PROJECT_DIR]/app/src/main/assets/readium-shared-js/builds/multiple-bundle
copy readium-shared-js/build-output/_single-bundle > [PROJECT_DIR]/app/src/main/assets/readium-shared-js/builds/single-bundle

@teddywest32 , do you do this copies ?

CC @clebeaupin ( @TienHai ) Merci a vous deux :)
Thanks!

I have been busy in other projects for the last three monthes.Now I move back to this android project.Strangely I meet a very rare problem,when I build this project according to your @TienHai  tuto.I build successfully last time in my 32bit ubuntu system but now I can not build correttly in 64bit  ubuntu system.Here is the problem.

this problem occurred because of some files or directories missing,and I found the file it needs and copy to the directory,but it seems the project will clean the directory every building the project,have you ever met this problem?Do you have any solution to fix the problem?

@Lylicfuture could you please try this new feature branch / pull request in your development environment:
https://github.com/readium/SDKLauncher-Android/pull/84
https://github.com/readium/readium-sdk/pull/224
This is a completely revamped cross-platform build workflow for SDKLauncher-Android (yes, it even works on Windows, out of the box!!). It is based on Android Studio and Gradle, and we can now seamlessly debug into both Java and C++ code. Please make sure to download / update the latest version of AndroidStudio from the Canary channel.
It would be great if you guys could give this a try, as the plan is to integrate this feature branch into the main develop branch, as a replacement for the antiquated (and broken) Eclipse-based build system.
Thank you very much!
Daniel
CC @clebeaupin

Hi @TienHai
I followed steps as defined by you in above section and facing some problems

First section below are the logs
:ePub3-Library:ndkBuild FAILED

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':ePub3-Library:ndkBuild'.

A problem occurred starting process 'command 'ndk-build''

Try:
Run with --info or --debug option to get more log output.

Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':ePub3-Library:ndkBuild'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64)
at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:52)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:203)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:185)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:62)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:50)
at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:25)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:110)
at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
at org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
at org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
at org.gradle.initialization.DefaultGradleLauncher$4.run(DefaultGradleLauncher.java:154)
at org.gradle.internal.Factories$1.create(Factories.java:22)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:52)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:151)
at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:32)
at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:99)
at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:93)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:62)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:93)
at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:82)
at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:94)
at org.gradle.tooling.internal.provider.runner.BuildModelActionRunner.run(BuildModelActionRunner.java:46)
at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
at org.gradle.tooling.internal.provider.runner.SubscribableBuildActionRunner.run(SubscribableBuildActionRunner.java:58)
at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:43)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:28)
at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:77)
at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:47)
at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:52)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:37)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
at org.gradle.util.Swapper.swap(Swapper.java:38)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.health.DaemonHealthTracker.execute(DaemonHealthTracker.java:47)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:66)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:71)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.health.HintGCAfterBuild.execute(HintGCAfterBuild.java:41)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)
at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:246)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command 'ndk-build''
at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:197)
at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:327)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:86)
... 2 more
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'ndk-build'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:68)
... 2 more
Caused by: java.io.IOException: Cannot run program "ndk-build" (in directory "/home/ravi/workspace/EPub3Library/ePub3-Library"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 4 more
Caused by: java.io.IOException: error=2, No such file or directory
... 5 more

BUILD FAILED

Second section
:Epub3Library:ndkBuild FAILED

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':Epub3Library:ndkBuild'.

A problem occurred starting process 'command 'ndk-build''

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
Total time: 2.435 secs

Hello @RaviTeja457 would you mind trying the new feature branch for Android Studio + Gradle? Please have a look at the comment immediately above your (in the GitHub discussion thread). Link: https://github.com/readium/readium-sdk/issues/180#issuecomment-175562814

The old Eclipse model is deprecated (by Google), now replaced with Android Studio IDE +Gradle build system. The pull requests have been merged into the develop branch.

is there something new about the android studio migration?
the code structure in its current state does not import nor compile using any NDK compile method found online.
ndk-compile.sh missing
MakeHeaders.sh under Platform/android is missing
is there someone who have a working stream that can compile by NDK without re-writing the whole scripts?...

@looknear Take a look at:  https://docs.google.com/document/d/1ebFQ-8BGoiamKO4K0ZiL1nH6b8wulJ-ORyZKiDEYi-8/edit

wow. that was simple. maybe to put these  few lines in the source README ? :)
so many import methods, half auto, half manual. no one mentioned to import the inner gradle dir...
Thanks. now will start checking why the app is crashing while trying to open a book... :)

The platform-dependent shell scripts have been replaced with a cross-platform Gradle build system, and totally integrated in Android Studio.

@looknear  Thanks for the feedback.  I have updated the readme in the Launcher repo (develop branch only so far, of course).

is there a known issue with books with custom fonts (ttf) which crashing on android 6+ ?
i have few books that open fine on older devices. but crash on android 6.
is there an issue opened about it?

IIRC, I successfully tested EPUBs containing obfuscated fonts on Marshmallow (6) and more recently Nougat (7).

what about RTL books? ...
i have a book that crash on android 6, and open (with default font) on older device.
both on same current SDKlauncher code.

@looknear  Can you share that book with us so we can look into it?  Or, of course, you can do it yourself - this is an open-source volunteer effort, after all.

i found the error. it was in the book. some TTF fonts definitions the systems didnt load and recover from.
changed to open font type - so now managed to open a book.
but it look like RTL display has some issues... all pages has a minus padding (text cut) on the right side of the page (start of RTL lines). but if you move to last page of the chapter - it is fixed. then back to the previous pages - they all look great.
but then again - in new chapter - text is cut on the right side. look like some strange "minus value" padding.
have you ever saw issue like that?
can share a sample book if you feel like checking it out.

by the way - some permission checks were missing while trying to open "epubtest" library for reading. on Android 6 need to preform a check on every operation. not just in the manifest. will make the changes and push it.

I followed the steps above but I still cant run this sdk.
I can't run the Makeheaders.sh even though i have bash and zsh installed.

MakeHeaders.sh is unnecessary, the Eclipse + shell scripts method is obsolete (and officially deprecated by Google too).
The new Android Studio / Gradle build system handles all of this automatically. What branch are you using? Please give the develop branch a try.

Thanks so much. It's working now. :)


Fix links on PyPI readme

##hacktoberfestivus
Fixes relative links in the README.md in PyPi. The relative links are converted to the absolute ones, pointing at the relevant GitHub repo files, including commit information.
Contributors: The GreenTeam w/
torgeha
JoakimJohansen
antoneskov

Contributor checklist

[x] :tada: This PR closes #244 .
[ ] :scroll: I have broken down my PR into the following tasks:

[ ] Task 1
[ ] Task 2

[ ] :robot: I have added tests, or extended existing tests, to cover any new features or bugs fixed in this PR.
[ ] :book: I have considered adding a new entry in CHANGELOG.md, and added it if should be communicated there.

Thanks @antoneskov et al. for the PR! Did some minor changes to get this through CI, otherwise LGTM! :+1: :tada:  :jack_o_lantern:


Add crash-reporting

At least a very basic one, possibly based on piwik and after feedback from user

It turns out that the ACRA library supports mail as report destination.


Test with Swift 5.7/Xcode 14

This allows us to detect issues with different Xcode versions.

I've reported the breakage of protocol conformance in https://github.com/apple/swift/issues/60514


Add file-type check to filter out non-images

Fixes #5
Todo:

[x] Ensure the found Buffer has a type
[x] Add a whitelist of mime-types

I definitely agree that we could/should return the mimetype and extension.. good idea 😊


Company page- Industry type values is not in same line with others

Company page- Industry type values is not in same line with others

this happens when name length is greater than 15 characters,
as we try to split to 2nd line

to solve this we need bigger column size, to save on size, we try to split big names into 2 lines


Updated File Structure Properly

#112 issue resolved
organized files in folders (single folder for html pages, one fro CSS, and one for js) and also link them

Perfect....is it working if convert to link....like GitHub page link....

yes, I have run it on VS code live server

bro....solve merge conflict

Thanks a lot sr 🤩☺


Error: The specified module could not be found.

Hi,
I'm trying to use mdns with node-webkit on Windows. I have tried both installation methods (npm install and also pulling down the latest and linking).
I have tried a vanilla install but get this error. Then I tried using nw-gyp configure --target=0.10.5 --target-arch=ia32  to specifically target the node webkit version and architecture I want. Again, same issue.
This is on Windows 7 with node 0.11.3 and node webkit 0.10.5. Below is a stack trace I get inside node webkit:
Error: The specified module could not be found.
C:\Users\build\AppData\Local\Temp\nw8680_21392\node_modules\mdns\build\Release\dns_sd_bindings.node
    at Error (native)
    at Module.load (module.js:354:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:362:17)
    at require (module.js:378:17)
    at Object. (C:\Users\build\AppData\Local\Temp\nw8680_21392\node_modules\mdns\lib\dns_sd.js:24:20)
    at Module._compile (module.js:454:26)
    at Object.Module._extensions..js (module.js:472:10)
    at Module.load (module.js:354:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:362:17)
    at require (module.js:378:17)

Since it does not find the module I assume the build fails...

Did you follow the installation instructions?
Please gist the full compiler output.

Yes. I have the Bonjour SDK installed properly and the Bonjour Home env variable set as well.
Here you go

Just to clarify, "npm test" passes successfully so I think the module is compiled properly.

Indeed, it builds just fine. No clue then... other than that it is probably $something with node-webkit and native modules. :-/
Maybe @y-lohse has an idea? AFAIK he has a setup similar to yours.

Ugh, I figured it out. According to this it's an old bug with node webkit. The executable actually needs to be named nw.exe to find the modules properly, I was renaming it in my grunt task to the actual app name.
Sorry for this @agnat

NP. Glad you figured it out.


Allow reading credentials from separate file

Fixes #61

The fix looks good to me, if @jerger looks good to you as well I'm good to merge it in.

Sorry for the delay - looks good to me also :-)


NIFI-5327 Adding Netflowv5 protocol parser

NIFI-5327 Netflowv5 protocol parser processor.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Flow Generator: https://flowalyzer-netflow-generator.soft112.com/
Flow Template: NetFlowv5_Test_Template.zip
[This template receives data over UDP and use this custom processor to parse it]
https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html - For more details ,Refer NetflowV5  section in this article.
Hope these will be useful.

Anything not windows specific?

Generator.zip
@ottobackwards  Attached data generator jar. Although this is rough code, hope it serves the purpose. Please check the readme for usage.

@PrashanthVenkatesan thanks for putting all the work in on this, I'm not going to be able to test this for a couple of day most likely, but I will

@ottobackwards Have any bandwidth to test this PR?

I will try.  Can you fill out the checkboxes in the PR template above?

When this PR goes in, I'll create a jira for Netflow5RecordReader

Build with tests and contrib-check was fine as well

@ottobackwards  Thanks for your valuable review.
I thought if user flow sends this json to some sink (say kafka), it would be good to have the raw_data with it. are you suggesting me to remove this ? ..
Providing template also for the above purpose, if the external system wants to parse the raw or understand the parsing configs, template would be necessary.  Also further ahead if we support other netflow related protocols that has concept of dynamic templates, it is worth sending templates via relationship.
I understand your point adding raw to json looks like conflating, but i felt creating the another relationship would create separate content claim for each record that might degrade the throughput of processor.
would like to know your view on my points..
I will incorporate all the other review comments and soon push the commit.

@PrashanthVenkatesan It looked to me that the raw data was NOT part of the json.  Am I mistaken?  I don't think that is good practice, having json + something.
If you put the raw content in a json field ( maybe encoded base64 or something ) that would work I think.
So IF I configured output to content and IF I select include raw in output THEN encode the raw and put it in the json.
Something like that.
I don't understand how the output of the Netflow could be different where you would need templates.  When would there be different json?

@ottobackwards  You're right. Adding RAW to json is configurable property.  I also see no usage of the RAW data downstream.  I will remove that property.. Fine??
However templates is required, one scenario is "assume user want to convert this json to avro in downstream(external to nifi). That time he needs to know the data type of each field to create avro schema.. Sending ONE-TIME template , would  be helpful in downstream to create schema."

Is that template ever going to change?

I think have an original relationship for flows that want to pass the data on as-is is a better option myself.

@ottobackwards  So you mean to create "original" relationship and emit the binary RAW data to that relationship??  Fine I can do that change.
Reg template, netflowv5 template doesn't change. But going forward if we write processor for other related protocol like netflowv9, ipfix,etc.. These protocols have dynamic template(template will change at runtime, we need to parse that from the incoming packet)...

In Nifi, they way that the template would be done, is not through a Processor, but instead you would implement this functionality as a RecordReader with a set schema ( like the syslog schemas ).  Then you could choose to write the data with the schema included in the avro.
Sending a template is not something that processors in standard nifi usually do, it will be enough to just document the schema in the additionalDetails.html documentation.
When this processor lands, I think I'm going to do the RecordReader.
Then again, we should get someone else's opinion.  @bbende  can you chime in here?

I haven't looked at any of the code so keep that in mind :)
If we are talking about NiFi templates, then they are just examples of how to use a processor or set of processors, which you can then give to someone to help them get started. We have a wiki page where people have posted some for the community:
https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
I think the recent work @ottobackwards did with the syslog record readers is a good example of what we can do for this case. Meaning later on we can implement a NetflowV5RecordReader, which then lets you use stuff like ConvertRecord to go from netflow to any format like JSON, CSV, Avro.
For now, if this processor always produces JSON, then documenting the expected output format in additionalDetails.html seems sufficient to me.
I don't really know enough about what people do with netflow data to know if keeping the raw message with the parsed one makes sense. I would say typically we wouldn't keep the raw message with the parsed one, but you could always have an option to control that if you thought it was necessary. The processor can also have an original relationship as Otto suggested, although at that point the original and parsed data are completely separate.

@bbende when he says Template he means a json record with schema information

I think conclusion at this point,

Sending the RAW binary message through "original" relationship
Remove the "template" relationship and document the json schema in additionalDetails.html ..
I will do these changes and push the code.

OK.  Contrib check and build passes.
Everything runs the way it should.  My question is on the data.
The src and dest addresses are just the numbers.  They are supposed to be ip addresses.
I think that formatting them as IP addresses rather than just as numbers would be more usable.
Is there something I'm missing?  I don't have an example of something else that shows net flow data.
Are there any other fields that are output 'raw' that could be formatted?

You're right. I will format those fields**(srcaddr, dstaddr & nexthop)** as IP address..

Also, thinking of it, the name of the Nar you have should be network processors nar, not just network nar.  When we do the controller services (record readers) they will need a nar and that is the convention.

Committers,
Any one available to review this PR?

@joewitt @bbende AFAIK, I have no resources for testing this against a live Cisco system. What are your thoughts on merging it if everything checks out but I can't do a live test?

@PrashanthVenkatesan you also have a merge conflict. That'll need to be resolved as well.

@PrashanthVenkatesan any updates?

Sorry for the delay @MikeThomsen . I will make changes and push by EOD.

@MikeThomsen One of the build failed due to some other reason. How can i retrigger the build without making new commit?

You can trigger a build by closing and reopening the pr.

@PrashanthVenkatesan if any of the builds passed, don't worry about it. It's the same test, just localized to different locale settings.

@ottobackwards  @MikeThomsen Thanks for all your support and guidance in my first OS contribution.  Soon, I will try to contribute for NetFlowv9 processor.


DRC warning: "The current configuration does not include the library ''."

With the latest main version da33521870e1b51d625efb8dc190ef643dab437f, each instance of kibuzzard generates a DRC failure:

This is with a freshly-installed KiCad 7.0.2

Same issue here. This is with KiCad 7.0.7 on MacOS Ventura.

This issue is mainly because the labels are generated, rather than coming from a library. IMO this doesn't matter especially since the labels are saved in KiCad project. But I understand the rationale behind the DRC check for this.
There are a few ways to fix this:

Manually mark each KiBuzzard label as excluded
Disabling that DRC check, or switch it to a warning
Extend KiBuzzard so that it saves each label into a global/project library.

I'm not sure it really makes sense to do the last item.


ja: Make docs/tasks/inject-data-application/_index.md follow v1.18 of the original text

This is a Feature Request

What would you like to be added
Update docs/tasks/inject-data-application/_index.md to follow v1.18 of the original (English) text.
Why is this needed
content/ja/docs/tasks/inject-data-application/_index.md is outdated.
Comments
/language ja
/good-first-issue
File to update
https://github.com/kubernetes/website/blob/dev-1.18-ja.1/content/ja/docs/tasks/inject-data-application/_index.md
Original
https://github.com/kubernetes/website/blob/fb6364d/content/en/docs/tasks/inject-data-application/_index.md
diff between translated and v1.18
https://gist.github.com/b8bc25682bffac8abb1a915b11138d09
Note
Currently, we use dev-1.18-ja.1 branch. Please open a PR targeting the branch.
ref
https://kubernetes.io/ja/docs/contribute/localization/
If you have a question, feel free to ask us at slack.k8s.io #kubernetes-docs-ja channel

/assign

/close


Kubeflow pipeline unable to run due to large packages taking limited space

In the current kubeflow pipeline resource configuration large packages like Pytorch will cause the pipeline to fail due to their size (around 4GB) taking space during package installation. The temporary fix for this seems to be to delete the created kubeflow component pods with:
kubectl get pods -n
kubectl delete pod () -n ()

A better fix would be to somehow enable kubeflow components to install the cpu only variant of torch, which in a regular venv can be installed with:
torch==2.3.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
torchvision==0.18.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

However, kubeflow component package installs don't understand the -f option, which is why I think a more lasting fix would be increasing the kubeflow pipeline resource configuration if possible.

@K123AsJ0k1 Is the issue that the component goes over the disk memory limit for the task? If that is the case, we can also increase that limit. I can't remember exactly how was the argument called, but I think it was something like disk_limit:
@component(
    base_image="python:3.10",
    packages_to_install=["numpy", "mlflow~=2.4.1"],
    output_component_file='components/evaluate_component.yaml',
    disk_limit='10Gi'
)

@JoaquinRivesGambin It is possible, but to me it seems more collective memory size than single component size. I will however test if that change works. Regardless, after checking the kubeflow pipeline docs, I found out that components can use index_urls that can be used similarly to -f option in pip installs. This means that we can reduce the torch package size with the following:
base_image = "python:3.10",
packages_to_install = [
      "python-swiftclient",
      "torch==2.3.0",
      "torchvision==0.18.0"
],
pip_index_urls=[
      "https://pypi.org/simple",
      "https://download.pytorch.org/whl/cpu",
      "https://download.pytorch.org/whl/cpu"
  ]


Issue threedots

Coverage increased (+0.08%) to 91.085% when pulling 56bd25ab38b74b7b246f132277cdb96b2297a9c1 on issue-threedots into 12a5f6e0cf453b13fc3e9145a98c095433c4541c on master.

Coverage increased (+0.08%) to 91.085% when pulling 56bd25ab38b74b7b246f132277cdb96b2297a9c1 on issue-threedots into 12a5f6e0cf453b13fc3e9145a98c095433c4541c on master.

@homu r+

:pushpin: Commit 56bd25a has been approved by kbknapp

:zap: Test exempted - status


installation issue

Hi,
I'm very sorry to bother you. I'm new to python and I fail to install you're package. I'll like to use epimap.
If I do :
conda create --name expiMap_env --yes
conda activate expiMap_env
pip install -U scarches or pip install scarches -r requirements.txt
On Anaconda terminal the installation took practically days in order to compute dependencies.
I've tried numerous way the last week but it does not work.
Would it be possible to share a.yml file for example ?
Thank you for your help,
Best,
Jean-Marie

Hi,
The problem was the python version... I used 3.9 (default) !
Thanks !
However, there seems to be still an issue :
"ImportError: cannot import name 'get_num_classes' from 'torchmetrics.utilities.data' (C:\ProgramData\Anaconda3\envs\expiMap_env4\lib\site-packages\torchmetrics\utilities\data.py)"


V2 Drawer (Bottom) - Scroll, then expand instead of the other way around

Describe the feature that you would like added
Right now the default behavior for Bottom Drawer when the drawer content's full height is taller than the height of the drawer is:

User does an 'up' gesture anywhere in the drawer (handle, drawer content)
Drawer expands to the entire screen (if content taller than or equal to the screen height)
If the content is taller than the screen height and content is a lazy list that is scrollable, user can scroll to see the entire list

I would like it to be (and this is the behavior of Google Bottom Drawer/Sheet):

User does an 'up' gesture within  the drawer content
Drawer maintains same height and allows user to scroll
If the drawer handle experiences the 'up' gesture, drawer expands

What component or utility would this be added to
This will greatly help in Loop Android's effort to implement Bottom Sheet inside the Workspace page editor.
Have you discussed this feature with our team, and if so, who
Mayank Mishra
Additional context/screenshots

Solution has been provided in PR  https://github.com/microsoft/fluentui-android/pull/339


Support writing FixedSizeList to Parquet

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Have array data which are all with fixed size, when I tried to write the RecordBatch with FixedSizeList column, it failed and reported:
"NYI("Datatype FixedSizeList(Field { name: \"item\", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, 128) is not yet supported")"

Describe the solution you'd like
I have searched this error and found at:
https://github.com/apache/arrow-rs/blob/0190408147a34c6c08fcc9ba57443c629c678ca6/parquet/src/arrow/arrow_writer/levels.rs#L161
It looks like not hard to support FixedSizeList just like List (not sure), the only thing need to do is constructing the LevelInfoBuilder for FixedSizeList.
Describe alternatives you've considered
No
Additional context
I'm happy to take it if it's simple as I described above

I think this should be relatively straightforward, as far as anything involving dremel is straightforward. This series of posts may be helpful for whoever picks this one up - https://arrow.apache.org/blog/2022/10/05/arrow-parquet-encoding-part-1/

@tustvold I'd be happy to take over the implementation for this (assuming you're not already working on it). I believe I have a working proof of concept, I just need to integrate it into the codebase and add tests.

Always happy to review PRs, FWIW I think the hard part is the read side

label_issue.py automatically added labels {'parquet'} from #4226