object getters return type error

type T = {
  data: {
    a: number;
  };
};

const b: T = {
  data: {
    a: 123123,
    b: 3444,    // throw type error
  },
};

const c: T = {
  get data() {
    return {
      a: 123,
      b: 333, // not throw type error
    };
  },
};

const d = {
  get data() {
    return {
      a: 123,
      b: 333
    };
  },
};

type td = typeof  d; // { a:number, b:number }

c.data.b not throw type error seems to be a mistake ?

See https://github.com/microsoft/TypeScript/wiki/FAQ#indirect-excess-properties-are-ok

See https://github.com/microsoft/TypeScript/wiki/FAQ#indirect-excess-properties-are-ok

bug b.data.b throw type error.   and in c.data no  type prompt like any,

Sigh... I really wish the wording of this error message would be changed.
This isn't a bug.  The excess property check error is not a type error, it's a lint check.  You can't rely on it to prevent extra properties from being put on an object because it won't always trigger (which is by design).  If that's what you need, see #12936.
The other issue you linked to this (#49511) is unrelated - there is nothing like an any involved here.  { a: number, b: number } is a legal subtype of { a: number }.

Indirect excess properties are not an error. Examples of indirect excess properties would be:

values returned by functions
values stored in intermediate variables

b is direct. You go straight from value→property.
c in indirect because it is one step removed. You have a getter, which is a function that returns a value.
From the linked FAQ, excess property checks are to defend against things like typos:
const p: Dimensions = {
    width: 32,
    height: 14,
    depht: 11 // <-- typo!!
}

But there are plenty of situations where you want or need to allow extra properties.

When inspecting the code I do find something looks like a bug: https://github.com/microsoft/TypeScript/issues/58484.
@lqzhgood , is it what you are talking about?

Sigh... I really wish the wording of this error message would be changed.
This isn't a bug. The excess property check error is not a type error, it's a lint check. You can't rely on it to prevent extra properties from being put on an object because it won't always trigger (which is by design). If that's what you need, see #12936.
The other issue you linked to this (#49511) is unrelated - there is nothing like an any involved here. { a: number, b: number } is a legal subtype of and assignable to { a: number }.

The focus of this issue is not on whether additional attributes can be added,
Is difference types of b.data and c.data by use getters
b.data and c.data should throw errors simultaneously or not

When inspecting the code I do find something looks like a bug: #58484. @lqzhgood , is it what you are talking about?

yes,  d.data and c.data (getter) Inconsistent types returned,  and c.data looks like any

Indirect excess properties are not an error. Examples of indirect excess properties would be:

values returned by functions
values stored in intermediate variables

b is direct. You go straight from value→property.
c in indirect because it is one step removed. You have a getter, which is a function that returns a value.
From the linked FAQ, excess property checks are to defend against things like typos:
const p: Dimensions = {
    width: 32,
    height: 14,
    depht: 11 // <-- typo!!
}

But there are plenty of situations where you want or need to allow extra properties.

Thank you for your answer,  but  c.data look like any in autocomplete.
https://www.typescriptlang.org/play/?#code/C4TwDgpgBAKlC8UDeAoKUAmBDYWBcya6U+UAdgK4C2ARhAE4DcR6AXuwZbQ8+gL7MBKFAGMA9mQDOwKDQJxEqdNlwElxUgEYATAGYdugDQtZBXQBZLh4lAD0tqMAAW9MQHdH4aA1f0T7VgIDIj5jIVEJaSgReQRCdABzCBkVLAAKAEp44npkinoybJstPWMbdDkoXWrre3IxGWdXD1BIKB8xP3KodhMhfjDmYXEpFLj1JJScdKz1dFzgfMK5jSDSkwqzaqg+3ihQlHDW6GAMOOOxADNlRjsHJBI8Ljp6azlnhn2gA

You are conflating two different things.  The autocomplete thing is unrelated to the type checking of c.data.

b.data and c.data should throw errors simultaneously or not

No, they shouldn't; it's already been explained several times why c.data doesn't have an error.  The excess property check doesn't apply to it, but it's still type-checked properly.  For example if you change the code to
const c: T = {
  get data() {
    return 42;
  },
};

then you get a type error.

You are conflating two different things. The autocomplete thing is unrelated to the type checking of c.data.

b.data and c.data should throw errors simultaneously or not

No, they shouldn't; it's already been explained several times why c.data doesn't have an error. The excess property check doesn't apply to it, but it's still type-checked properly. For example if you change the code to
const c: T = {
  get data() {
    return 42;
  },
};

then you get a type error.

Thank you, I understand my mistake
The real problem should be automatic completion
The problem of automatic completion made me think that c.data had changed to any type, resulting in the loss of type checking

In that case, this is a retroactive duplicate of #58484 😄

the real problem is https://github.com/microsoft/TypeScript/issues/58484.   I will close this issue


How do I send notification (like: messages, offers and packs etc) to Facebook messenger ?

I have user id's to which I want to send notifications.
Are there any ways integrated with botkit framework ?

bot.say(msg_obj) is what you are looking for. Proactively sending a message to a user via their PSID.
Here's the docs

Hello I want to also send the message but the docs is not accessible anymore using botkit 0.7.0.  I have tried the following way in bot.js :
bot.say({text:Hello, you  have a new person looking for you,channel:data.sharerid});
When I saw the above is not sending to the user I want to notify, I tried :
let botsharer = controller.spawn({});
 botsharer.say({text:`Hello, you  have a new person looking for you`,channel:data.sharerid});

But still it is not sending to the user any message. I don't wanna use bot.reply because the person I want to notify is different from the person the chatbot is talking to currently.


CertificateRequest not recreated after the certificate's issuer has been updated

When a Certificate resource gets into a failure state and that this failure comes from a misconfiguration in your Issuer resource, you might want to edit the Issuer and force a renewal of the certificate with the command
cmctl renew cert-1

Instead of seeing the Certificate resource being re-issued, you will see the following message:
Not re-issuing certificate as an attempt has been made in the last hour

This issue is closely related to https://github.com/jetstack/cert-manager/pull/3444.
/kind bug

Think this was mentioned before (not sure where!), that it might be a good idea to add a --force flag to cmctl renew which overrides this behavior.
Given the command is manually invoked, it is probably a safe thing to do.


Change configuration parameter write_deadline type (from integer to time.Duration)

[X] Feature Request or Change Proposal

Feature Requests
Change the write_deadline configuration type from being an integer to be a string that can be parsed as a duration (say "25ms") to allow lower than 1 second deadline from configuration.
Use Case:
There may be some cases of 1 to many subscribers where one would want to reduce the write deadline so that one slow consumer in the list does not slow down too much the rest of group.
Proposed Change:
Support for write_deadline in the configuration was introduced in PR #421 as a number of seconds. The corresponding option, however, is a time.Duration. Since this is in master but not part of an official release, I wonder if this would be ok to modify the configuration parsing to look for a time.Duration instead of an integer.
People that are using the parameter (say: write_deadline: 1) would get an error saying that this parameter is expected to be a duration, and would need to change to write_deadline: "1s" for instance.
Who Benefits From The Change(s)?
People that need a smaller than 1 second write deadline from the configuration file.
Alternative Approaches
Introduce a new parameter parsed as time.Duration that would override the current write_deadline.
cc @petemiron This is based on the discussion that we had. I can submit a PR if desired.

Yes. Please create a PR. this is perfect. Thanks!


Fix validate

These commits enable the validate function to work with JunOS devices (and probably others) and also fix a problem with manager.raise_mode (setting raise mode was not working before).

Hi,
Can you check with the latest version 0.4.3?
Thnx in advance.
Best,
Leo

Yes, the rpc-error is detected in 0.4.3. I'll open another pull request for the other problems.


Specify set of icons to use

font awesome
materialize ui

fixed with https://github.com/jrm2k6/react-markdown-editor/commit/048409cdfb3274bf94fc01cbabe1f871fe462846


Which model did you used for extract feature?

Which model did you used for extract feature?
I wonder Which model did you used for extract feature?
Hope you can reply me soon. Thank you so much and sorry for my bad English

We used I3D features pre-trained on Kinetics.
For more details, please check: https://github.com/ahsaniqbal/Kinetics-FeatureExtractor


Add Win7' screenshot

Taken in a VirtualBox VM running Modern.IE' Windows 7 image.

Ah Win7, probably the best release of Windows before and after. Very stable, and very not bullshit compared to 8, 10, and 11.


set pbc_offsift and pos as float64

I set edata["pbc_offshift"] and ndata["pos"] as float64 to correctly handle creating line graphs when some bond lengths are right at the cutoff.
I added a test for this edge case with a made-up structure.

Thanks.


store.save() does not work

Cargo toml:
[build-dependencies]
tauri-build = { version = "1.0.0-rc.4", features = [] }

[dependencies]
serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] }
tauri = { version = "1.0.0-rc.4", features = ["clipboard-all", "dialog-all", "fs-read-file", "fs-write-file", "shell-all", "window-all"] }

[dependencies.tauri-plugin-store]
git = "https://github.com/tauri-apps/tauri-plugin-store"
branch = "dev"

[features]
# by default Tauri runs in production mode
# when `tauri dev` runs it is executed with `cargo run --no-default-features` if `devPath` is an URL
default = [ "custom-protocol" ]
# this feature is used used for production builds where `devPath` points to the filesystem
# DO NOT remove this
custom-protocol = [ "tauri/custom-protocol" ]

main.rs:
#![cfg_attr(
  all(not(debug_assertions), target_os = "windows"),
  windows_subsystem = "windows"
)]

use tauri_plugin_store::PluginBuilder;

fn main() {
  tauri::Builder::default()
    .plugin(PluginBuilder::default().build())
    .run(tauri::generate_context!())
    .expect("error while running tauri application");
}

package.json:
{
  "dependencies": {
    "tauri-plugin-store-api": "github:tauri-apps/tauri-plugin-store#dev",
  }
}

React code:
const store = new Store('.settings.dat');

let storeBases = await store.get('bases') as number[];
storeBases = storeBases ? [...storeBases, baseToAdd] : [baseToAdd];
await store.set('bases', storeBases);
await store.save();

The store operations work till the app is running (navigating away or into this page).
But the settings file is not persisted on disk and subsequent runs have a blank store.
I also have both readFile and writeFile enabled in the allowList.

I also have both readFile and writeFile enabled in the allowList.

That is not necessary, plugins shouldn't (and the official ones don't) depend on API functions.
Now to the real issue, this might be an issue with serialization. Any outputs in stdout?

tauri info:
$ tauri info

Operating System - Windows, version 10.0.19043 X64
Webview2 - 99.0.1150.55
Visual Studio Build Tools:
   - Visual Studio Build Tools 2019

Node.js environment
  Node.js - 16.13.2
  @tauri-apps/cli - 1.0.0-rc.5
  @tauri-apps/api - 1.0.0-rc.2

Global packages
  npm - 8.1.2
  pnpm - Not installed
  yarn - 1.22.17

Rust environment
  rustup - 1.24.3
  rustc - 1.59.0
  cargo - 1.59.0
  toolchain - stable-x86_64-pc-windows-msvc

App directory structure
/.git
/.github
/build
/node_modules
/public
/src
/src-tauri

App
  tauri - 1.0.0-rc.4
  tauri-build - 1.0.0-rc.4
  tao - 0.6.2
  wry - 0.13.3
  build-type - bundle
  CSP - unset
  distDir - ../build
  devPath - http://localhost:3000/
  framework - React

No additional information/errors in stdout.
Nothing in the webview console as well.

To re-populate the store from disk you need to call store.load(). Note that this is currently blocked by #64.

@FabianLars , where is the .settings.dat located in windows system? I don't see the file is created next to .exe file. but it does save and load.

@Tonghua-Li C:\Users\usernams\AppData\Local\bundle-identifier/, with "bundle-identifier" being the one configured in tauri.conf.json (never keep the default "com.tauri.dev").


I want to be able to upload the file directly to the application for translation instantly for a single document and not the file path

          > Hello @rahmlad-aramide , you can provide a file path and then read and feed the contents of that file into the body parameter. For example: `// Replace 'filePath' with the path to your file const filePath = 'd:/testdocs/test-input.txt'; // Read the contents of the file asynchronously const fileContent = await readFileAsync(filePath, 'utf8');

const options: DocumentTranslateParameters = {
  queryParameters: {
    targetLanguage: "hi",
  },
  contentType: "multipart/form-data",
  body: [
    {
      name: "document",
      body: fileContent,
      filename: "test-input.txt",
      contentType: "text/plain",
    },
  ],
};`

If you have a file in Azure blob, and want that to be translated into the right format, then you can use the example from here: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/translation/ai-translation-document-rest/samples/v1-beta/typescript/src/batchDocumentTranslation.ts

That's the challenge I'm facing, because I'm trying to integrate it into my frontend application whereby the user will upload the file and accessing the absolute file path of a user's uploaded file directly in the browser is not possible due to security reasons. Browsers do not expose the information to web applications to protect user privacy and system security. Likewise, using the batch translation, the output file is not produced instantly, or is it possible that I provide the URL to the azure blob storage to the synchronous file translator after uploading it there or there's no solution at all?
Thanks in advance for your help.
cc: @hamshavathimunibyraiah
Originally posted by @rahmlad-aramide in https://github.com/Azure/azure-sdk-for-js/issues/30312#issuecomment-2217103184

Meanwhile, after I tried to use the method that @hamshavathimunibyraiah gave above, I'm getting:
error: {
  code: 'InvalidRequest',
  message: 'The format parameter is not valid.',
  target: 'ContentType',
  innerError: {
    code: 'InvalidFormat',
    message: 'The format parameter is not valid.'
  }
}


Concurrent ApplyMethod calls to same Mobject should work

List of Changes

Enabled multiple ApplyMethod calls in a single play call.

Motivation

A possible fix for #713 and #722
Explanation for Changes

The reason why multiple ApplyMethod calls currently don't work is, I believe, because a new target is created through a copy of mobject for each ApplyMethod call.
My idea is to generate a new target to the mobject only if no targets are generated till that point. This can be done by checking if the attribute target is None (which means there are no targets generated so far). If mobject.target is not None, meaning a target was previously set, no new target will be generated.
Testing Status

Working example
class ApplyMethodTest(MovingCameraScene):
    def construct(self):
        c = Circle()
        self.add(c)
        self.wait()
        c.generate_target()
        c.target.scale(2)
        self.play(MoveToTarget(c))
        self.play(
            ApplyMethod(c.to_edge),
            c.set_color, BLUE,
            c.set_opacity, 0.5
        )
        self.wait()
        self.play(*map(FadeOut, self.mobjects))
        self.wait()

        r = Rectangle()
        self.play(ShowCreation(r))
        self.wait()
        self.play(
            ApplyMethod(r.set_height,6,True),
            r.set_opacity,1,
            ApplyMethod(r.set_color,RED)
        )
        self.wait()
        self.play(*map(FadeOut, self.mobjects))
        self.wait()

        c = Circle()
        c.shift(UR * 2)
        c.set_fill(ORANGE, opacity=1)
        gc = c.get_center()
        gcw = c.get_width()

        self.add(c)
        self.wait()
        self.play(
            ApplyMethod(self.camera_frame.move_to, gc),
            self.camera_frame.set_width, gcw * 2,
            )
        self.wait()

Acknowledgement

[x] I have read the Contributing Guidelines

This makes sense to me. @behackl would you be able to comment? I believe you are working on something similar with AnimationGroup.
@Pragyaan-6988 this is something that could really use adding new tests. Would you be able to add a couple of simple unit tests so we don't regress on this feature in the future? If you don't know how, let us know and we can help you.

Also, unfortunately the docs are not building with the following error.
File "/home/docs/checkouts/readthedocs.org/user_builds/manimce/envs/772/lib/python3.8/site-packages/manim/mobject/types/image_mobject.py", line 229, in interpolate_color
    assert mobject1.pixel_array.shape == mobject2.pixel_array.shape, (
AssertionError: Mobject pixel array shapes incompatible for interpolation.
Mobject 1 (ImageMobjectFromCamera) : (90, 180, 4)
Mobject 2 (ImageMobjectFromCamera) : (60, 360, 4)

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/manimce/envs/772/lib/python3.8/site-packages/manim/mobject/types/image_mobject.py", line 229, in interpolate_color
    assert mobject1.pixel_array.shape == mobject2.pixel_array.shape, (
AssertionError: Mobject pixel array shapes incompatible for interpolation.
Mobject 1 (ImageMobjectFromCamera) : (90, 180, 4)
Mobject 2 (ImageMobjectFromCamera) : (60, 360, 4)

This makes me think that your change broke one of the examples in our gallery. I don't have a ton of time right now, but I can help you figure out which example it is later.

The example broken by this change is https://docs.manim.community/en/latest/examples.html#movingzoomedscenearound -- something weird seems to be happening. I can't investigate right now whether the fact that this broke is okay (because it shouldn't have worked in the first place), or because this uncovers a problem with this PR.

@Pragyaan-6988 this is something that could really use adding new tests. Would you be able to add a couple of simple unit tests so we don't regress on this feature in the future? If you don't know how, let us know and we can help you.

@leotrs I've spent some time trying to figure out why this fails but couldn't come to a meaningful conclusion. May be I'm overlooking something. I'll look into this further when I have time, but if you guys have any ideas, please do share. I'm on the discord (Abhijith) if you'd like to discuss it there. Also, I have no idea about the unit tests. I'll ask for help when this issue is resolved. Thank you.

@Pragyaan-6988 sorry for the delay - I'm looking into it, though unfortunately I am not an expert in this part of the codebase. Let's see if anyone on discord can figure it out.

@Pragyaan-6988 sorry for the delay - I'm looking into it, though unfortunately I am not an expert in this part of the codebase. Let's see if anyone on discord can figure it out.

@leotrs No worries, this one is tricky  😅. I've actually realized that this isn't a good approach. Take this code for example:
class ApplyMethodTest(Scene):
    def construct(self):
        c = Circle()
        c.generate_target()
        c.target.to_edge()

        self.add(c)
        self.play(ApplyMethod(c.scale, 2))

The circle gets scaled and also moves to the left edge which shouldn't happen.
I made some changes to Scene.compile_play_args_to_animation_list and the results look good..

SGTM, Thanks for your cotnribution!
Nevertheless this would require a test. Read the wiki on github for more infos.
I'm not sure what would be better between a classic unit test or a graphical unit test.

Thanks for the review @huguesdevimeux ! I think I have to update this PR to work with the new .animate syntax. I haven't looked into it yet. I should also learn about the unit tests, so I'll update this PR with all that stuff in few days. Thank you.

SGTM, Thanks for your cotnribution!
Nevertheless this would require a test. Read the wiki on github for more infos.
I'm not sure what would be better between a classic unit test or a graphical unit test.

Thanks for the review @huguesdevimeux ! I think I have to update this PR to work with the new .animate syntax. I haven't looked into it yet. I should also learn about the unit tests, so I'll update this PR with all that stuff in few days. Thank you.

Hi, sorry for not updating on this for so long. So I was requested to add unit tests to this, however I don't think this PR is relevant anymore, given that the .animate syntax (together with method chaining) potentially replaces ApplyMethod entirely once 1118  is merged.
The only drawback is that it'd still not support multiple method animations on same mobject (each with different animation kwargs - which method chaining doesn't support). I believe this is an extremely rare situation, but still it'd be good if we can support that. However that shouldn't be in this PR, so I'm closing this.


SDC IDE UX

[x]  Implement tab, Shift+ tab behavior
[x] Rename apply changes -> save questionnaire
[ ] Changes in the editor immediately change the form
[x] to save questionnaire a use should press save questionnaire

New issue is created for "Changes in the editor immediately change the form" - https://github.com/beda-software/sdc-ide/issues/61


[Feature Request] Add "Open in modrinth/curseforge" button while in download page

Is your feature request related to a problem? Please describe.
Since Mod Manager sometimes fails to download mods, so I think we should add a button to download the mod manually.
Describe the solution you'd like
Add "Open in modrinth/curseforge" button while on the download page

Maybe not as an own button but when you click the install/update with the control button down then it will open the respective page


fixes #226

For avoid #226 I should call close() method on the running loop, but for awaiting the result it's not needed. Just returning a coroutine object from connector.

duplicate of #231


Use Optional On Getters is not configurable in Maven pom configuration

I have a fix for it done, but getting permission denied when trying to push up my feature branch...

Hi Martin. Thanks for looking at this.
If you fork this repo, you can update your git remote to point at your own fork. Then push the branch to your fork and create a PR.

Cool will do, cheers!

PR Created https://github.com/joelittlejohn/jsonschema2pojo/pull/895

Thanks!!


Does not alternate between threads (test/main)

It only prints to the screen from the main thread.
When run on QEMU. Haven't tried on RPI

It works on the real hardware.


Sagemath uses wrong path for three.min.js when calling plot3d

Describe the bug
When using the three viewer for plot3d, the generated html file contains the wrong path for three.min.js
Steps To Reproduce
Call the following code in sage
x,y=var('x,y')
plot3d(exp(-x^2-y^2), (x,-1,1), (y,-1,1))

Then your browser would open the generated file in ~/.sage/tmp/<hostname>/.
The browser console would report that
Loading failed for the <script> with source “file:///nix/store/4208lbj85fx0ja3szndn1r6frmbmkq7c-threejs-sage-r122/lib/node_modules/three/r122/three.min.js”

Uncaught ReferenceError: THREE is not defined

Upon further inspection, the wanted file is actually at /nix/store/<hash>/lib/node_modules/three/build/three.min.js and upon loading the page with this update path, it works.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Notify maintainers
@timokau @omasanori @7c6f434c @collares

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"

 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.68, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210908_3c56f62`
 - channels(root): `"nixos-21.11pre314199.f6e9e908ccb"`
 - channels(<username>): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

Thanks for the bug report. Unfortunately we can't use vanilla three.js because it does not include some exports used by sage (see the README at https://github.com/sagemath/threejs-sage), but it looks like threejs-sage is indeed installed into the wrong directory. I will fix it today.

By the way, the threejs-sage installation directory was changed upstream in https://trac.sagemath.org/ticket/30972, which landed in 9.4. I missed this when updating Sage in #130396.


add link to the protocol

https://github.com/Co-Epi/CEN for now, to be updated if we move it.

Can we pull this out and make it a bit more clear. Rather than linking it inline - maybe make a new section below the body with details?

https://github.com/TCNCoalition/website/pull/15


Improve handling of bad classpaths

/cc @baroquebobcat

Looks like the Github/Travis integration is stuck again. Here's the URL of the corresponding Travis build: https://travis-ci.org/twitter/rsc/builds/426997342.


update(userspace/falco): add libsinsp state metrics option

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

/kind release

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area engine

/area tests

/area proposals

/area CI

What this PR does / why we need it:
See the corresponding libs PR https://github.com/falcosecurity/libs/pull/1433
Which issue(s) this PR fixes:
https://github.com/falcosecurity/libs/issues/1347

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

update(userspace/falco): add libsinsp state metrics option

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/hold until the libs PR https://github.com/falcosecurity/libs/pull/1433 is merged
Then requires bumping libs as there were some minor breaking refactors.
Here is some example output showing the new counters and metrics when running Falco on a dev box while monitoring all spawned processes and any file opens:
{
  "output_fields": {
    "evt.source": "syscall",
    "evt.time":<PHONE_NUMBER>047532110,
    "falco.container_memory_used": 0,
    "falco.cpu_usage_perc": 4.8,
    "falco.cpu_usage_perc_total_host": 5.7,
    "falco.duration_sec": 80,
    "falco.evts_rate_sec":<PHONE_NUMBER>616168,
    "falco.host_boot_ts":<PHONE_NUMBER>000000000,
    "falco.host_num_cpus": 20,
    "falco.hostname": "",
    "falco.kernel_release": "",
    "falco.memory_pss": 65,
    "falco.memory_rss": 68,
    "falco.memory_used_host": 10734,
    "falco.memory_vsz": 1133,
    "falco.n_added_fds": 94985,
    "falco.n_added_threads": 131,
    "falco.n_cached_fd_lookups": 190444,
    "falco.n_cached_thread_lookups": 485510,
    "falco.n_containers": 1,
    "falco.n_drops_full_threadtable": 0,
    "falco.n_failed_fd_lookups": 8903,
    "falco.n_failed_thread_lookups": 149,
    "falco.n_fds": 154329,
    "falco.n_noncached_fd_lookups": 195010,
    "falco.n_noncached_thread_lookups": 11954,
    "falco.n_removed_fds": 95217,
    "falco.n_removed_threads": 134,
    "falco.n_retrieve_evts_drops": 0,
    "falco.n_retrieved_evts": 97367,
    "falco.n_store_evts_drops": 0,
    "falco.n_stored_evts": 97403,
    "falco.n_threads": 2425,
    "falco.num_evts": 389371,
    "falco.num_evts_prev": 385403,
    "falco.open_fds_host": 22280,
    "falco.outputs_queue_num_drops": 0,
    "falco.procs_running_host": 2,
    "falco.start_ts":<PHONE_NUMBER>957361054,
    "falco.version": "",
    "scap.engine_name": "bpf"
  },
  "sample": 16
}

/hold until the libs PR falcosecurity/libs#1433 is merged Then requires bumping libs as there were some minor breaking refactors.
Here is some example output showing the new counters and metrics when running Falco on a dev box while monitoring all spawned processes and any file opens:
{
  "output_fields": {
    "evt.source": "syscall",
    "evt.time":<PHONE_NUMBER>047532110,
    "falco.container_memory_used": 0,
    "falco.cpu_usage_perc": 4.8,
    "falco.cpu_usage_perc_total_host": 5.7,
    "falco.duration_sec": 80,
    "falco.evts_rate_sec":<PHONE_NUMBER>616168,
    "falco.host_boot_ts":<PHONE_NUMBER>000000000,
    "falco.host_num_cpus": 20,
    "falco.hostname": "",
    "falco.kernel_release": "",
    "falco.memory_pss": 65,
    "falco.memory_rss": 68,
    "falco.memory_used_host": 10734,
    "falco.memory_vsz": 1133,
    "falco.n_added_fds": 94985,
    "falco.n_added_threads": 131,
    "falco.n_cached_fd_lookups": 190444,
    "falco.n_cached_thread_lookups": 485510,
    "falco.n_containers": 1,
    "falco.n_drops_full_threadtable": 0,
    "falco.n_failed_fd_lookups": 8903,
    "falco.n_failed_thread_lookups": 149,
    "falco.n_fds": 154329,
    "falco.n_missing_container_images": 0,
    "falco.n_noncached_fd_lookups": 195010,
    "falco.n_noncached_thread_lookups": 11954,
    "falco.n_removed_fds": 95217,
    "falco.n_removed_threads": 134,
    "falco.n_retrieve_evts_drops": 0,
    "falco.n_retrieved_evts": 97367,
    "falco.n_store_evts_drops": 0,
    "falco.n_stored_evts": 97403,
    "falco.n_threads": 2425,
    "falco.num_evts": 389371,
    "falco.num_evts_prev": 385403,
    "falco.open_fds_host": 22280,
    "falco.outputs_queue_num_drops": 0,
    "falco.procs_running_host": 2,
    "falco.start_ts":<PHONE_NUMBER>957361054,
    "falco.version": "",
    "scap.engine_name": "bpf"
  },
  "sample": 16
}

Do we want to expose all these metrics? some of them seem to expose some internal details, like n_noncached_fd_lookups, n_noncached_thread_lookups, and others with the cache word. I'm not sure we can guarantee these metrics will remain stable in the future, maybe we will change how we look inside our tables, or how we use the caches, WDYT?
Does this metric mean that we have 2 hosts running on the system?
"falco.procs_running_host": 2,

Side note: we added 131 threads but we failed 149 lookups, seems pretty strange :thinking:
    "falco.n_added_threads": 131,
    "falco.n_failed_thread_lookups": 149,

/hold until the libs PR falcosecurity/libs#1433 is merged Then requires bumping libs as there were some minor breaking refactors.
Here is some example output showing the new counters and metrics when running Falco on a dev box while monitoring all spawned processes and any file opens:
{
  "output_fields": {
    "evt.source": "syscall",
    "evt.time":<PHONE_NUMBER>047532110,
    "falco.container_memory_used": 0,
    "falco.cpu_usage_perc": 4.8,
    "falco.cpu_usage_perc_total_host": 5.7,
    "falco.duration_sec": 80,
    "falco.evts_rate_sec":<PHONE_NUMBER>616168,
    "falco.host_boot_ts":<PHONE_NUMBER>000000000,
    "falco.host_num_cpus": 20,
    "falco.hostname": "",
    "falco.kernel_release": "",
    "falco.memory_pss": 65,
    "falco.memory_rss": 68,
    "falco.memory_used_host": 10734,
    "falco.memory_vsz": 1133,
    "falco.n_added_fds": 94985,
    "falco.n_added_threads": 131,
    "falco.n_cached_fd_lookups": 190444,
    "falco.n_cached_thread_lookups": 485510,
    "falco.n_containers": 1,
    "falco.n_drops_full_threadtable": 0,
    "falco.n_failed_fd_lookups": 8903,
    "falco.n_failed_thread_lookups": 149,
    "falco.n_fds": 154329,
    "falco.n_missing_container_images": 0,
    "falco.n_noncached_fd_lookups": 195010,
    "falco.n_noncached_thread_lookups": 11954,
    "falco.n_removed_fds": 95217,
    "falco.n_removed_threads": 134,
    "falco.n_retrieve_evts_drops": 0,
    "falco.n_retrieved_evts": 97367,
    "falco.n_store_evts_drops": 0,
    "falco.n_stored_evts": 97403,
    "falco.n_threads": 2425,
    "falco.num_evts": 389371,
    "falco.num_evts_prev": 385403,
    "falco.open_fds_host": 22280,
    "falco.outputs_queue_num_drops": 0,
    "falco.procs_running_host": 2,
    "falco.start_ts":<PHONE_NUMBER>957361054,
    "falco.version": "",
    "scap.engine_name": "bpf"
  },
  "sample": 16
}

Do we want to expose all these metrics? some of them seem to expose some internal details, like n_noncached_fd_lookups, n_noncached_thread_lookups, and others with the cache word. I'm not sure we can guarantee these metrics will remain stable in the future, maybe we will change how we look inside our tables, or how we use the caches, WDYT?
Does this metric mean that we have 2 hosts running on the system?
"falco.procs_running_host": 2,

Side note: we added 131 threads but we failed 149 lookups, seems pretty strange 🤔
    "falco.n_added_threads": 131,
    "falco.n_failed_thread_lookups": 149,

The idea was me deploying this so that we have data on what libsinsp state metrics are actually useful. I wouldn't be surprised if the data revealed some improvement opportunities for our state handling.
Re falco.procs_running_host that's the procs currently running on the Linux host based on https://github.com/falcosecurity/libs/pull/1433/files#diff-0c8e1b4c431bdf6d186fdbf777b2e158a65b2df9a8f03a24217c8a89d0fdad6eR53. We can either discuss the naming or see if we can fetch something better to have some sort of ground truth wrt how many processes are currently running. That way we could see if the state table grows too much out of bound relative to the currently running processes.

some of them seem to expose some internal details

The idea was me deploying this so that we have data on what libsinsp state metrics are actually useful

It seems like we don't want users to rely on internal details stats because they might get confused but we are willing to experiment with them. Before final release we need to make sure we're not exposing stuff that is not useful at all, or possibly mark it as internal. It looks to me (but correct me if I'm wrong) that this PR does not change which metrics are exposed but only makes it build properly with the latest libs. I would suggest an issue to keep track of this. Especially after experimentation there is stuff that will be much clearer.

We have already an issue https://github.com/falcosecurity/libs/issues/1463 for this.

Re falco.procs_running_host that's the procs currently running on the Linux host based on https://github.com/falcosecurity/libs/pull/1433/files#diff-0c8e1b4c431bdf6d186fdbf777b2e158a65b2df9a8f03a24217c8a89d0fdad6eR53. We can either discuss the naming or see if we can fetch something better to have some sort of ground truth wrt how many processes are currently running. That way we could see if the state table grows too much out of bound relative to the currently running processes.

ok, thank you. Yes maybe we can find a way to document what each metric means somewhere

It seems like we don't want users to rely on internal details stats because they might get confused but we are willing to experiment with them. Before final release we need to make sure we're not exposing stuff that is not useful at all, or possibly mark it as internal.

Yep maybe we can exclude them by default with a strncmp and enable them only when we are in a sort of "debug/internal" mode, maybe we can even mark some of them as internal in sinsp with a flag

LGTM label has been added.  Git tree hash: 0a3ec79560f2960531c2e90e96702eb0f70393c7

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, incertum, LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,LucaGuerra,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Update event only fires when typing very slowly

I tried adding debounce to the input template but it didn't help. Unless I type at the rate of 1 key per second, the ajax requests are basically ignored completely and nothing happens.

I have fixed it but I didn't publish it because of some issues on my pc. I just managed to publish it.
Thanks :)

That didn't fix it. It still doesn't respond to quick typing. It also seems that the query is always one character behind what I'm typing, so if i'm searching for blog the query is blo.

A apologize, I still had the hacked in debounce. It seems to be working now :) :+1:

It's ok :)

Exactly how did you manage to get debounce working? If I set it on the input element, debounce="300" it doesn't respond to quick typing and is always one character behind.

Hi @rogierborst!
You could try this: @input="update | debounce 300"
http://vuejs.org/api/#debounce

Thanks!
I figured it out myself and came back here to post my findings, only to see you already reacted 2 minutes after my question.

Hi, I just tried the debouncing using @input="update | debounce 300", but webpack says its an invalid expression.  Any ideas?


logInputs endpoint

Needs https://github.com/mlflow/mlflow/pull/13410

See https://github.com/psycopg/psycopg2/issues/1737 for the pinned psycopg2-binary


Is it possible to "free" the physical Bluetooth interface so that it can be used by external commands?

I would like to be able to use the physical Bluetooth interface without stopping the go application so I need some way to free up the device once it has been initialised. Is this possible?
While the code is running I get errors as expected:
package main

import (
	"flag"
	"fmt"
	"log"
	"time"

	"github.com/currantlabs/ble/examples/lib/dev"
)

var (
	device = flag.String("device", "default", "implementation of ble")
)

func main() {
	_, err := dev.NewDevice(*device)
	if err != nil {
		log.Fatalf("can't new device : %s", err)
	}
	fmt.Println("opened")

	time.Sleep(100 * time.Second)
}

 ~  gatttool -I -t random -b E0:1A:35:05:47:EB
[E0:1A:35:05:47:EB][LE]> connect
Attempting to connect to E0:1A:35:05:47:EB
Error: connect: No route to host (113)
[E0:1A:35:05:47:EB][LE]>
[E0:1A:35:05:47:EB][LE]>
 ~  sudo hciconfig hci0 reset
Can't down device hci0: Device or resource busy (16)
 ✘  ~  sudo hciconfig hci0 down
Can't down device hci0: Device or resource busy (16)
 ✘  ~  hciconfig dev
hci0:    Type: Primary  Bus: USB
    BD Address: E4:B3:18:4D:40:8F  ACL MTU: 1021:4  SCO MTU: 96:6
    UP RUNNING
    RX bytes:49338 acl:2 sco:0 events:3888 errors:0
    TX bytes:625918 acl:181 sco:0 commands:4087 errors:0

 ~  gatttool -I -t random -b E0:1A:35:05:47:EB
[E0:1A:35:05:47:EB][LE]> connect
Attempting to connect to E0:1A:35:05:47:EB
Error: connect: No route to host (113)
[E0:1A:35:05:47:EB][LE]>

Is it possible to free the device created and then use external commands during the sleep period?

Hi @josephroberts from a reading of the current code this is not currently possible, at least not in the manner you described.
At least on Linux, the NewDevice() function is opening the HCI and does not release it until Stop() is called, at which point you would need another NewDevice() call to reopen the connection.

Speaking of the "Stop()" method  (in ble/linux/device.go), it doesn't appear to work. I try to use it to stop the Gatt server and it is completely ineffectual. Looking at the code I can see why. The method simply calls HCl.Close(), and HCI.Close() does nothing but return nil. I tried modifying the "Stop()" method to instead call HCI.Stop(), but that is ineffectual as well.  What I want to do is detect when a peripheral has connected to the Gatt server; get the address of the peripheral; and if it is not what is expected, disconnect the server from the peripheral. I have figured out how to do everything except the disconnect part. I figured I could simply stop the Gatt server to accomplish the disconnect, but I can't find a way to do this. The aforementioned "Stop()" method would seem to be intended to do this but as I wrote above, it's not getting the job done. Any ideas?
Thanks

@bbartol The Stop() here is to release the HCI device (or HCI socket), although currently it is not implemented yet as you indicated :). #30 is working on this though. I'm pretty sure something like that worked before though I haven't got it work on my current environment yet. You can give it a shot.
However, from your description, it sounds more like a "disconnect" at the connection level.
We used to have something like onConnect(conn), which will be invoked when the server is connected, and you can pull the connection info from that conn.  Is this what you were looking for?
We should add that functionality back soon.

@roylee17, Thanks for the shout back. I have figured out how to do everything that you described as onConnect(conn)'s functionality. What's left is to figure out how to disconnect from the server side. Nothing I have tried so far has worked. Calling Disconnect stops my whole program; and as I wrote before, the code to Stop() the server is ineffectual (because it has only been stubbed out so far). I got so far as finding the code where the HCI device is first opened. In that code, an HCIdown and HCIup is executed at the beginning to effect a reset of the device before proceeding to bind the device to the HCI channel.  So I thought maybe I could do the same thing to reset the device and that would cause the server to stop.  What I found, however, is that neither HCIdown nor HCIup would work once the device is bound to the HCI channel (they both return non-nIl for error). So then I embarked on trying to figure out how to unbind the device. I searched through the code for a couple of hours and found nothing to do this and became concerned that I was embarking down a rabbit hole. So that's where I'm at.
Thanks again.

@bbartol

Calling Disconnect stops my whole program

Is it blocked or crashed? Can you try it with go routine?
Certain cases will re-enter our HCI handling state machine resulting a deadlock. Callback() is one of the cases. I'll add the onConnect/onDisconnect notification/callback back soon, so we can work on it on the same page. This is the preferred approach for scenario.
On the other hand. The device up/down/reset in #30 is another way to "work around" your scenario.  As you tried, once the socket got bound, it can't be up/down/reset. With regular TCP/IP sockets, we can use some socket options, or shutdown() instead of close() on the socket. But that doesn't seem to work on the HCI socket.
I'm pretty sure that worked for me earlier though I haven't reproduced it in my current environment. Unless I was using multiple HCI usb dongles, and the auto fallback logic (passing -1 as device id) fooled me into believe that I'm reusing the same HCI device while it actually picked another for me...
Anyway, I'm building my kernel from source so I can peak and poke the it and track what's happening underneath.

@roylee17 @bbartol I am spending the week trying to get to the bottom of this. I think the HCI interface is EBUSY the second time around because of an unfinished read on the socket - see this line in the strace log below [pid  6479] 10:22:43.319529 read(3,  <unfinished ...>, the read never gets resumed.
Any ideas on how to complete the read either before closing the socket or when opening it?
System specs:
$ uname -a
Linux x1pro 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Linux
$ go version
go version go1.7.5 linux/amd64

main.go
package main

import (
	"flag"
	"log"
	"time"

	"github.com/currantlabs/ble/examples/lib/dev"
)

var (
	device = flag.String("device", "default", "implementation of ble")
	name   = flag.String("name", "Gopher", "name of remote peripheral")
	addr   = flag.String("addr", "", "address of remote peripheral (MAC on Linux, UUID on OS X)")
	sub    = flag.Duration("sub", 0, "subscribe to notification and indication for a specified period")
	sd	   = flag.Duration("sd", 5*time.Second, "scanning duration, 0 for indefinitely")
)

func main() {
	d, err := dev.NewDevice(*device)
	if err != nil {
		log.Fatalf("can't new device : %s", err)
	} else {
		log.Printf("Opened")
	}

	time.Sleep(1 * time.Second)

	d.Stop();

	time.Sleep(1 * time.Second)

	d, err = dev.NewDevice(*device)
	if err != nil {
		log.Fatalf("can't new device : %s", err)
	} else {
		log.Printf("Opened")
	}
}

Strace output:
$ sudo strace -f -tt -e trace=ioctl,socket,close,open,read,write ./ble-test-currantlabs
10:22:43.221018 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
10:22:43.221138 close(3)                = 0
10:22:43.221167 open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
10:22:43.221193 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320`\0\0\0\0\0\0"..., 832) = 832
10:22:43.221317 close(3)                = 0
10:22:43.221350 open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
10:22:43.221391 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\3\2\0\0\0\0\0"..., 832) = 832
10:22:43.221546 close(3)                = 0
strace: Process 6478 attached
strace: Process 6479 attached
strace: Process 6480 attached
strace: Process 6481 attached
[pid  6477] 10:22:43.225149 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid  6477] 10:22:43.225372 read(3, "128\n", 4096) = 4
[pid  6477] 10:22:43.225406 read(3, "", 4092) = 0
[pid  6477] 10:22:43.225441 close(3)    = 0
[pid  6477] 10:22:43.225493 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  6477] 10:22:43.225545 close(3)    = 0
[pid  6477] 10:22:43.225592 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  6477] 10:22:43.225653 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid  6477] 10:22:43.225716 close(4)    = 0
[pid  6477] 10:22:43.225744 close(3)    = 0
[pid  6477] 10:22:43.225824 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  6477] 10:22:43.225855 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid  6477] 10:22:43.225883 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  6477] 10:22:43.225915 ioctl(3, HCIDEVUP, 0) = 0
[pid  6477] 10:22:43.277745 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  6477] 10:22:43.299644 write(3, "\1\3\f\0", 4) = 4
[pid  6479] 10:22:43.299959 read(3, "\4\16\4\2\3\f\0", 4096) = 7
[pid  6479] 10:22:43.308560 read(3,  <unfinished ...>
[pid  6477] 10:22:43.308827 write(3, "\1\t\20\0", 4) = 4
[pid  6479] 10:22:43.309392 <... read resumed> "\4\16\n\1\t\20\0\217@M\30\263\344", 4096) = 13
[pid  6479] 10:22:43.309670 read(3,  <unfinished ...>
[pid  6477] 10:22:43.309964 write(3, "\1\5\20\0", 4) = 4
[pid  6479] 10:22:43.310338 <... read resumed> "\4\16\v\1\5\20\0\375\3`\4\0\6\0", 4096) = 14
[pid  6479] 10:22:43.310532 read(3,  <unfinished ...>
[pid  6477] 10:22:43.310753 write(3, "\1\2 \0", 4) = 4
[pid  6479] 10:22:43.311380 <... read resumed> "\4\16\7\1\2 \0\33\0\7", 4096) = 10
[pid  6479] 10:22:43.311686 read(3,  <unfinished ...>
[pid  6477] 10:22:43.312016 write(3, "\1\7 \0", 4) = 4
[pid  6479] 10:22:43.313344 <... read resumed> "\4\16\5\1\7 \0\7", 4096) = 8
[pid  6479] 10:22:43.313570 read(3,  <unfinished ...>
[pid  6480] 10:22:43.314017 write(3, "\1\1 \10\37\0\0\0\0\0\0\0", 12) = 12
[pid  6479] 10:22:43.315353 <... read resumed> "\4\16\4\1\1 \0", 4096) = 7
[pid  6479] 10:22:43.315572 read(3,  <unfinished ...>
[pid  6480] 10:22:43.315747 write(3, "\1\1\f\10\377\377\373\377\7\370\277=", 12) = 12
[pid  6479] 10:22:43.316325 <... read resumed> "\4\16\4\1\1\f\0", 4096) = 7
[pid  6479] 10:22:43.316471 read(3,  <unfinished ...>
[pid  6480] 10:22:43.316671 write(3, "\1m\f\2\1\0", 6) = 6
[pid  6479] 10:22:43.317339 <... read resumed> "\4\16\4\1m\f\0", 4096) = 7
[pid  6479] 10:22:43.317476 read(3,  <unfinished ...>
[pid  6480] 10:22:43.317747 write(3, "\1\6 \17 \0 \0\0\0\0\0\0\0\0\0\0\7\0", 19) = 19
[pid  6479] 10:22:43.318336 <... read resumed> "\4\16\4\1\6 \0", 4096) = 7
[pid  6479] 10:22:43.318591 read(3,  <unfinished ...>
[pid  6480] 10:22:43.318873 write(3, "\1\v \7\1\4\0\4\0\0\0", 11) = 11
[pid  6479] 10:22:43.319337 <... read resumed> "\4\16\4\1\v \0", 4096) = 7
[pid  6479] 10:22:43.319529 read(3,  <unfinished ...>
[pid  6480] 10:22:43.320252 read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 3687
[pid  6480] 10:22:43.320411 read(4, "", 4096) = 0
[pid  6480] 10:22:43.320531 close(4)    = 0
[pid  6480] 10:22:43.320747 write(2, "2017/01/30 10:22:43 Opened\n", 272017/01/30 10:22:43 Opened
) = 27
strace: Process 6482 attached
[pid  6480] 10:22:44.322405 write(2, "2017/01/30 10:22:44 device stop\n", 322017/01/30 10:22:44 device stop
) = 32
[pid  6480] 10:22:44.322750 write(2, "2017/01/30 10:22:44 HCI public c"..., 372017/01/30 10:22:44 HCI public close
) = 37
[pid  6480] 10:22:44.323025 write(2, "2017/01/30 10:22:44 HCI private "..., 382017/01/30 10:22:44 HCI private close
) = 38
[pid  6480] 10:22:44.323222 write(2, "2017/01/30 10:22:44 Socket close"..., 332017/01/30 10:22:44 Socket close
) = 33
[pid  6480] 10:22:44.323364 close(3)    = 0
[pid  6480] 10:22:45.324217 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  6480] 10:22:45.324378 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid  6480] 10:22:45.324450 ioctl(3, HCIDEVDOWN, 0) = -1 EBUSY (Device or resource busy)
[pid  6480] 10:22:45.324709 write(2, "2017/01/30 10:22:45 can't new de"..., 1282017/01/30 10:22:45 can't new device : can't init hci: no devices available: (hci0: can't down device: device or resource busy)
) = 128
[pid  6480] 10:22:45.324899 +++ exited with 1 +++
[pid  6482] 10:22:45.324923 +++ exited with 1 +++
[pid  6481] 10:22:45.324983 +++ exited with 1 +++
[pid  6478] 10:22:45.324999 +++ exited with 1 +++
[pid  6479] 10:22:45.326381 <... read resumed> <unfinished ...>) = ?
[pid  6479] 10:22:45.326776 +++ exited with 1 +++
10:22:45.326799 +++ exited with 1 +++

This is the blocking line. Is it necessary to always trigger a write to complete the read? The finished reads always do this...
[pid  6479] 10:22:43.308560 read(3,  <unfinished ...>
[pid  6477] 10:22:43.308827 write(3, "\1\t\20\0", 4) = 4
[pid  6479] 10:22:43.309392 <... read resumed> "\4\16\n\1\t\20\0\217@M\30\263\344", 4096) = 13

The unix.Read function is only used by a couple of libraries, is it possible the unix.Read function has a bug?
I modified the Read function to look like this and Unlocked never prints the last time round.
func (s *Socket) Read(p []byte) (int, error) {
	s.rmu.Lock()
    log.Printf("locked")
	defer s.rmu.Unlock()
	n, err := unix.Read(s.fd, p)
    log.Printf("Unlocked")
	return n, errors.Wrap(err, "can't read hci socket")
}

I did notice something strange - the read resumes when the process is killed. This line in the strace output below [pid 28494] 12:29:24.711194 <... read resumed> <unfinished ...>) = ?
[sudo] password for joe:
12:29:12.598744 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
12:29:12.599050 close(3)                = 0
12:29:12.599158 open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
12:29:12.599247 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320`\0\0\0\0\0\0"..., 832) = 832
12:29:12.599559 close(3)                = 0
12:29:12.599615 open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
12:29:12.599658 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\3\2\0\0\0\0\0"..., 832) = 832
12:29:12.599797 close(3)                = 0
strace: Process 28493 attached
strace: Process 28494 attached
strace: Process 28495 attached
strace: Process 28496 attached
[pid 28492] 12:29:12.602930 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid 28492] 12:29:12.603137 read(3, "128\n", 4096) = 4
[pid 28492] 12:29:12.603170 read(3, "", 4092) = 0
[pid 28492] 12:29:12.603193 close(3)    = 0
[pid 28492] 12:29:12.603224 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 28492] 12:29:12.603252 close(3)    = 0
[pid 28492] 12:29:12.603279 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 28492] 12:29:12.603328 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid 28492] 12:29:12.603376 close(4)    = 0
[pid 28492] 12:29:12.603398 close(3)    = 0
[pid 28492] 12:29:12.603478 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid 28492] 12:29:12.603503 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid 28492] 12:29:12.603550 read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 3687
[pid 28492] 12:29:12.603578 read(4, "", 4096) = 0
[pid 28492] 12:29:12.603599 close(4)    = 0
[pid 28492] 12:29:12.603634 write(2, "2017/01/30 12:29:12 Starting ope"..., 342017/01/30 12:29:12 Starting open
) = 34
[pid 28492] 12:29:12.603660 write(2, "2017/01/30 12:29:12 Polling\n", 282017/01/30 12:29:12 Polling
) = 28
[pid 28492] 12:29:13.605458 write(2, "2017/01/30 12:29:13 Polled\n", 272017/01/30 12:29:13 Polled
) = 27
[pid 28492] 12:29:13.605707 ioctl(3, HCIDEVDOWN, 0) = 0
[pid 28492] 12:29:13.605934 ioctl(3, HCIDEVUP, 0) = 0
[pid 28492] 12:29:13.658808 ioctl(3, HCIDEVDOWN, 0) = 0
[pid 28492] 12:29:13.679811 write(2, "2017/01/30 12:29:13 Finished ope"..., 342017/01/30 12:29:13 Finished open
) = 34
[pid 28492] 12:29:13.680094 write(3, "\1\3\f\0", 4) = 4
[pid 28492] 12:29:13.680231 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28492] 12:29:13.680421 read(3, "\4\16\4\2\3\f\0", 4096) = 7
[pid 28492] 12:29:13.689222 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28492] 12:29:13.689394 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28492] 12:29:13.689611 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28492] 12:29:13.689722 read(3,  <unfinished ...>
[pid 28494] 12:29:13.689989 write(3, "\1\t\20\0", 4) = 4
[pid 28492] 12:29:13.691121 <... read resumed> "\4\16\n\1\t\20\0\217@M\30\263\344", 4096) = 13
[pid 28492] 12:29:13.691214 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28492] 12:29:13.691386 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28492] 12:29:13.691606 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28492] 12:29:13.691709 read(3,  <unfinished ...>
[pid 28494] 12:29:13.691903 write(3, "\1\5\20\0", 4) = 4
[pid 28492] 12:29:13.693066 <... read resumed> "\4\16\v\1\5\20\0\375\3`\4\0\6\0", 4096) = 14
[pid 28492] 12:29:13.693126 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28492] 12:29:13.693228 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28492] 12:29:13.693399 write(3, "\1\2 \0", 4) = 4
[pid 28494] 12:29:13.693532 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.693635 read(3, "\4\16\7\1\2 \0\33\0\7", 4096) = 10
[pid 28494] 12:29:13.694119 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.694222 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.694413 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.694502 read(3,  <unfinished ...>
[pid 28492] 12:29:13.694618 write(3, "\1\7 \0", 4) = 4
[pid 28494] 12:29:13.695055 <... read resumed> "\4\16\5\1\7 \0\7", 4096) = 8
[pid 28494] 12:29:13.695132 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.695255 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.695448 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.695553 read(3,  <unfinished ...>
[pid 28492] 12:29:13.695677 write(3, "\1\1 \10\37\0\0\0\0\0\0\0", 12) = 12
[pid 28494] 12:29:13.696046 <... read resumed> "\4\16\4\1\1 \0", 4096) = 7
[pid 28494] 12:29:13.696106 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.696209 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.696375 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.696467 read(3,  <unfinished ...>
[pid 28492] 12:29:13.696575 write(3, "\1\1\f\10\377\377\373\377\7\370\277=", 12) = 12
[pid 28494] 12:29:13.697043 <... read resumed> "\4\16\4\1\1\f\0", 4096) = 7
[pid 28494] 12:29:13.697102 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.697206 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.697363 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.697454 read(3,  <unfinished ...>
[pid 28492] 12:29:13.697562 write(3, "\1m\f\2\1\0", 6) = 6
[pid 28494] 12:29:13.698085 <... read resumed> "\4\16\4\1m\f\0", 4096) = 7
[pid 28494] 12:29:13.698146 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.698250 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.698407 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.698499 read(3,  <unfinished ...>
[pid 28492] 12:29:13.698631 write(3, "\1\6 \17 \0 \0\0\0\0\0\0\0\0\0\0\7\0", 19) = 19
[pid 28494] 12:29:13.699042 <... read resumed> "\4\16\4\1\6 \0", 4096) = 7
[pid 28494] 12:29:13.699103 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.699207 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.699366 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.699460 read(3,  <unfinished ...>
[pid 28492] 12:29:13.699563 write(3, "\1\v \7\1\4\0\4\0\0\0", 11) = 11
[pid 28494] 12:29:13.700042 <... read resumed> "\4\16\4\1\v \0", 4096) = 7
[pid 28494] 12:29:13.700102 write(2, "2017/01/30 12:29:13 Unlocked\n", 292017/01/30 12:29:13 Unlocked
) = 29
[pid 28494] 12:29:13.700205 write(2, "2017/01/30 12:29:13 Read\n", 252017/01/30 12:29:13 Read
) = 25
[pid 28494] 12:29:13.700361 write(2, "2017/01/30 12:29:13 locked\n", 272017/01/30 12:29:13 locked
) = 27
[pid 28494] 12:29:13.700456 read(3,  <unfinished ...>
[pid 28492] 12:29:13.700783 write(2, "2017/01/30 12:29:13 Opened\n", 272017/01/30 12:29:13 Opened
) = 27
strace: Process 28500 attached
strace: Process 28501 attached
[pid 28492] 12:29:18.702772 write(2, "2017/01/30 12:29:18 device stop\n", 322017/01/30 12:29:18 device stop
) = 32
[pid 28492] 12:29:18.703115 write(2, "2017/01/30 12:29:18 HCI public c"..., 372017/01/30 12:29:18 HCI public close
) = 37
[pid 28492] 12:29:18.703389 write(2, "2017/01/30 12:29:18 HCI private "..., 382017/01/30 12:29:18 HCI private close
) = 38
[pid 28492] 12:29:18.703623 write(2, "2017/01/30 12:29:18 Socket close"..., 332017/01/30 12:29:18 Socket close
) = 33
[pid 28492] 12:29:18.703778 close(3)    = 0
[pid 28492] 12:29:23.705743 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid 28492] 12:29:23.705995 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid 28492] 12:29:23.706254 write(2, "2017/01/30 12:29:23 Starting ope"..., 342017/01/30 12:29:23 Starting open
) = 34
[pid 28492] 12:29:23.706469 write(2, "2017/01/30 12:29:23 Polling\n", 282017/01/30 12:29:23 Polling
) = 28
[pid 28492] 12:29:24.708944 write(2, "2017/01/30 12:29:24 Polled\n", 272017/01/30 12:29:24 Polled
) = 27
[pid 28492] 12:29:24.709312 ioctl(3, HCIDEVDOWN, 0) = -1 EBUSY (Device or resource busy)
[pid 28492] 12:29:24.709840 write(2, "2017/01/30 12:29:24 can't new de"..., 1282017/01/30 12:29:24 can't new device : can't init hci: no devices available: (hci0: can't down device: device or resource busy)
) = 128
[pid 28501] 12:29:24.710327 +++ exited with 1 +++
[pid 28500] 12:29:24.710381 +++ exited with 1 +++
[pid 28496] 12:29:24.710422 +++ exited with 1 +++
[pid 28495] 12:29:24.710477 +++ exited with 1 +++
[pid 28493] 12:29:24.710512 +++ exited with 1 +++
[pid 28494] 12:29:24.711194 <... read resumed> <unfinished ...>) = ?
[pid 28494] 12:29:24.712223 +++ exited with 1 +++
12:29:24.712284 +++ exited with 1 +++

So far I have tried to force the read to finish with a few methods, none of which work.
I attempted to flush the socket before closing it with each of the commented out lines.
func (s *Socket) Close() error {
    log.Printf("Socket close")
    // unix.Sync()
    // unix.Fsync(s.fd)
    // unix.Fdatasync(s.fd)
	return errors.Wrap(unix.Close(s.fd), "can't close hci socket")
}

I tried to poll for data when the socket is first opened.
func open(fd, id int) (*Socket, error) {
    log.Printf("Starting open")

    log.Printf("Polling")
	// poll for 20ms to see if any data becomes available, then clear it
	pfds := []unix.PollFd{unix.PollFd{Fd: int32(fd), Events: unix.POLLIN}}
	unix.Poll(pfds, 1000)
	if pfds[0].Revents&unix.POLLIN > 0 {
		b := make([]byte, 100)
		unix.Read(fd, b)
	}
    log.Printf("Polled")

	// Reset the device in case previous session didn't cleanup properly.
	if err := ioctl(uintptr(fd), hciDownDevice, uintptr(id)); err != nil {
		return nil, errors.Wrap(err, "can't down device")
	}
	if err := ioctl(uintptr(fd), hciUpDevice, uintptr(id)); err != nil {
		return nil, errors.Wrap(err, "can't up device")
	}

	// HCI User Channel requires exclusive access to the device.
	// The device has to be down at the time of binding.
	if err := ioctl(uintptr(fd), hciDownDevice, uintptr(id)); err != nil {
		return nil, errors.Wrap(err, "can't down device")
	}

	// Bind the RAW socket to HCI User Channel
	sa := unix.SockaddrHCI{Dev: uint16(id), Channel: unix.HCI_CHANNEL_USER}
	if err := unix.Bind(fd, &sa); err != nil {
		return nil, errors.Wrap(err, "can't bind socket to hci user channel")
	}

	// poll for 20ms to see if any data becomes available, then clear it
	pfds = []unix.PollFd{unix.PollFd{Fd: int32(fd), Events: unix.POLLIN}}
	unix.Poll(pfds, 20)
	if pfds[0].Revents&unix.POLLIN > 0 {
		b := make([]byte, 100)
		unix.Read(fd, b)
	}

    log.Printf("Finished open")

	return &Socket{fd: fd}, nil
}

@josephroberts
Take a look at this.
The read() will block until there is something to read and close() does not unblock the read.

@moogle19 thanks, so perhaps we are issuing an extra read that is not needed.

I have narrowed it down to the read and writes - small steps haha - by changing the hci Init() function to comment out and read/write parts.
func (h *HCI) Init() error {
	h.evth[0x3E] = h.handleLEMeta
	h.evth[evt.CommandCompleteCode] = h.handleCommandComplete
	h.evth[evt.CommandStatusCode] = h.handleCommandStatus
	h.evth[evt.DisconnectionCompleteCode] = h.handleDisconnectionComplete
	h.evth[evt.NumberOfCompletedPacketsCode] = h.handleNumberOfCompletedPackets

	h.subh[evt.LEAdvertisingReportSubCode] = h.handleLEAdvertisingReport
	h.subh[evt.LEConnectionCompleteSubCode] = h.handleLEConnectionComplete
	h.subh[evt.LEConnectionUpdateCompleteSubCode] = h.handleLEConnectionUpdateComplete
	h.subh[evt.LELongTermKeyRequestSubCode] = h.handleLELongTermKeyRequest
	// evt.EncryptionChangeCode:                     todo),
	// evt.ReadRemoteVersionInformationCompleteCode: todo),
	// evt.HardwareErrorCode:                        todo),
	// evt.DataBufferOverflowCode:                   todo),
	// evt.EncryptionKeyRefreshCompleteCode:         todo),
	// evt.AuthenticatedPayloadTimeoutExpiredCode:   todo),
	// evt.LEReadRemoteUsedFeaturesCompleteSubCode:   todo),
	// evt.LERemoteConnectionParameterRequestSubCode: todo),

	skt, err := socket.NewSocket(h.id)
	if err != nil {
		return err
	}
	h.skt = skt

	h.chCmdBufs <- make([]byte, 64)

	// go h.sktLoop()
	// h.init()

	// Pre-allocate buffers with additional head room for lower layer headers.
	// HCI header (1 Byte) + ACL Data Header (4 bytes) + L2CAP PDU (or fragment)
	// h.pool = NewPool(1+4+h.bufSize, h.bufCnt-1)

        // h.Send(&h.params.advParams, nil)
        // h.Send(&h.params.scanParams, nil)
	return nil
}

Here is the strace output, showing the interface being Uped and Downed on the second time round:
14:03:00.498685 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
14:03:00.498957 close(3)                = 0
14:03:00.499044 open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
14:03:00.499107 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320`\0\0\0\0\0\0"..., 832) = 832
14:03:00.499369 close(3)                = 0
14:03:00.499432 open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
14:03:00.499485 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\3\2\0\0\0\0\0"..., 832) = 832
14:03:00.499704 close(3)                = 0
strace: Process 8274 attached
strace: Process 8275 attached
strace: Process 8276 attached
strace: Process 8277 attached
[pid  8273] 14:03:00.505739 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid  8273] 14:03:00.505999 read(3, "128\n", 4096) = 4
[pid  8273] 14:03:00.506037 read(3, "", 4092) = 0
[pid  8273] 14:03:00.506068 close(3)    = 0
[pid  8273] 14:03:00.506111 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  8273] 14:03:00.506155 close(3)    = 0
[pid  8273] 14:03:00.506190 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  8273] 14:03:00.506264 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid  8273] 14:03:00.506335 close(4)    = 0
[pid  8273] 14:03:00.506388 close(3)    = 0
[pid  8273] 14:03:00.506502 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  8273] 14:03:00.506555 ioctl(3, HCIGETDEVLIST, 0xc42004dc0c) = 0
[pid  8273] 14:03:00.506602 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  8273] 14:03:00.506650 ioctl(3, HCIDEVUP, 0) = 0
[pid  8273] 14:03:00.559983 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  8273] 14:03:00.582630 read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 3687
[pid  8273] 14:03:00.582725 read(4, "", 4096) = 0
[pid  8273] 14:03:00.582768 close(4)    = 0
[pid  8273] 14:03:00.582832 write(2, "2017/01/30 14:03:00 Opened\n", 272017/01/30 14:03:00 Opened
) = 27
[pid  8273] 14:03:01.583373 close(3)    = 0
[pid  8273] 14:03:02.587812 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  8273] 14:03:02.588231 ioctl(3, HCIGETDEVLIST, 0xc42004dc0c) = 0
[pid  8273] 14:03:02.588587 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  8273] 14:03:02.588752 ioctl(3, HCIDEVUP, 0) = 0
[pid  8273] 14:03:02.640996 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  8273] 14:03:02.662194 write(2, "2017/01/30 14:03:02 Opened\n", 272017/01/30 14:03:02 Opened
) = 27
[pid  8276] 14:03:02.662489 +++ exited with 0 +++
[pid  8277] 14:03:02.662541 +++ exited with 0 +++
[pid  8275] 14:03:02.662565 +++ exited with 0 +++
[pid  8274] 14:03:02.663258 +++ exited with 0 +++
14:03:02.663301 +++ exited with 0 +++

@josephroberts
It is this Read() that is blocking which is called by this loop.
The problem is, that the loop is needed to check if there is data on the socket.

@moogle19 yep. I am trying to solve this using poll, it gets a little further and then hangs.
Modification to socket.go
func (s *Socket) Read(p []byte) (int, error) {
	s.rmu.Lock()
	defer s.rmu.Unlock()

	pfds := []unix.PollFd{unix.PollFd{Fd: int32(s.fd), Events: unix.POLLIN}}
	unix.Poll(pfds, 0)
	if pfds[0].Revents&unix.POLLIN > 0 {
		n, err := unix.Read(s.fd, p)
		return n, errors.Wrap(err, "can't read hci socket")
	}

	return 0, nil
}

Modification to hci.go
func (h *HCI) sktLoop() {
	b := make([]byte, 4096)
	defer close(h.done)
	for {
		n, err := h.skt.Read(b)
		if err != nil {
			h.err = fmt.Errorf("skt: %s", err)
			return
		}
		if n == 0 {
			continue
		}
		p := make([]byte, n)
		copy(p, b)
		if err := h.handlePkt(p); err != nil {
			h.err = fmt.Errorf("skt: %s", err)
			return
		}
	}
}

Strace output
15:07:56.188948 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
15:07:56.189403 close(3)                = 0
15:07:56.189484 open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
15:07:56.189548 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320`\0\0\0\0\0\0"..., 832) = 832
15:07:56.189754 close(3)                = 0
15:07:56.189815 open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
15:07:56.189866 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\3\2\0\0\0\0\0"..., 832) = 832
15:07:56.190000 close(3)                = 0
strace: Process 7376 attached
strace: Process 7377 attached
strace: Process 7378 attached
strace: Process 7379 attached
[pid  7375] 15:07:56.194198 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
[pid  7375] 15:07:56.194602 read(3, "128\n", 4096) = 4
[pid  7375] 15:07:56.194664 read(3, "", 4092) = 0
[pid  7375] 15:07:56.194737 close(3)    = 0
[pid  7375] 15:07:56.194798 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  7375] 15:07:56.194864 close(3)    = 0
[pid  7375] 15:07:56.194929 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid  7375] 15:07:56.195041 socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid  7375] 15:07:56.195112 close(4)    = 0
[pid  7375] 15:07:56.195151 close(3)    = 0
[pid  7375] 15:07:56.195292 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  7375] 15:07:56.195339 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid  7375] 15:07:56.195384 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  7375] 15:07:56.195415 ioctl(3, HCIDEVUP, 0) = 0
[pid  7375] 15:07:56.247351 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  7377] 15:07:56.268671 write(3, "\1\3\f\0", 4) = 4
[pid  7375] 15:07:56.281340 read(3, "\4\16\4\2\3\f\0", 4096) = 7
[pid  7377] 15:07:56.282732 write(3, "\1\t\20\0", 4) = 4
[pid  7375] 15:07:56.291654 read(3, "\4\16\n\1\t\20\0\217@M\30\263\344", 4096) = 13
[pid  7377] 15:07:56.292097 write(3, "\1\5\20\0", 4) = 4
[pid  7375] 15:07:56.293006 read(3, "\4\16\v\1\5\20\0\375\3`\4\0\6\0", 4096) = 14
[pid  7377] 15:07:56.293480 write(3, "\1\2 \0", 4) = 4
[pid  7375] 15:07:56.294515 read(3, "\4\16\7\1\2 \0\33\0\7", 4096) = 10
[pid  7377] 15:07:56.294856 write(3, "\1\7 \0", 4) = 4
[pid  7375] 15:07:56.296026 read(3, "\4\16\5\1\7 \0\7", 4096) = 8
[pid  7377] 15:07:56.296507 write(3, "\1\1 \10\37\0\0\0\0\0\0\0", 12) = 12
[pid  7375] 15:07:56.296967 read(3, "\4\16\4\1\1 \0", 4096) = 7
[pid  7377] 15:07:56.297252 write(3, "\1\1\f\10\377\377\373\377\7\370\277=", 12) = 12
[pid  7375] 15:07:56.297980 read(3, "\4\16\4\1\1\f\0", 4096) = 7
[pid  7377] 15:07:56.298277 write(3, "\1m\f\2\1\0", 6) = 6
[pid  7375] 15:07:56.299000 read(3, "\4\16\4\1m\f\0", 4096) = 7
[pid  7377] 15:07:56.299391 write(3, "\1\6 \17 \0 \0\0\0\0\0\0\0\0\0\0\7\0", 19) = 19
[pid  7375] 15:07:56.300017 read(3, "\4\16\4\1\6 \0", 4096) = 7
[pid  7377] 15:07:56.300639 write(3, "\1\v \7\1\4\0\4\0\0\0", 11) = 11
[pid  7375] 15:07:56.301983 read(3, "\4\16\4\1\v \0", 4096) = 7
[pid  7377] 15:07:56.302713 read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"..., 4096) = 3687
[pid  7377] 15:07:56.302849 read(4, "", 4096) = 0
[pid  7377] 15:07:56.302923 close(4)    = 0
[pid  7377] 15:07:56.302987 write(2, "2017/01/30 15:07:56 Opened\n", 272017/01/30 15:07:56 Opened
) = 27
strace: Process 7380 attached
[pid  7377] 15:07:57.303152 close(3)    = 0
[pid  7377] 15:07:58.303894 socket(AF_BLUETOOTH, SOCK_RAW, 1) = 3
[pid  7377] 15:07:58.303943 ioctl(3, HCIGETDEVLIST, 0xc42004dc04) = 0
[pid  7377] 15:07:58.303975 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  7377] 15:07:58.303999 ioctl(3, HCIDEVUP, 0) = 0
[pid  7377] 15:07:58.354958 ioctl(3, HCIDEVDOWN, 0) = 0
[pid  7377] 15:07:58.377053 write(3, "\1\3\f\0", 4) = 4
[pid  7375] 15:07:58.386797 read(3, "\4\16\4\2\3\f\0", 4096) = 7
[pid  7378] 15:07:58.390170 read(3, ^Cstrace: Process 7375 detached
strace: Process 7376 detached
strace: Process 7377 detached
strace: Process 7378 detached
 <detached ...>
strace: Process 7379 detached
strace: Process 7380 detached

it hangs here: [pid  7378] 15:07:58.390170 read(3, ^Cstrace: Process 7375 detached whilst trying to setup the interface for the second time, so I am gonnna debug that now.
BTW - I am being verbose here for my own notes :P

Non-blocking I/O Select / Poll/ Epoll breaks the read into two separate system calls, which might introduce latency if context switch happens in between. (latency might be visible or not depends on the platform)
Let's see if we can tackle this within GO land (tricky though...).

Fixed by #32

Thanks everyone for the feedback to my original post.  I was able to accomplish the one piece that was vexing me (disconnecting the client from the server side) by calling Device.HCI.conns[ClientMapKey].Close().  The trick was discerning ClientMapKey (the key that references the connection to the client that is currently connected to my Gatt server).
I was able to accomplish everything that I need (at least so far) by adding the attached code to my local copy of the ble library.  I organized this code so that none of the existing code (other than the example I discuss below) needed to be modified.  If you unzip (keeping the folder structure) into your local copy of the github.com/currantlabs/ble folder (for example, my local copy is at /go/src/github.com/currantlabs/ble), you should be able to rebuild your code with any of the functionality I added and try it out if you want.  Included in the attached (in ble/examples/blesh_AGEX) is a scaled down and heavily modified copy of the original blesh example.  Changes that I made to said example are commented as "AGEX" (//AGEX).  In summary, they are as follows:
main.go:

Added 2 parameters to the curr struct - "device_AGEX", and "name":

  var curr struct {
	  device       ble.Device
  	  client       ble.Client
	  clients      map[string]ble.Client
	  uuid         ble.UUID
	  addr         ble.Addr
	  profile      *ble.Profile
	  device_AGEX  ble.Device_AGEX //AGEX
	  name         string //AGEX
  }

Modified func main - stripped out everything but support for the "serve" command.  Since everything I am trying to do is centered around running a Gatt server, I didn't need the other commands; thus I decided to remove them for simplicity's sake.
Modified func setup to call linux.NewDevice() directly rather than through NewDevice->DefaultDevice.
Modified func setup to instantiate curr.device_AGEX to get access to the functionality I have added.
Modified func cmdServe.  I basically turned this function into my sandbox.
Added functions "ScanForConnAddr" and "DisconnectAndKillServer".  The supporting code for these functions is in the other files (outside of the examples folder).

adv.go:

Added this line of code to func advHandler:
curr.name = a.LocalName()

Other

Copied the files count.go, echo,go, and uuids.go in ble/examples/lib over to ble/examples/blesh_AGEX so that I could flatten the code a bit and make my example to be contained all in one folder.  I didn't need any of the files in ble/examples/lib/dev nor did I need battery.go in ble/examples/lib.
Deleted files lnx.go, util.go, and exp.go in ble/examples/blesh_AGEX - didn't need them

ble.zip


Example Projects Compilation Errors

Hello, we are trying to compile the example projects for U585-IOT Kit , here we get the following errors for the SBSFU_Boot Project.
My colleague had this kind of compilation error:
Windows_NT Postbuild with python script Traceback (most recent call last):   File "D:\repo\cellular\Anjay-stm32-azurertos-client\Middlewares\Third_Party\mcuboot\scripts\imgtool.py", line 19, in <module>     import click ModuleNotFoundError: No module named 'click' postbuild.sh failed
When I try to compile the same project I get
Windows_NT Postbuild with python script Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases. postbuild.sh failed make[1]: *** [makefile:96: post-build] Error 1 make: *** [makefile:62: all] Error 2 "make -j12 all" terminated with exit code 2. Build might be incomplete.
Python is installed and can be seen on the path
PS C:\Users\admin\Desktop\Azure\Anjay-stm32-azurertos-client> $env:PATH C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.16.10262.0_x64__8wekyb3d8bbwe;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Users\admin\AppData\Local\Programs\Python\Python311\Scripts;C:\Users\admin\AppData\Local\Programs\Python\Python311;C:\Users\admin\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Java\jdk-19\bin;;C:\Program Files\JetBrains\PyCharm Community Edition 2022.3.1\bin;;C:\Users\admin\AppData\Local\Programs\Microsoft VS Code\bin PS C:\Users\admin\Desktop\Azure\Anjay-stm32-azurertos-client> python --version Python 3.11.3
What are we missing?

Hi @gencay3535
For now we only support build process on Linux. We can probably prepare a Windows version by the end of this week.
I'll let you know when it's ready.
Regards
Witold

Hi @gencay3535
I just updated the repo. In https://github.com/AVSystem/Anjay-stm32-azurertos-client/blob/main/Projects/B-U585I-IOT02A/README.md you can find new Dependencies section which contain the list of python modules that you need to run the building and flashing process. Furthermore in shell scripts we added PATH for default STM32CubeProgrammer location on Windows.
Regards
Witold

Hello @witekgeb  Thank you for the update, I have installed all the dependencies but I had python not found error from the IDE Projects\B-U585I-IOT02A\SBSFU_Boot\STM32CubeIDE\postbuild.sh and my output for the python --version is python 3.11 then I made a modification at the postbuild.sh in line 29:
-cmd="python3 " +cmd="python3 "
Now I can compile, this would create a problem for the project?

My other question is this; This project is based on TF-M or X-CUBE-SBSFU?

@gencay3535 our Windows computers recognize cmd="python3 " probably due to some additional dependencies on the machine. The correct way on Windows is to call cmd="python ". We'll fix that, but for now your modification should be enough.

We are based our code on the SBSFU example: https://github.com/STMicroelectronics/STM32CubeU5/tree/v1.0.2/Projects/B-U585I-IOT02A/Applications/SBSFU


Update T10 about extremally disconnected

closes #533

Looks good.
One minor thing.  When a mathse reference refers to a specific mathse answer to a mathse question, we link to that particular answer (which is the case here), and we usually put in the "name" field "Answer to ..." like this:

name: Answer to "What separation is required to ensure extremally disconnected spaces are sequentially discrete?"

See https://topology.pi-base.org/spaces/S000172/references for comparison.  But it's no big deal.  Let me know either way and I'll approve.

I changed the PR title to something more descriptive.  If we later look at the list of PRs that got merged, it's good to have a more informative title to know what that was about without having to open each PR.

Let me know either way and I'll approve.

I'll fix it this time, to help me get in the habit of doing it the right way.

Actually, wait, I think the citation was the same as the question...

Hmm, I think I was just confused, I guess there must be a convention on stack exchange where the reference to the question is the same number as that of the first answer?  That might explain things.
In any case, if it still doesn't look right let me know @prabau, I'm pretty sure I did it correctly.

For clarity: here's the ID for a question:

To get a direct ID to an answer (can also be used to get the ID of a question):


Add manual_assert

What it does
Detects if-then-panic! that can be replaced with assert!.
Why is this bad?
assert! is simpler than if-then-panic!.
Example
let sad_people: Array<&str> = arrray![];
if !sad_people.is_empty() {
    panic!("there are sad people: {:?}", sad_people);
}

Use instead:
let sad_people: Array<&str> = array![];
assert!(sad_people.is_empty(), "there are sad people: {:?}", sad_people);

https://rust-lang.github.io/rust-clippy/master/index.html#/manual_assert

Hi, I would like to work on this!

let's go!


push bookmark modal to the right to not cover the zoom controls

^^ existing ^^

^^ proposed ^^

Coverage remained the same at 98.44% when pulling ec43c9ac1f391c9f46fea39c7dea21847799c26e on CarlyLanglois:example/bookmarks-modal-position into a9dd8a529e00da655cc36e5043cfd8c3dc161112 on boundlessgeo:master.


Null check operator used on a null value inside ListController.getOffsetToReveal

Got this error from one of user devices running on iOS 17.5.1 and Flutter 3.22.3.
Not sure what is up with that as value is being asserted for null two times before this error is thrown.
TypeError: Null check operator used on a null value
  #0      ListController.getOffsetToReveal (package:super_sliver_list/src/super_sliver_list.dart:275)
  #1      ListController.jumpToItem (package:super_sliver_list/src/super_sliver_list.dart:93)
...
  #3      ChangeNotifier.notifyListeners (package:flutter/src/foundation/change_notifier.dart:433)
...
  #10      _RootZone.runUnaryGuarded (dart:async/zone.dart:1594)
  #11      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:365)
  #12      _DelayedData.perform (dart:async/stream_impl.dart:541)
  #13      _PendingEvents.handleNext (dart:async/stream_impl.dart:646)
  #14      _PendingEvents.schedule.<fn> (dart:async/stream_impl.dart:617)
  #15      _microtaskLoop (dart:async/schedule_microtask.dart:40)
  #16      _startMicrotaskLoop (dart:async/schedule_microtask.dart:49)

Getting more of those from user's devices. Flutter 3.24.4 now and also on Android devices.
@knopp any clues how to debug this? Does this mean that the ExtentManager delegate is not set?
I'm somewhat puzzled that assertion on 274 pass but the non-null dereference on the next line is failing.

Hey @ekuleshov, are you running the app in debug mode? As assertions don't work in profile and release mode.

Hey @ekuleshov, are you running the app in debug mode? As assertions don't work in profile and release mode.

Unlikely. This stack trace is from a user device with app installed from the app store.


issue #188

Thanks for the PR!
The way you did it would break backwards-compatibility. What if someone wants to actually write null as a string into a table? Also 0 and NULL are not interchangeable, e.g. someone having 0 money and NULL money are two completely different things.


Added ability to track changes for property minLength, maxLength, minValue, maxValue, extendedTypeName attributes

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.aurelija.monkevice seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Not able to get data from config server when updated my app from 2.0 -> 2.1

Hello,
I have an existing web api project(dotnet core 2.0.7) which connects to Config Server and fetches data. I updated my project (dotnet core 2.1.0), and since then i am unable to fetch data from config server.
I only did the following changes to my project
.csproj
Earlier
<PropertyGroup>
<TargetFramework>netcoreapp2.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.All" Version="2.0.7" />
<PackageReference Include="Pivotal.Extensions.Configuration.ConfigServerCore" Version="2.0.0" />
</ItemGroup>
Now
<PropertyGroup>
<TargetFramework>netcoreapp2.1</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.All" Version="2.1.0" />
<PackageReference Include="Pivotal.Extensions.Configuration.ConfigServerCore" Version="2.1.0" />
</ItemGroup>
Can you please help

Do you get any error messages? It could help if you enable logging

Hello @TimHess
Thanks , i found the issue
In the logs i found
Could not locate PropertySource: System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.) ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
This issue i was facing while using httpClient with dotnet core 2.1, as a fix i added
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
at the starting of Program.cs
Now its working fine.
Thanks again


Access rights considerations in a docker installation

I have an image present at the right location (I checked), with the right authorizations, with a fairly standard name: IMG_20180408_153807553_HDR.jpg
And yet it is not treated by the Imagor server with this error:
{"Op":"Get","URL":"https://pho/tst/IMG_20180408_153807553_HDR.jpg","Err":{"Op":"dial","Net":"tcp","Source":null,"Addr":null,"Err":{"Err":"no such host","Name":"pho","Server":"<IP_ADDRESS>:53","IsTimeout":false,"IsTemporary":false,"IsNotFound":true}}}
Other images with underscores or uppercase letters are being treated fine in the exact same environment.
The only difference is that the image weighs 3MB but that should not be an issue and is not what the error reports anyway.
Please let me know what I am doing wrong.

I rebooted the server and it works fine now.
Sorry.

Actually, it is fairly random, because another image is not being treated now. I have a hunch it is because of the image size (the last one is 3.6MB)

Here is the screen copy of the /mnt/data/ folder in the stack:

The image is indeed here and yet I get thie new error message:
{"Op":"Get","URL":"https://pho/tst/IMG_20180408_152248953_HDR.jpg","Err":{"Op":"dial","Net":"tcp","Source":null,"Addr":null,"Err":{"Err":"no such host","Name":"pho","Server":"<IP_ADDRESS>:53","IsTimeout":false,"IsTemporary":false,"IsNotFound":true}}}

OK, I get it now.
If I change the access rights to 777 (which is insecure) to the images in the folder, they are treated.
Could you add a notice in teh documentation explaining what access rights you expect the image to have?
-rwxrwxrwx : good but terribly insecure
-rw: not good

It is currently requesting 0755 permission for mkdir
and 0666 for create file.
v0.2.4 ddded permission options for file storage mkdir and write, by setting these env variables:
      FILE_STORAGE_MKDIR_PERMISSION: 0755
      FILE_STORAGE_WRITE_PERMISSION: 0666

https://hub.docker.com/layers/shumc/imagor/0.2.4/images/sha256-2e87998ea5e83eb74720b01decb50b9209dfd37f60a2ef04277ef87556a991de?context=repo

Great, that was the rightg thing to do.


Traineddata path issue on Windows 10.

When i run
python -m table_ocr.demo https://raw.githubusercontent.com/eihli/image-table-ocr/master/resources/test_data/simple.png
i get
pytesseract.pytesseract.TesseractError: (1, 'Error opening data file C:UsersGetyAppDataLocalProgramsPythonPython38libsite-packagestable_ocrtessdata/table-ocr.traineddata Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory. Failed loading language \'table-ocr\' Tesseract couldn\'t load any languages! Could not initialize tesseract.')
(note file path does not have '/')
File does exist
I tried setting env variable TESSDATA_PREFIX - same error.
as well as specifying path in cli python -m table_ocr.demo https://raw.githubusercontent.com/eihli/image-table-ocr/master/resources/test_data/simple.png --tessdata-dir C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\Lib\site-packages\table_ocr\tessdata
I am on Windows 10.

Sorry to say that I have very little knowledge of Windows 10. I'll leave this comment open for a while in case anyone else has a suggestion.

I'd write my workaround.
https://github.com/eihli/image-table-ocr/blob/49205462a3fb68240fd6a3d441ae7cf979b43daa/table_ocr/ocr_image/__init__.py#L30
Replace all backslash to forward slashes. tessdata_dir.replace("\\", "/").

Also I found out that the built-in Tesseract data that my installation has is more superior compared to the shipped one, so I removed the related line entirely.

Я бы написал свой обходной путь.
https://github.com/eihli/image-table-ocr/blob/49205462a3fb68240fd6a3d441ae7cf979b43daa/table_ocr/ocr_image/__init__.py#L30
Замените все символы обратной косой черты на прямые косые черты. tessdata_dir.replace("\\", "/").
После внесения необходимых изменений программа работает. Windows 10, Python 3.10.4.

Thank you, I try to find problem all day

If anyone wants to submit a patch to make this more portable across Linux/Windows, please do!

你好！邮件已收到，谢谢！


[mDNS] :mdns_lite app needs a restart

If the :mdns_lite app stops it stays stopped.
( Ad-hoc Testing with FarmbotOS v8.0.2 )

Hmmm . . on further diagnosis, maybe I got this wrong.  Closing for now.


Automatically deployed queue after deleting log without noticable warning

Maybe I overlooked it, bit in that case i feel it would not be prominent enough. Good would be a hint with an unchecked checkbox saying do you want to additionally remove the queue/deploy all versions from queue. Default should be that this is not happening automatically, since we have had this problem now multiple times that the bugfix for the lock was not deployed because it was not part of the queue, but the bug itself. I do not see a good reason for this behaviour of deploying the queue automatically at this point

tbh I'd like to remove queues entirely. But I'm not sure if we can get there quickly, so we will re-think the warnings that are shown.

the unexpected lock behavior (deploying a queue after deleting a lock) is removed in v 0.4.47: https://github.com/freiheit-com/kuberpult/blob/main/CHANGELOG.md#0447


Add default tool outputs to .gitignore

Following the contributing guide in the docs creates the following:

.coverage - for coverage information
.nox/ - for nox environments

Adding these to the .gitignore improves the developer experience.

Thank you


add HSM Identity proposal

Hello, we are trying to use HSM with the GO SDK, our objective is to use it with a wallet, some kind of identityprovider, we are creating this PR to interact with you, so you can guide us further:

Is this the right approach for this project?
Using Cryptosuite with PKCS11 interface is not as clear as we wanted it to be.
Using identitymanager or using the mspwallet to create a manager is not explained thoroughly

We want to do something like this(inspired by the node SDK): https://github.com/hyperledger/fabric-sdk-node/blob/main/fabric-network/src/impl/wallet/hsmx509identity.ts
We have some of the possible interfaces and some of the functions, but when we think how to link everything with a wallet, we just don't know if the path we are following is correct
Hope you can guide us, thanks a lot!

bstasyszyn can you give us some guidance? thanks in advance

Hi @jsebastianms1 - sure, I think you are on the right path, but the current wallet framework might need extending a bit to support HSM.  I guess (but I haven't tried) you can create a new wallet implementation that stores the HSM configuration and user certificate, and pass that down to the underlying PKCS11 via BCCSP.
Example configuration for the pkcs11 integration test: https://github.com/hyperledger/fabric-sdk-go/blob/main/test/fixtures/config/config_e2e_pkcs11.yaml#L42-L53
This is how PKCS11 cryptosuite is loaded: https://github.com/hyperledger/fabric-sdk-go/blob/main/test/integration/e2e/pkcs11/e2e_test.go

Hi @andrew-coleman  I have a doubt, i did the thing you suggested, like this:

hsm wallet implementation that stores HSM configuration and user certificate
hsmx509identity that stores type,version,mspid and credentials that are composed only by the Public Certificate
When i call the function WithIdentity from gateway.go im trying to recreate the new identity, switching based on the type (X509 and Hsmx509) like this:

switch creds.idType() { 		case x509Type: 			enrollmentCertificate = []byte(creds.(*X509Identity).Certificate()) 			privateKey, _ = fabricCaUtil.ImportBCCSPKeyFromPEMBytes([]byte(creds.(*X509Identity).Key()), cryptosuite.GetDefault(), true) 		case Hsmx509type: 			enrollmentCertificate = []byte(creds.(*Hsmx509Identity).Certificate()) 			conf := wallet.(*Wallet).store.(*hsmWalletStore).hsmConf 			print(string(conf)) 			sdk, _ := fabsdk.New(config.FromFile(conf)) 			configBackend, _ := sdk.Config() 			cryptoSuiteConfig := cryptosuite.ConfigFromBackend(configBackend) 			csp, _ := pkcs11.GetSuiteByConfig(cryptoSuiteConfig) 			certPubK, _ := csp.KeyImport(creds.(*Hsmx509Identity).Credentials.Certificate, &bccsp.ECDSAGoPublicKeyImportOpts{Temporary: true}) 			// Get the key given the SKI value 			ski := certPubK.SKI() 			privateKey, _ = csp.GetKey(ski) 		}
Right now, i really dont know if this is the path that you were trying to lead me to or if im totally lost, im having trouble obtaining the sdk from the path of the wallet, i used a similar file like the yaml you showed me.
i have a new fork, the other one was recently deleted, what can you suggest me to push this new implementation?
Thanks in advance


imposible to install phpoffice/phpspreadsheet

Is there a way to install phpoffice/phpspreadsheet, I have try a lot of possibilities, but non of them seem to work. I always get the next message:
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - phpoffice/phpspreadsheet 1.14.1 requires ext-gd * -> the requested PHP extension gd is missing from your system.
    - phpoffice/phpspreadsheet 1.14.0 requires ext-gd * -> the requested PHP extension gd is missing from your system.
    - Installation request for phpoffice/phpspreadsheet ^1.14 -> satisfiable by phpoffice/phpspreadsheet[1.14.0, 1.14.1].

  To enable extensions, verify that they are enabled in your .ini files:
    - /usr/local/etc/php/php-cli.ini
    - /usr/local/etc/php/conf.d/docker-php-ext-sodium.ini
    - /usr/local/etc/php/conf.d/docker-php-ext-zip.ini
  You can also run `php --ini` inside terminal to see which files are used by PHP in CLI mode.

Installation failed, reverting ./composer.json to its original content.

Any way, this is the last  version php.dockerfile I end up, as you can see, it has gd installed, I'm out of ideas, Thanks for any help
FROM php:7.3-fpm-alpine

COPY --from=composer:latest /usr/bin/composer /usr/bin/composer

WORKDIR /var/www/html

#Install imagemagick:
ENV MAGICK_HOME=/usr

RUN apk --no-cache update \
    && apk --no-cache upgrade \
    && apk add --update freetype-dev libwebp-dev libpng libpng-dev zlib-dev libxpm-dev gd \
    libjpeg-turbo-dev libxml2-dev autoconf g++ imagemagick-dev imagemagick libtool make \
    && docker-php-ext-configure gd \
        --with-gd \
        --with-freetype-dir=/usr/include/ \
        --with-png-dir=/usr/include/ \
        --with-jpeg-dir=/usr/include/ \
    && docker-php-ext-install gd \
    && docker-php-ext-install mbstring \
    && docker-php-ext-install mysqli \
    && docker-php-ext-install opcache \
    && docker-php-ext-install soap \
    && docker-php-ext-install pdo pdo_mysql \
    && pecl install imagick \
    && docker-php-ext-enable imagick \
    && docker-php-ext-enable gd \
    && apk del autoconf g++ libtool make

@gbelot2003 I faced the same issue and found out, that it was related to the way composer is served. The way @aschmelyun did, was to create it as a service, but this service doesn't know all the extensions you are adding, I assume it's something to do with the php-cli.ini, not having the extensions.
They way I fixed this was to remove the composer service, from docker-compose.yml and add this to the end of the Dockerfile for the php build:
COPY --from=composer:latest /usr/bin/composer /usr/bin/composer
Now you can run composer like this:
docker-compose run --rm php composer

They way I fixed this was to remove the composer service, from docker-compose.yml and add this to the end of the Dockerfile for the php build:

I really miss this part, thanks you!!


Loss sometimes goes to nan even with the gradient clipping

Didn't figure out why yet and any advice for this is welcome!

@carpedm20 in my NTM implementation (and in a couple of others I saw out there) nans were usually caused by one of the following:

Initializing the memory to zero. The memory appears in the denominator of the cosine distance and that makes it nan. Check if that is not your case and possibly add a small constant in the denominator and avoid initializing the memory to all zeros (make it a small constant).

negative sharpening value. that creates a complex number and also makes the cost function go nan

I think there was a third case but I don't remember right now. Good luck debugging! :D

@EderSantana  could you explain what is the meaning of negative sharpening value? Thanks

@carpedm20 what about w = powed_conv_w / ( tf.reduce_sum(powed_conv_w) + 1e-6) ?

the sharpening value is is uses as pow(input, sharpening). So it can't be negative. Use a nonlinearity like softplus to avoid getting negative values: sharpening = tf.nn.softplus(sharpening).

Having a negative sharpening value wouldn't make a real become imaginary. But in the paper Graves explicitly states that the sharpening value is >= 1, so softplus(gamma) + 1 would work fine.
a^(-b) = 1/(a^b)